User Guide¶

Overview¶

JPype is an effort to allow Python programs full access to Java class libraries. This is achieved not through re-implementing Python, as Jython/JPython has done, but rather through interfacing at the native level in both virtual machines.

Eventually, it should be possible to replace Java with Python in many, though not all, situations. JSP, Servlets, RMI servers and IDE plugins are all good candidates.

Why such a project?¶

As much as I enjoy programming in Python, there is no denying that Java has the bulk of the mindshare. Just look on Sourceforge, at the time of creation of this project, there were 3267 Python-related projects, and 12126 Java- related projects. And that’s not counting commercial interests.

Server-side Python is also pretty weak. Zope may be a great application server, but I have never been able to figure it out. Java, on the other hand, shines on the server.

So in order to both enjoy the language, and have access to the most popular libraries, I have started this project.

What about Jython?¶

Jython (formerly known as JPython) is a great idea. However, it suffers from a large number of drawbacks, i.e. it always lags behind CPython, it is slow and it does not allow access to most Python extensions.

My idea allows using both kinds of libraries in tandem, so the developer is free to pick and choose.

Using JPype¶

Here is a sample program to demonstrate how to use JPype:

from jpype import *
startJVM(getDefaultJVMPath(), "-ea")
java.lang.System.out.println("hello world")
shutdownJVM()

This is of course a simple hello world type of application. Yet it shows the 2 most important calls: startJVM and shutdownJVM.

The rest will be explained in more detail in the next sections.

Core Ideas¶

Threading¶

Any non-trivial application will have need of threading. Be it implicitly by using a GUI, or because you’re writing a multi-user server. Or explicitly for performance reason.

The only real problem here is making sure Java threads and Python threads cooperate correctly. Thankfully, this is pretty easy to do.

Python Threads¶

For the most part, Python threads based on OS level threads (i.e. posix threads) will work without problem. The only thing to remember is to call jpype.attachThreadToJVM() in the thread body to make the JVM usable from that thread. For threads that you do not start yourself, you can call isThreadAttachedToJVM() to check.

Java Threads¶

At the moment, it is not possible to use threads created from Java, since there is no callback available.

Other Threads¶

Some Python libraries offer other kinds of thread, (i.e. microthreads). How they interact with Java depends on their nature. As stated earlier, any OS- level threads will work without problem. Emulated threads, like microthreads, will appear as a single thread to Java, so special care will have to be taken for synchronization.

Synchronization¶

Java synchronization support can be split into two categories. The first is the synchronized keyword, both as prefix on a method and as a block inside a method. The second are the different methods available on the Object class (notify, notifyAll, wait).

To support the synchronized functionality, JPype defines a method called synchronized(O). O has to be a Java object or Java class. The return value is a monitor object that will keep the synchronization on as long as the object is kept alive. Use Python with statement to control the exact scope. Do not hold onto the object indefinitly without a with statement, the lock will be not be broken until the monitor is garbage collected. CPython and PyPy have difference GC rules. See synchronized for details of how to properly synchronize.

The other synchronization methods are available as-is on any JObject. However, as general rule one should not use synchronization methods on Java String as internal string representations may not be complete objects.

For synchronization that does not have to be shared with Java code, I suggest using Python’s support instead of Java’s, as it’ll be more natural and easier.

Performance¶

JPype uses JNI, which is well known in the Java world as not being the most efficient of interfaces. Further, JPype bridges two very different runtime environments, performing conversion back and forth as needed. Both of these can impose rather large performance bottlenecks.

JNI is the standard native interface for most, if not all, JVMs, so there is no getting around it. Down the road, it is possible that interfacing with CNI (GCC’s java native interface) may be used. The only way to minimize the JNI cost is to move some code over to Java.

Follow the regular Python philosophy : Write it all in Python, then write only those parts that need it in C. Except this time, it’s write the parts that need it in Java.

For the conversion costs, again, nothing much can be done. In cases where a given object (be it a string, an object, an array, etc …) is passed often into Java, you can pre-convert it once using the wrappers, and then pass in the wrappers. For most situations, this should solve the problem.

As a final note, while a JPype program will likely be slower than its pure Java counterpart, it has a good chance of being faster than the pure Python version of it. The JVM is a memory hog, but does a good job of optimizing code execution speeds.

Inner Classes¶

For the most part, inner classes can be used like normal classes, with the following differences:

Inner classes in Java natively use $ to separate the outer class from the inner class. For example, inner class Foo defined inside class Bar is called Bar.Foo in Java, but its real native name is Bar$Foo.
Inner classes appear as member of the containing class. Thus to access them simply import the outer class and call them as members.
Non-static inner classes cannot be instantiated from Python code. Instances received from Java code that can be used without problem.

Arrays¶

JPype has full support for receiving Java arrays and passing them to Java methods. Java arrays, wrapped in the JArray wrapper class, behave like Python lists, except that their size is fixed, and that the contents are of a specific type.

Multi-dimensional arrays (array of arrays) also work without problem.

As of version 0.5.5.3 we use NumPy arrays to interchange data with Java. This is much faster than using lists, since we do not need to handle every single array element but can process all data at once.

If you do not want this optional feature, because eg. it depends on NumPy, you can opt it out in the installation process by passing “–disable-numpy” to setup.py. To opt out with pip you need to append the additional argument “–global-option=’–disable-numpy’. This possibility exists since version 0.5.6.

Creating Java arrays from Python¶

The JArray wrapper is used to create Arrays from Python code. The code to create an array is like this:

JArray(type, num_dims)(sz or sequence)

Type is either a Java Class (as a String or a Java Class object) or a Wrapper type. num_dims is the number of dimensions to build the array and defaults to 1.

sz is the actual number of elements in the arrays, and sequence is a sequence to initialize the array with.

The logic behind this is that JArray(type, ndims) returns an Array Class, which can then be called like any other class to create an instance.

Type conversion¶

One of the most complex parts of a bridge system like JPype is finding a way to seamlessly translate between Python types and Java types. The following table will show what implicit conversions occur, both Python to Java and Java to Python. Explicit conversion, which happens when a Python object is wrapped, is converted in each wrapper.

Conversion from Python to Java¶

This type of conversion happens when a Python object is used either as a parameter to a Java method or to set the value of a Java field.

Type Matching¶

JPype defines different levels of “match” between Python objects and Java types. These levels are:

none, There is no way to convert.
explicit (E), JPype can convert the desired type, but only explicitly via the wrapper classes. This means the proper wrapper class will access this type as argument.
implicit (I), JPype will convert as needed.
exact> (X), Like implicit, but when deciding with method overload to use, one where all the parameters match “exact” will take precedence over “implicit” matches.

There are special rules for java.lang.Object as compared with a specific Java object. In Java, primitives are boxed automatically when passing to a java.lang.Object.

Python\Java	byte	short	int	long	float	double	boolean	char	String	Array	Object	java.lang.Object	java.lang.Class
int	I [1]	I [1]	X	I	I [3]	I [3]	X [8]					I [11]
long	I [1]	I [1]	I [1]	X	I [3]	I [3]						I [11]
float					I [1]	X						I [12]
sequence
dictionary
string								I [2]	X			I
unicode								I [2]	X			I
JByte	X											I [9]
JShort		X										I [9]
JInt			X									I [9]
JLong				X								I [9]
JFloat					X							I [9]
JDouble						X						I [9]
JBoolean							X					I [9]
JChar								X				I [9]
JString									X			I
JArray										I/X [4]		I
JObject										I/X [6]	I/X [7]	I/X [7]
JClass												I	X
“Boxed”[10]	I	I	I	I	I	I	I					I

[1]	(1, 2, 3, 4, 5, 6) Conversion will occur if the Python value fits in the Java native type.

[2]	(1, 2) Conversion occurs if the Python string or unicode is of length 1.

[3]	(1, 2, 3, 4) Java defines conversions from integer types to floating point types as implicit conversion. Java’s conversion rules are based on the range and can be lossy. See (http://stackoverflow.com/questions/11908429/java-allows-implicit-conversion-of-int-to-float-why)

[4]	Number of dimensions must match, and the types must be compatible.

[6]	Only if the specified type is an compatible array class.

[7]	(1, 2) Exact is the object class is an exact match, otherwise implicit.

[8]	Only the values True and False are implicitly converted to booleans.

[9]	(1, 2, 3, 4, 5, 6, 7, 8) Primitives are boxed as per Java rules.

[10]	Java boxed types are mapped to python primitives, but will produce an implicit conversion even if the python type is an exact match. This is to allow for resolution between methods that take both a java primitve and a java boxed type.

[11]	(1, 2) Boxed to `java.lang.Long` as there is no difference between long and int in Python3,

[12]	Boxed to `java.lang.Double`

Converting from Java to Python¶

The rules here are much simpler.

Java byte, short and int are converted to Python int.
Java long is converted to Python long.
Java float and double are converted to Python float.
Java boolean is converted to Python int of value 1 or 0.
Java char is converted to Python unicode of length 1.
All Java objects are converted to JObject.
Java Throwable is converted to JException derived from JObject.
Java String is converted to JString derived from JObject.
Java arrays are converted to JArray derived from JObject.
Java boxed types are converted to JObject with extensions of python primitives on return.

Casting¶

The main problem with exposing Java classes and methods to Python, is that Java allows overloading a method. That is, multiple methods can have the same name as long as they have different parameters. Python does not allow that. Most of the time, this is not a problem. Most overloaded methods have very different parameters and no confusion takes place.

When JPype is unable to decide with overload of a method to call, the user must resolve the ambiguity. That’s where the wrapper classes come in.

Take for example the java.io.PrintStream class. This class has a variant of the print and println methods!

So for the following code:

from jpype import *
startJVM(getDefaultJVMPath(), "-ea")
java.lang.System.out.println(1)
shutdownJVM()

JPype will automatically choose the println(int) method, because the Python int matches exactly with the Java int, while all the other integral types are only “implicit” matches. However, if that is not the version you wanted to call …

Changing the line thus:

from jpype import *
startJVM(getDefaultJVMPath(), "-ea")
java.lang.System.out.println(JByte(1)) # <--- wrap the 1 in a JByte
shutdownJVM()

tells JPype to choose the byte version.

Note that wrapped object will only match to a method which takes EXACTLY that type, even if the type is compatible. Using a JByte wrapper to call a method requiring an int will fail.

One other area where wrappers help is performance. Native types convert quite fast, but strings, and later tuples, maps, etc … conversions can be very costly. If you’re going to make many Java calls with a complex object, wrapping it once and then using the wrapper will make a huge difference.

Casting using the Java types is also usedful when placing objects in generic containers such as Java List or Map. Both primitive and boxed type Java object derive from the corresponding Python type, so they will work with any Python call.

Boxed types¶

Both python primitives and Boxed types are immutable. Thus boxed types are inherited from the python primitives. This means that a boxed type regardless of whether produced as a return or created explicitely are treated as python types. They will obey all the conversion rules corresponding to a python type as implicit matches. In addition, they will produce an exact match with their corresponding java type. The type conversion for this is somewhat looser than java. While java provides automatic unboxing of a Integer to a double primitive, jpype can implicitly convert Integer to a Double boxed.

To box a primitive into a specific type such as to place in on a java.util.List use JObject on the desired boxed type. For example:

from jpype.types import *
from jpype import java
# ...
lst = java.util.ArrayList()
lst.add(JObject(JInt(1)))
print(type(lst.get(0)))

Implementing interfaces¶

At times it is necessary to implement an interface in python especially to use classes that require java lambdas. To implement an interface contruct a python class and decorate it with annotations @JImplements and @JOverride.

from jpype import JImplements, JOverride
from java.lang.util import DoubleUnaryOperator
# ...
@JImplements(DoubleUnaryOperator)
class MyImpl(object):
    @JOverride
    def applyAsDouble(self, value):
        return 123+value

The java interface may specified by a java wrapper or using a string naming the class. Multiple interfaces can be implemented by a single class by giving a list of interfaces. Alternatively, the interface can be implemented using JProxy.

In a future release, Python callables will be able to automatically match to interfaces that have the Java annotation @FunctionalInterface.

JProxy¶

The JProxy allows Python code to “implement” any number of Java interfaces, so as to receive callbacks through them.

Using JProxy is simple. The constructor takes 2 arguments. The first is one or a sequence of string of JClass objects, defining the interfaces to be “implemented”. The second must be a keyword argument, and be either dict or inst. If dict is specified, then the 2nd argument must be a dictionary, with the keys the method names as defined in the interface(s), and the values callable objects. If inst an object instance must be given, with methods defined for the methods declared in the interface(s). Either way, when Java calls the interface method, the corresponding Python callable is looked up and called.

Of course, this is not the same as subclassing Java classes in Python. However, most Java APIs are built so that subclassing is not needed. Good examples of this are AWT and SWING. Except for relatively advanced features, it is possible to build complete UIs without creating a single subclass.

For those cases where subclassing is absolutely necessary (i.e. using Java’s SAXP classes), it is generally easy to create an interface and a simple subclass that delegates the calls to that interface.

Sample code :¶

Assume a Java interface like:

public interface ITestInterface2
{
        int testMethod();
        String testMethod2();
}

You can create a proxy implementing this interface in 2 ways. First, with a class:

class C :
        def testMethod(self) :
                return 42

        def testMethod2(self) :
                return "Bar"

c = C()
proxy = JProxy("ITestInterface2", inst=c)

or you can do it with a dictionary

def _testMethod() :
return 32

def _testMethod2() :
return "Fooo!"

d = {
      'testMethod' : _testMethod,
      'testMethod2' : _testMethod2,
}
proxy = JProxy("ITestInterface2", dict=d)

Java Exceptions¶

Error handling is an important part of any non-trivial program. All Java exceptions occurring within java code raise a jpype.JException which derives from python Exception. These can be caught either using a specific java exception or generically as a jpype.JException or java.lang.Throwable. You can then use the stacktrace(), str(), and args to access extended information.

Here is an example:

try :
        # Code that throws a java.lang.RuntimeException
except java.lang.RuntimeException as ex:
      print("Caught the runtime exception : ", str(ex))
      print(ex.stacktrace())

Multiple java exceptions can be caught together or separately:

try:
      #  ...
except (java.lang.ClassCastException, java.lang.NullPointerException) as ex:
      print("Caught multiple exceptions : ", str(ex))
      print(ex.stacktrace())
except java.lang.RuntimeException as ex:
      print("Caught runtime exception : ", str(ex))
      print(ex.stacktrace())
except jpype.JException:
      print("Caught base exception : ", str(ex))
      print(ex.stacktrace())
except Exception as ex:
      print("Caught python exception :", str(ex))

Exceptions can be raised in proxies to throw an exception back to java.

Exceptions within the jpype core are issued with the most appropriate python exception type such as TypeError, ValueError, AttributeError, or OSError.

Using jpype.JException with a class name as a string was supported in previous JPype versions but is currently deprecated.

Customizers¶

Java wrappers can be customized to better match the expected behavior in python. Customizers are defined using annotations. Currently the annotations @JImplementionFor and @JOverride can be applied to a regular class to customize an existing class. @JImplementationFor requires the class name as a string so that it can be applied to the class before the JVM is started. @JOverride can be applied method to hide the java implementation allowing a python functionality to be placed into method. If a java method is overridden it is renamed with an proceeding underscore to appear as a private method. Optional arguments to @JOverride can be used to control the renaminging and force the method override to apply to all classes that derive from a base class (“sticky”).

Generally speaking, a customizer should be defined before the first instance of a given class is created so that the class wrapper and all instances will have the customization.

Example taken from JPype java.util.Collection customizer:

@JImplementationFor("java.util.Collection")
class _JCollection(object):

    # Support of len(obj)
    def __len__(self):
        return self.size()

    def __delitem__(self, i):
        return self.remove(i)

    # addAll does not automatically convert to
    # a Collection, so we can augment that
    # behavior here.
    @JOverride(sticky=True)
    def addAll(self, v):
        if isPythonSequence(v):
            r = False
            for i in v:
                r = self.add(i) or r
            return r
        else:
            return self._addAll(v)

The name of the class does not matter for the purposes of customizer though it should probabily be a private class so that it does not get used accidentally. The customizer code will steal from the prototype class rather than acting as a base class, thus ensuring that the methods will appear on the most derived python class and are not hidden by the java implementations. The customizer will copy methods, callable objects, __new__, class member strings, and properties.

Known limitations¶

This section lists those limitations that are unlikely to change, as they come from external sources.

Restarting the JVM¶

JPype caches many resources to the JVM. Those resource are still allocated after the JVM is shutdown as there are still Python objects that point to those resources. If the JVM is restarted, those stale Python objects will be in a broken state and the new JVM instance will obtain the references to these resulting in a memory leak. Thus it is not possible to start the JVM after it has been shutdown with the current implementation.

Running multiple JVM¶

JPype uses the Python global import module dictionary, a global Python to Java class map, and global JNI typemanager map. These resources are all tied to the JVM that is started or attached. Thus operating more than one JVM does not appear to be possible under the current implementation. Difficulties that would need to be overcome to remove this limitation include:

Which JVM would a static class method call. Thus the class types would need to be JVM specific (ie. JClass('org.MyObject', jvm=JVM1))
How would can a wrapper for two different JVM coexist in the jpype._jclass module with the same name if different class is required for each JVM.
How would the user specify which JVM a class resource is created in when importing a module.
How would objects in one JVM be passed to another.
How can boxed and String types hold which JVM they will box to on type conversion.

Thus it appears prohibitive to support multiple JVMs in the JPype class model.

Unloading the JVM¶

The JNI API defines a method called destroyJVM(). However, this method does not work. That is, Sun’s JVMs do not allow unloading. For this reason, after calling shutdownJVM(), if you attempt calling startJVM() again you will get a non-specific exception. There is nothing wrong (that I can see) in JPype. So if Sun gets around to supporting its own properly, or if you use JPype with a non-SUN JVM that does (I believe IBM’s JVMs support JNI invocation, but I do not know if their destroyJVM works properly), JPype will be able to take advantage of it. As the time of writing, the latest stable Sun JVM was 1.4.2_04.

Methods dependent on “current” class¶

There are a few methods in the Java libraries that rely on finding information on the calling class. So these methods, if called directly from Python code, will fail because there is no calling Java class, and the JNI API does not provide methods to simulate one.

At the moment, the methods known to fail are :

java.sql.DriverManager.getConnection(…)¶

For some reason, this class verifies that the driver class as loaded in the “current” classloader is the same as previously registered. Since there is no “current” classloader, it defaults to the internal classloader, which typically does not find the driver. To remedy, simply instantiate the driver yourself and call its connect(...) method.

Unsupported Python versions¶

PyPy 2.7 has issues with the Python meta class programming. PyPy 3 appears to work, but does not have very aggressive memory deallocation. Thus PyPy 3 fails the leak test.

Unsupported Java virtual machines¶

The open JVM implementations Cacao and JamVM are known not to work with JPype.

Cygwin¶

Cygwin is currently usable in JPype, but has a number of issues for which there is no current solution. The python2 compilation on cygwin has bugs in a threading implementation that lead to crashes in the test bench. Cygwin does not appear to pass environment variables to the JVM properly resulting in unusual behavior with certain windows calls. The path separator for Cygwin does not match that of the Java dll, thus specification of class paths must account for this. Subject to these issues JPype is usable.

PyPy¶

The GC routine in PyPy does not play well with Java. It runs when it thinks that Python is running out of resources. Thus a code that allocates a lot of Java memory and deletes the Python objects will still be holding the Java memory until Python is garbage collected. This means that out of memory failures can be issued during heavy operation.

Advanced Topics¶

Using JPype for debugging Java code¶

One common use of JPype is not to develop programs in Python, but rather to function as a Read-Eval-Print Loop for Java. When operating Java though Python as a method of developing or debugging Java there are a few tricks that can be used to simplify the job, beyond being able to probe and plot the Java data structures interactively. These methods include:

Attaching a debugger to the Java JVM being run under JPype.
Attaching debugging information to a Java exception.
Serializing the state of a Java process to be evaluated at a later point.

We will briefly discuss each of these methods.

Attaching a Debugger¶

Interacting with Java through a shell is great, but sometimes it is necessary to drop down to a debugger. To make this happen we need to start the JVM with options to support remote debugging.

jpype.startJVM("-Xint", "-Xdebug", "-Xnoagent",
  "-Xrunjdwp:transport=dt_socket,server=y,address=12999,suspend=n")

Then add a marker in your program when it is time to attach the debugger in the form of a pause statement.

input("pause to attach debugger")
myobj.callProblematicMethod()

When Python reaches that point during execution, switch to a Java IDE such as Netbeans and select Debug : Attach Debugger. That brings up a window (see example below). After attaching (and setting desired break points) go back to Python and hit enter to continue. Netbeans should come to the foreground when a breakpoint is hit.

Attach data to an Exception¶

Sometimes getting to the level of a debugger is challenging especially if the code is large and error occurs rarely. In this case it is often benefitial to simply attach data to an exception. To do this, we need to write a small utility class. Java exceptions are not strictly speaking expandable, but they can be chained. Thus, it we create a dummy exception holding a java.util.Map and attach it to as the cause of the exception, it will be passed back down the call stack until it reaches Python. We can then use getCause() to retrieve the map containing the relevant data.

Capturing the state¶

If the program is not running in an interactive shell or the program run time is long, we may not want to deal with the problem during execution. In this case, we can serialize the state of the relevant classes and variables. To use this option, we simply make sure all of the classes in Java that we are using are Serializable, then add a condition that detects the faulty algorithm state. When the fault occurs, we create a java.util.HashMap and populate it with the values we wish to be able to examine from within Python. We then use Java serialization to write this state file to disk. We then execute the program and collect the resulting state files.

We can then return later with an interactive Python shell, and launch JPype with a classpath for the jars and possibly a connection to debugger. We load the state file into memory and we can then probe or execute the methods that lead up to the fault.