JPype User Guide

JPype Introduction

JPype is a Python module to provide full access to Java from within Python. Unlike Jython, JPype does not achive this by re-implementing Python, but instead by interfacing both virtual machines at the native level. This shared memory based approach achieves good computing performance, while providing the access to the entirety of CPython and Java libraries. This approach allows direct memory access between the two machines, implementation of Java interfaces in Python, and even use of Java threading.

JPype Use Cases

Here are three typical reasons to use JPype.

  • Access to a Java library from a Python program (Python oriented)
  • Visualization of Java data structures (Java oriented)
  • Interactive Java and Python development including scientific and mathematical programming.

Let’s explore each of these options.

Case 1: Access to a Java library

Suppose you are a hard core Python programmer. You can easily use lambdas, threading, dictionary hacking, monkey patching, been there, done that. You are hard at work on your latest project but you just need to pip in the database driver for your customers database and you can call it a night. Unfortunately, it appears that your customers database will not connect to the Python database API. The whole thing is custom and the customer isn’t going to supply you with a Python version. They did sent you a Java driver for the database but fat lot of good that will do for you.

Stumbling through the internet you find a module that says it can natively load Java packages as Python modules. Well, it worth a shot…

So first thing the guide says is that you need to install Java and set up a JAVA_HOME environment variable pointing to the JRE. Then start the JVM with classpath pointed to customers jar file. The customer sent over an example in Java so you just have to port it into Python.

package com.paying.customer;

import com.paying.customer.DataBase

public class MyExample {
   public void main(String[] args) {
     Database db = new Database("our_records");
     try (DatabaseConnection c = db.connect())
     {
        c.runQuery();
        while (c.hasRecords())
        {
          Record record = db.nextRecord();
          ...
        }
     }
  }
}

It does not look too horrible to translate. You just need to look past all those pointless type declarations and meaningless braces. Once you do, you can glue this into Python and get back to what you really love, like performing dictionary comprehensions on multiple keys.

You glance over the JPype quick start guide. It has a few useful patterns… set the class path, start the JVM, remove all the type declarations, and you are done.

# Boiler plate stuff to start the module
import jpype
import jpype.imports
from jpype.types import *

# Launch the JVM
jpype.startJVM(classpath=['jars/database.jar'])

# import the Java modules
from com.paying.customer import DataBase

# Copy in the patterns from the guide to replace the example code
db = Database("our_records")
with  db.connect() as DatabaseConnection:
   c.runQuery()
   while c.hasRecords():
       record = db.nextRecord()
       ...

Launch it in the interactive window. You can get back to programming in Python once you get a good night sleep.

Case 2: Visualization of Java structures

Suppose you are a hard core Java programmer. Weakly typed languages are for wimps, if it isn’t garbage collected it is garbage. Unfortunately your latest project has suffered a nasty data structure problem in one of the threads. You managed to capture the data structure in a serialized form but if you could just make graph and call a few functions this would be so much easier. But the interactive Java shell that you are using doesn’t really have much in the way of visualization and your don’t have time to write a whole graphing applet just to display this dataset.

So poking around on the internet you find that Python has exactly the visualization that you need for the problem, but it only runs in CPython. So in order to visualize the structure, you need to get it into Python, extract the data structures and, send it to the plotting routine.

You install conda, follow the install instructions to connect to conda-forge, pull JPype1, and launch the first Python interactive environment that appear to produce a plot.

You get the shell open and paste in the boilerplate start commands, and load in your serialized object.

import jpype
import jpype.imports

jpype.startJVM(classpath = ['jars/*', 'test/classes'])

from java.nio.file import Files, Paths
from java.io import ObjectInputStream

with Files.newInputStream(Paths.get("myobject.ser") as stream:
   ois = new ObjectInputStream(stream)
   obj = ois.readObject()

print(obj) # prints org.bigstuff.MyObject@7382f612

It appears that the structure is loaded. The problematic structure requires you call the getData method with the correct index.

d = obj.getData(1)

> TypeError: No matching overloads found for org.bigstuff.MyObject.getData(int),
> options are:
       public double[] org.bigstuff.MyObject.getData(double)
       public double[] org.bigstuff.MyObject.getData(int)

Looks like you are going to have to pick the right overload as it can’t figure out which overload to use. Darn weakly typed language, how to get the right type in so that you can plot the right data. It says that you can use the casting operators.

from jpype.types import *
d = obj.getData(JInt(1))
print(type(d))  # prints double[]

Great. Now you just need to figure out how to convert from a Java array into our something our visualization code can deal with. As nothing indicates that you need to convert the array, you just copy out of the visualization tool example and watch what happens.

import matplot.pyplot as plt
plt.plot(d)
plt.show()

A graph appears on the screen. Meaning that NumPy has not issue dealing with Java arrays. It looks like ever 4th element in the array is zero. It must be the PR the new guy put in. And off you go back to the wonderful world of Java back to the safety of curly braces and semicolons.

Case 3: Interactive Java

Suppose you are a laboratory intern running experiments at Hawkins National Laboratory. (For the purpose of this exercise we will ignore the fact that Hawkins was shut down in 1984 and Java was created in 1995). You have the test subject strapped in and you just need to start the experiment. So you pull up Jupyter notebook your boss gave you and run through the cells. You need to added some heart wave monitor to the list of graphed results.

The relevant section of the API for the Experiment appears to be

package gov.hnl.experiment;

public interface Monitor {
   public void onMeasurement(Measurement measurement);
}

public interface Measurement {
   public double getTime();
   public double getHeartRate();
   public double getBrainActivity();
   public double getDrugFlowRate();
   public boolean isNoseBleeding();
}

public class Experiment {
   public void addCondition(Instant t, Condition c);
   public void addMoniter(Monitor m);
   public void run();
}

The notebook already has all the test conditions for the experiment set up and the JVM is started, so you just need to implement the monitor.

Based on the previous examples, you start by defining a monitor class

from jpype import JImplements, JOverride
from gov.hnl.experiment import Monitor

@JImplements(Monitor)
class HeartMonitor:
   def __init__(self):
      self.readings = []
   @JOverride
   def onMeasurement(self, measurement):
      self.readings.append([measurement.getTime(), measurement.getHeartRate()])
   def getResults(self):
      return np.array(self.readings)

There is a bit to unpack here. You have implemented a Java class from within Python. The Java implementation is simply an ordinary Python class which has be decorated with @JImplements and @JOverride. When you forgot to place the @JOverride, it gave you the response:

NotImplementedError: Interface 'gov.hnl.experiment.Monitor' requires
method 'onMeasurement' to be implemented.

But once you added the @JOverride, it worked properly. The subject appears to be getting impatient so you hurry up and set up a short run to make sure it is working.

hm = HeartMonitor()
experiment.addMonitor(hm)
experiment.run()
readings = hm.getResults()
plt.plot(readings[:,0], readings[:,1)
plt.show()

To your surprise, it says unable to find method addMonitor with an error message:

AttributeError: 'gov.hnl.experiment.Experiment' object has no attribute 'addMonitor'

You open the cell and type experiment.add<TAB>. The line completes with experiment.addMoniter. Whoops, looks like there is typo in the interface. You make a quick correction and see a nice plot of the last 30 seconds pop up in a window. Job well done, so you set the runtime back to one hour. Looks like you still have time to make the intern woodlands hike and forest picnic. Though you wonder if maybe next year you should sign up for another laboratory. Maybe next year, you will try to sign up for those orbital lasers the President was talking about in the March. That sounds like real fun.

(This advanced demonstration utilized the concept of Proxies and Code completion)

The JPype Philosophy

JPype is designed to allow the user to exercise Java as fluidly as possible from within Python. We can break this down into a few specific design goals.

  • Make Java appear Pythonic. Make it so a Python programmer feels comfortable making use of Java concepts. This means making use of Python concepts to create very Python looking code and at times bending Python concepts to conform to Java’s expectations.
  • Make Python appear like Java. Present concepts from Java with a syntax that resembles Java so that Java users can work with Python without a huge learning curve.
  • Present everything that Java has to offer to Python. Every library, package, and Java feature if possible should be accessible. The goal of bridge is to open up places and not to restrict flow.
  • Keep the design as simple as possible. Mixing languages is already complex enough so don’t required the user to learn a huge arsenal of unique methods. Instead keep it simple with well defined rules and reuse these concepts. For example, all array types originate from JArray, and thus using one can also use isinstance to check if a class is an array type. Rather than introducing factory that does a similar job to an existing one, instead use a keyword argument on the current factory.
  • Favor clarity over performance. This doesn’t mean not trying to optimize paths, but just as premature optimization is the bane of programmers, requiring writing to maximize speed is a poor long term choice, especially in a language such as Python were weak typing can promote bit rot.
  • If a new method has to be introduced, make look familiar. Java programmers look to a method named “of” to convert to a type on factories such as a Stream, thus JArray.of converts a Python NumPy array to Java. Python programmers expect that memory backed objects can be converted into bytes for rapid transfer using a memory view, thus memoryview(array) will perform that task.
  • Provide an obvious way for both Python and Java programmers to perform tasks. On this front JPype and Python disagree. In Python’s philosophy there should be one – and preferably only one – obvious way to do things. But we are bridging two worlds and thus obviousness is in the eye of the beholder.

The end result is that JPype has a small footprint while providing access to Java (and other JVM based languages) with a minimum of effort.

Languages other than Java

JPype is primarily focused on providing the best possible wrapper for Java in Python. However, the Java Virtual Machine (JVM) is used for many popular languages such a Kotlin and Scala. As such JPype can be used for any language which used the JVM.

That said each language has its own special properties that tend to be represented in different ways. If you would like JPype fully to operate on your particular language the following is required.

  • Set up a test bench for your language under the test directory. Use ivy to pull in the required jar files required to run it and exercise each of the required language features that need to be exercised.
  • Write a language specific quick start guide for your language defining how things should appear in both your language of choice and within Python highlighting those things that are different from how Java.
  • Set up a test harness that exercises your language for each language feature and place a setup script like test_java that builds the harness.

Alternatives

JPype is not the only Python module of its kind that acts as a bridge to Java. Depending on your programming requirements, one of the alternatives may be a better fit. Specifically JPype is designed for clarity and high levels of integration between the Python and Java virtual machine. As such it makes use of JNI and thus inherits all of the benefits and limitations that JNI imposes. With JPype, both virtual machines are running in the same process and are sharing the same memory space and threads. JPype can thus intermingle Python and Java threads and exchange memory quickly. But by extension you can’t start and stop the JVM machine but instead must keep both machines throughout the lifespan of the program. High integration means tightly coupled and thus it embodies the musketeers motto. If Python crashes, so does Java as they only have one process to live in.

A few alternatives with different philosophies and limitations are given in the following section. Please take my review comments with the appropriate grain of salt. When I was tasked with finding a replacement for Matlab Java integration for our project test bench, I evaluated a number of alternatives Python bridge codes. I selected JPype primarily because it presented the most integrated API and documentation which would be suitable for getting physicists up to speed quickly. Thus your criteria may yield a different selection. Its underlying technology was underwhelming and thus I have had the pleasure of many hours reworking stuff under the hood.

For more details on what you can’t do with JPype, please see Limitations.

Jython

Jython is a reimplementation of Python in Java. As a result it has much lower costs to share data structures between Java and Python and potentially much higher level of integration. Noted downsides of Jython are that it has lagged well behind the state of the art in Python; it has a limited selection of modules that can be used; and the Python object thrashing is not particularly well fit in Java virtual machine leading to some known performance issues.

Py4J

Py4J uses a remote tunnel to operate the JVM. This has the advantage that the remote JVM does not share the same memory space and multiple JVMs can be controlled. It provides a fairly general API, but the overall integration to Python is as one would expect when operating a remote channel operating more like an RPC front-end. It seems well documented and capable. Although I haven’t done benchmarking, a remote access JVM will have a transfer penalty when moving data.

Jep

Jep stands for Java embedded Python. It is a mirror image of JPype. Rather that focusing on accessing Java from within Python, this project is geared towards allowing Java to access Python as sub-interpreter. The syntax for accessing Java resources from within the embedded Python is quite similar with support for imports. Notable downsides are that although Python supports multiple interpreters many Python modules do not, thus some of the advantages of the use of Python many be hard to realize. In addition, the documentation is a bit underwhelming thus it is difficult to see how capable it is from the limited examples.

Javabridge

Javabridge is direct low level JNI control from Python. The integration level is quite low on this, but it does serve the purpose of providing the JNI API to Python rather than attempting to wrap Java in a Python skin. The downside being of course you would really have to know a lot of JNI to make effective use of it.

jpy

This is the most similar package to JPype in terms of project goals. They have achieved more capabilities in terms of a Java from Python than JPype which does not support any reverse capabilities. It is currently unclear if this project is still active as the most recent release is dated 2014. The integration level with Python is fairly low currently though what they do provide is a similar API to JPype.

About this guide

The JPype User Guide is targeted toward programmers who are strong in either Python who wish to make use of Java or those who are strong with Java and are looking to use Python as a Java development tool. As such we will compare and contrast the differences between the languages and provide examples suitable to help illustrate how to translate from one language to the other on the assumption that being strong in one language will allow you to easily grasp the corresponding relations in the other. If you don’t have a strong background in either language an appropriate language tutorial may be necessary.

JPype will hide virtually all of the JNI layer such that there is no direct access to JNI concepts. As such attempting to use JNI knowledge will likely lead to incorrect assumptions such as incorrectly attempting to use JNI naming and method signatures in the JPype API. Where JNI limitations do appear we will discuss the consequences imposed in programming. No knowledge of JNI is required to use this guide or JPype.

JPype only works with Python 3, thus all examples will be using Python version 3 syntax and assume the use of the Python 3 new style object model. The naming conventions of JPype follow the Java rules rather than those of Python. This is a deliberate choice as it would be dangerous to try to mangle Java method and field names into Python conventions and risk a name collision. Thus if method must have Java conventions then the rest of the module should follow the same pattern for consistency.

Getting JPype started

This document holds numerous JPype examples. For the purposes of clarity the module is assumed to have been started with the following command

# Import the module
import jpype

# Allow Java modules to be imported
import jpype.imports

# Import all standard Java types into the global scope
from jpype.types import *

# Import each of the decorators into the global scope
from jpype import JImplements, JOverride, JImplementationFor

# Start JVM with Java types on return
jpype.startJVM(convertStrings=False)

# Import default Java packages
import java.lang
import java.util

This is not the only style used by JPype users. Some people feel it is best to limit the number for symbols in the global scope and instead start with a minimalistic approach.

import jpype as jp                 # Import the module
jp.startJVM(convertStrings=False)  # Start the module

Either style is usable and we do not wish to force any particular style on the user. But as the extra jp. tends to just clutter up the space and implies that JPype should always be used as a namespace due to namespace conflicts, we have favored the global import style. JPype only exposes 40 symbols total including a few deprecated functions and classes. The 13 most commonly used Java types are wrapped in a special module jpype.types which can be used to import all for the needed factories and types with a single command without worrying about importing potentially problematic symbols.

We will detail the starting process more later in the guide. See Starting the JVM.

JPype Concepts

At its heart, JPype is about providing a bridge to use Java within Python. Depending on your prospective that can either be a means of accessing Java libraries from within Python or a way to use Java using Python syntax for interactivity and visualization. This mean not only exposing a limited API but instead trying to provide the entirety of the Java language with Python.

To do this, JPype maps each of the Java concepts to the nearest concept in Python wherever they are similar enough to operate without confusion. We have tried to keep this as Pythonic as possible, though it is never without some rough edges.

Python and Java share many of the same concepts. Types, class, objects, function, methods, and members. But in other places they are rather different. Python lacks casting, type declarations, overloading, and many other features of a strongly typed language, thus we must expose those concepts into the Python syntax as best we can. Java for instance has class annotation and Python have class decorators. Both serve the purpose of augmenting a class with further information, but are very different in execution.

We have broken the mapping down in nine distinct concepts. Some elements serve multiple functions.

Type Factories
These are meta classes that allow one to declare a particular Java type in Python. The result of type factories are wrapper classes. (JClass and JArray) Factories also exist to implement Java classes from within Python (JProxy)
Meta Classes
These are classes to describe different properties of Java classes such as to check if a class is an Interface. (JInterface)
Base Classes
These are JPype names for Java classes in Python that exist without importing any specific Java class. Concepts such as Object, String, and Exception are defined and can be used in instance checks. For example, to catch all Java exceptions regardless of type, we would catch JException. These are mainly for convenience though they do have some extra functionality. Most of these functions are being phased out in favor of Java syntax. For example, catching java.lang.Throwable will catch everything that JException will catch. (Jarray, JObject, JString, and JException)
Wrapper Classes
These correspond to each Java class. Thus can be used to access static variables, static methods, cast, and construct object. They are used wherever a Java type would be used in the Java syntax such as creating an array or accessing the class instance. These class wrappers are customized in Python to allow a direct mapping from Java concepts to Python one. These are all created dynamically corresponding to each Java class. For most of this document we will refer to these simply as a “class”. (java.lang.Object, java.lang.String, etc) Many wrappers are customized to match Python abstract base classes ABC (java.util.List, java.util.Map)
Object Instances
These are Java objects. They operate just like Python objects with Java public fields mapped to Python attributes and Java methods to Python methods. For this document we will refer to an object instance simply as an “object”. The object instance is split into two halves. The Python portion is referred to as the “handle” that points the Java “instance”. The lifetime of the “instance” is tied to the handle thus Java objects do not disappear until the Python handle is disposed of. Objects can be cast to match the required type and hold methods and fields.
Primitive types
Each of the 8 Java primitive types are defined. These are used to cast to a Java type or to construct arrays. (JBoolean, JChar, JByte, JShort, JInt, JLong, JFloat, and JDouble)
Decorators
Java has a number of keywords such as extending a class or implementing an interface. Those pieces of meta data can’t directly be expressed with the Python syntax, but instead have been been expressed as annotations that can be placed on classes or functions to augment them with Java specific information. (@JImplements, @JOverride, @JImplementationFor)
Mapping Java syntax to Python
Many Java concepts like try with resources can be mapped into Python directly (as the with statement), or Java try, throw, catch mapping to Python try, raise, except. Others such as synchronize do not have an exact Python match. Those have instead been mapped to special functions that interact with Python syntax.. (synchronized, with, try, import)
JVM control functions
The JVM requires specific actions corresponding to JNI functions in order to start, shutdown, and define threading behavior. These top level control functions are held in the jpype module. (startJVM, shutdownJVM)

We will detail each of these concepts in greater detail in the later sections.

Name mangling

When providing Java package, classes, methods, and fields to Python, there are occasionally naming conflicts. For example, if one has a method called with then it would conflict with the Python keyword with. Wherever this occurs, JPype renames the offending symbol with a trailing under bar. Java symbols with a leading or trailing under bars are consider to be privates and may not appear in the JPype wrapper entirely with the exception of package names.

The following Python words will trigger name mangling of a Java name:

False None True and as
async await def del elif
except exec from global in
is lambda nonlocal not or
pass print raise with yield

JPype Types

Both Java and Python have a concept of a type. Every variable refers to an object which has a defined type. A type defines the data that the variable is currently holding and how that variable can be used. In this chapter we will learn how Java and Python types relate to one another, how to create import types from Java, and how to use types to create Java objects.

Stay strong in a weak language

Before we get into the details of the types that JPype provides, we first need to contrast some of the fundamental language differences between Java and Python. Python is inherently a weakly typed language. Any variable can take any type and the type of a particular variable can change over the lifetime of a program. Types themselves can be mutable as you can patch an existing type to add new behaviors. Python methods can in principle take any type of object as an argument, however if the interface is limited it will produce a TypeError to indicate a particular argument requires a specific type. Python objects and classes are open. Each class and object is basically a dictionary storing a set of key value pairs. Types implemented in native C are often more closed and thus can’t have their method dictionaries or data members altered arbitrarily. But subject to a few restrictions based implementation, it is pretty much the wild west.

In contrast, Java is a strongly typed language. Each variable can only take a value of the specified class or a class that derives from the specified class. Each Java method takes only a specific number and type of arguments. The type and number are all checked at compile type to ensure there is little possibility of error. As each method requires a specific number and type of arguments, a method can be overloaded by having two different implementations which take a different list of types sharing the same method name. A primitive variable can never hold an object and it can only be converted to or from other primitive types unless it is specifically cast to that type. Java objects and classes are completely closed. The methods and fields for a particular class and object are defined entirely at compile time. Though it is possible create classes with a dictionary allowing expansion, this is not the Java norm and no standard mechanism exists.

Thus we need to introduce a few Java terms to the Python vocabulary. These are “conversion” and “cast”.

Java conversions

A conversion is a permitted change from an object of one type to another. Conversions have three different degrees. These are: exact, implicit, and explicit.

Exact conversions are those in which the type of an object is identical. In Java each class has only one definition thus there is no need for an exact conversion. But when dealing with Python we have objects that are effectively identical for which exact conversion rules apply. For example, a Java string and a Python string both bind equally well to a method which requires a string, thus this is an exact conversion for the purposes of bind types.

The next level of conversion is implicit. An implicit conversion is one that Java would perform automatically. For example converting a derived class to is base class when setting a field would be an implicit conversion. Java defines a number of other conversions such as converting a primitive to a boxed type or from a boxed type back to a primitive as implicit conversions..

Of course not every cast is safe to perform. For example, converting an object whose type is currently viewed as a base type to a derived type is not performed automatically nor is converting from one boxed type to another. For those operations the conversion must be explicitly requested, hence these are explicit conversions. To request an explicit conversion an object must be “cast” using a cast operator. In Java, a cast is requested by placing the type name in parentheses in front of the object to be cast. Unfortunately, the same syntax is not allowed in Python. Not every conversion is possible between Java types. Types that cannot be converted are considerer to be conversion type “none”.

Details on the standard conversions provided by JPype are given in the section Type Matching.

Java casting

To access a casting operation we use the casting JObject wrapper. JObject accepts two arguments. The first argument is the object to convert and the second is the type to cast to. The second argument should always be a Java type specified using a class wrapper, a Java class instance, or a string. Casting will also add a hidden class argument to the resulting object such that it is treated as the cast type for the duration of that variable lifespan. Therefore, a variable create by casting is stuck as that type and cannot revert back to its original for the purposes of method resolution.

The object construction and casting are sometimes a bit blurry. For example, when one casts a sequence to a Java list, we will end up constructing a new Java list that contains the elements of the original Python sequence. In general JPype constructors only provide access the Java constructor methods that are defined in the Java documentation. Casting on the other hand is entirely the domain of whatever JPype has defined including user defined casts.

Casting is performed through the Python class JObject. JObject is called with two arguments which are the object to be cast and the type to cast too. The cast first consults the conversion table to decide if the cast it permitted and produces a TypeError if the conversion is not possible.

JObject also serves as a abstract base class for testing if an object instance belongs to Java. All objects that belong to Java will return true when tested with isinstance. Like Python’s sequence, JObject is an abstract base class. No classes actual derive from JObject.

Of particular interest is the concept of Java null. In Java, null is a typeless entity which can be placed wherever an object is taken to indicate that the object is not available. The equivalent concept in Python is None. Thus all methods that accept any object type that permit a null will accept None as an augment with implicit conversion. However, sometime it is necessary to pass an explicit type to the method resolution. To achieve this in JPype use JObject(None, type) which will create a null pointer with the desired type. To test if something is null we have to compare the handle to None. This unfortunately trips up some code quality checkers. The idiom in Python is obj is None, but as this only matches things that Python considers identical, we must instead use obj==None.

Type enforcement appears in three different places within JPype. These are whenever a Java method is called, whenever a Java field is set, and whenever Python returns a value back to Java.

Method resolution

Because Java supports method overloading and Python does not, JPype wraps Java methods as a “method dispatch”. The dispatch is a collection of all of the methods from class and all of its parents which share the same name. The job of the dispatch is chose the method to call.

Enforcement of the strong typing of Java must be performed at runtime within Python. Each time a method is invoked, JPype must match against the list of all possible methods that the class implements and choose the best possible overload. For this reason the methods that appear in a JPype class will not be the actual Java methods, but rather a “dispatch” whose job is deciding which method should be called based on the type of the provided arguments.

If no method is found that matches the provided arguments, the method dispatch will produce a TypeError. This is the exact same outcome that Python uses when enforcing type safety within a function. If a type doesn’t match a TypeError will be produced.

Dispatch example

When JPype is unable to decide which overload of a method to call, the user must resolve the ambiguity. This is where casting comes in.

Take for example the java.io.PrintStream class. This class has a variant of the print and println methods!

So for the following code:

java.lang.System.out.println(1)

JPype will automatically choose the println(long) method, because the Python int matches exactly with the Java long, while all the other numerical types are only “implicit” matches. However, if that is not the version you wanted to call you must cast it. In this case we will use a primitive type to construct the correct type.

Changing the line thus:

java.lang.System.out.println(JByte(1)) # <--- wrap the 1 in a JByte

This tells JPype to choose the byte version. When dealing with Java types, JPype follows the standard Java matching rules. Types can implicitly grow to larger types but will not shrink without an explicit cast.

Primitive Types

Unlike Python, Java makes a distinction between objects and primitive data types. Primitives represent the minimum data that can be manipulated by a computer. These stand in contrast to objects which have the ability to contain any combination of data types and object within themselves, and can be inherited from.

Java primitives come in three flavors. The logical primitive boolean can only take the logical value true and false. The textual primitive char represents one character in a string. Numerical primitives are intended for fixed point or floating point calculations. Numerical primitives come in many sizes depending on how much storage is required. In Java, integer numerical primitives are always signed and thus can only reach half their range in terms of bits up or down relative to their storage size.

JPype has mapped each of the primitive types into Python classes. To avoid conflicts with Python, JPype has named each primitive with a capital letter J followed by the primitive name starting with an upper case letter.

JBoolean
A boolean is the logical primitive as it can only take values True and False. It should properly be an extension of the Python concept bool but that type is not extendable. Thus instead it must inherit from int. This type is rarely seen in JPype as the values True and False are considered an exact match to JBoolean argument. Methods which return a JBoolean will always return a Python bool rather than a Java primitive type.
JChar
A character is the textual primitive that corresponds to exactly one character in a string. Or at least that was the concept at the time. Java characters can only represent 16 bits. But there are currently 143,924 defined characters in Unicode. Thus, there are certain characters that can only be represented as two Unicode characters. The textual primitives are not intended to perform numerical functions, but are instead encoded. As per the old joke, what does 1 plus 1 equal? Which of course the correct answer is b. As such characters should not be treated as just another unsigned short. Python has no concept of a textual only type. Thus when returning a character type, we instead return a string length 1. JChar supports the Java numerical operations, but just as in Java it will automatically promote to a Python int when used in a numerical operation. There are of course lots of useful mathematical operations that can be performed on textual primitives, but doing so risks breaking the encoding and can result in uninterpretable data.
JByte, Short, Int, Long
These types represent fixed point quantities with ranges of 8, 16, 32, and 64 bits. Each of these type inherit from a Python int type. A method or field returning an integer primitive will return a type derived from int. Methods accepting an integer primitive will take either an Java integer primitive or a Python int or anything that quacks like a int so long as it can be converted into that primitive range without truncation.
JFloat, JDouble
These two types hold floating point and correspond to either single point (32 bit) or double point (64 bit) precision. Python does not have a concept of precision and thus both of these derive from the Python type float. As per Java rules numbers greater than the range correspond to the values of positive and negative infinity. Conversions from Python types are ranged check and will produce a OverflowError if the value doesn’t fit into the request types. If an overflow error is not desired, first cast the value into the request size prior to calling. Methods that return a Java floating point primitive will always return a value derived from float.

The classes for Java primitives are closed and should not be extended. As with all Java values any information attached to the Python representation is lost when passing that value to Java.

Objects & Classes

In contrast to primitive data type, objects can hold any combination of primitives or objects. Thus they represent structured data. Objects can also hold methods which operate on that data. Objects can inherit from one another.

However unlike Python, Java objects must have a fixed structure which defines its type. These are referred to the object’s class. Here is a point of confusion. Java has two different class concepts: the class definition and the class instance. When you import a class or refer to a method using the class name you are accessing the class definition. When you call getClass on an object it returns a class instance. The class instance is a object whose structure can be used to access the data and methods that define the class through reflection. The class instance cannot directly access the fields or method within a class but instead provides its own interface for querying the class. For the purposes of this document a “class” will refer to the class definition which corresponds to the Python concept of a class. Wherever the Java reflection class is being referred to we will use the term “class instance”. The term “type” is synonymous with a “class” in Java, though often the term “type” is only used when inclusively discussing the type of primitives and objects, while the term “class” generally refers to just the types associated with objects.

All objects in Java inherit from the same base class java.lang.Object, but Java does not support multiple inheritance. Thus each class can only inherit from a single parent. Multiple inheritance, mix-ins, and diamond pattern are not possible in Java. Instead Java uses the concept of an interface. Any Java class can inherit as many interfaces as it wants, but these interfaces may not contain any data elements. As they do not contain data elements there can be no ambiguity as to what data a particular lookup.

The meta class JInterface is used to check if a class type is an interface using isinstance. Classes that are pure interfaces cannot be instantiated, thus, there is not such thing as an abstract instance. Therefore, every Java object should have Objects cannot actual be pure interfaces. To represent this in Python every interface inherits java.lang.Object methods even through it does not have java.lang.Object as a parent. This ensures that anonymous classes and lambdas have full object behavior.

Classes

In JPype, Java classes are instances of the Python type and function like any ordinary Python class. However unlike Python types, Java classes are closed and cannot be extended. To enforce extension restrictions, all Java classes are created from a special private meta class called _jpype._JClass. This gatekeeper ensures that the attributes of classes cannot be changed accidentally nor extended. The type tree of Java is fixed and closed.

All Java classes have the following functionality.

Class constructor
The class constructor is accessed by using the Python call syntax (). This special method invokes a dispatch whenever the class is called as a function. If an matching constructor is found a new Java instance is created and a Python handle to that instance is returned. In the case of primitive types, the constructor creates a Java value with the exact type requested.
Get attribute
The Python . operator gets an attribute from a class with a specified name. If no method or field exists a AttributeError will be raised. For public static methods, the getattr will produce a Python descriptor which can be called to invoke the static method. For public static fields, a Python descriptor will be produced that allows the field to be get or set depending on whether the field is final or not. Public instance methods and instance fields will produce a function that can be applied to a Java object to execute that method or access the field. Function accessors are non-virtual and thus they can provide access to behaviors that have been hidden by a derived class.
Set attribute
In general, JPype only allows the setting of public non-final fields. If you attempt to set any attribute on an object that does not correspond to a settable field it will produce an AttributeError. There is one exception to this rule. Sometime it is necessary to attach addition private meta data to classes and objects. Attributes that begin with an underbar are consider to be Python private attributes. Private attributes handled by the default Python attribute handler allowing these attributes to be attached to to attach data to the Python handle. This data is invisible to Java and it is retained only on the Python instance. If an object with Python meta data is passed to Java and Java returns the object, the new Python handle will not contain any of the attached data as this data was lost when the object was passed to Java.
class_ Attribute
For Java classes there is a special attribute called class. This is a keyword in Python so name mangling applies. This is a class instance of type java.lang.Class. It can be used to access fields and methods.
Inner classes
For methods and fields, public inner classes appear as attributes of the class. These are regular types that can be used to construct objects, create array, or cast.
String
The Java method toString is mapped into the Python function str(obj).
Equality
The Java method equals() has been mapped to Python == with special augmentations for null pointers. Java == is not exposed directly as it would lead to numerous errors. In principle, Java == should map to the Python concept of is but it is not currently possible to overload Python in such a way to achieve the desired effect.
Hash

The Java method hashCode is mapped to Python hash(obj) function. There are special augmentations for strings and nulls. Strings will return the same hash code as returned by Python so that Java strings and Python strings produce the same dictionary lookups. Null pointers produce the same hash value as None.

Java defines hashCode on many objects including mutable ones. Often the hashCode for a mutable object changes when the object is changed. Only use immutable Java object (String, Instant, Boxed types) as dictionary keys or risk undefined behavior.

Java objects are instances of Java classes and have all of the methods defined in the Java class including static members. However, the get attribute method converts public instance members and fields into descriptors which act on the object.

Now that we have defined the basics of Java objects and classes, we will define a few special classes that operate a bit differently.

Array Classes

In Java all arrays are also objects, but they cannot define any methods beyond a limited set of Java array operations. These operations have been mapped into Python to their closest Python equivalent.

Arrays also have a special type factory to produce them. In principle one can create an array class using JClass but the signature required would need to use the proper name as required for the Java method java.lang.Class.forName. Instead we call the factory to create a new type to use.

The signature for JArray is JArray(type, [dims=1]). The type argument accepts any Java type including primitives and constructs a new array class. This class can be used to create new instances, cast, or as the input to the array factory. The resulting object has a constructor method which take either a number, which is the desired size of the array, or a sequence which hold the elements of the array. If the members of the initializer sequence are not Java members then each will be converted. If any element cannot be converted a TypeError will be raised.

JArray is an abstract base class for all Java classes that are produced. Thus, one can test if something is an array class using issubclass and if Java object is an array using isinstance.

Java arrays provide a few additional Python methods:

Get Item
Arrays are of course a collection of elements. As such array elements can be accessed using the Python [] operator. For multidimensional arrays JPype uses Java style access with a series of index operations such as jarray[4][2] rather than NumPy like multidimensional access.
Get Slice
Arrays can be accessed using a slice like a Python list. The slice operator is [start:stop:step]. It should be noted that array slice are in fact views to the original array so any alteration to the slice will affect the original array. Array slices are cloned when passed back to Java. To force a clone immediately, use the clone method. Please note that applying the slice operator to a slice produces a new slice. Thus there can sometimes be an ambiguity between multidimensional access and repeated slicing.
Set Item
Array items can be set using the Python []= operator.
Set Slice
Multiple array items can be set using a slice assigned with a sequence. The sequence must have the same length as the slice. If this condition is not met, an exception will be raised. If the items to be transferred are a buffer, then a faster buffer transfer assignment will be used. When buffer transfers are used individual elements are not checked for range, but instead cast just like NumPy. Thus, if we have the elements we wish to assign to the array contained within a NumPy array named na we can transfer all of them using jarray[:] = na.
Buffer transfer
Buffer transfers from a Java array also work for primitive types. Thus we can simply call the Python memoryview(jarray) function to create a buffer that can be used to transfer any portion of a Java array out. Memory views of Java arrays are not writable.
For each
Java arrays can be used as the input to a Python for statement. To iterate each element use for elem in jarray:. They can also be used in list comprehensions.
Clone
Java arrays can be duplicated using the method clone. To create a copy call jarray.clone(). This operates both on arrays and slice views.
Length
Arrays in Java have a defined an immutable length. As such the Python len(array) function will produce the array length. However, as that does not match Java expectations, JPype also adds an attribute for length so that Java idiom jarray.length also works as expected.

In addition, the Java class JChar[] has some addition customizations to help work better with string types.

Java arrays are currently missing some of the requirements to act as a collections.abc.Sequence. When working with Java arrays it is also useful to use the Java array utilities class java.util.Arrays as it has many methods that provide additional functionality. Java arrays do not support any additional mathematical operations at this time.

Buffer classes

In addition to array types, JPype also supports Java nio buffer types. Buffers in Java come in two flavors. Array backed buffers have no special access. Direct buffers are can converted to Python buffers with both read and write capabilities.

Each primitive type in Java has its own buffer type named based on the primitive type. java.nio.ByteBuffer has the greatest control allowing any type to be read and written to it. Buffers in Java function are like memory mapped files and have a concept of a read and write pointer which is used to traverse the array. They also have direct index access to their specified primitive type.

Java buffer provide an additional Python method:

Buffer transfer
Buffer transfers from a Java buffer works for a direct buffer. Array backed buffers will raise a BufferError. Use the Python memoryview(jarray) function to create a buffer that can be used to transfer any portion of a Java buffer out. Memory views of Java buffers are readable and writable.

Buffers do not currently support element-wise access.

Boxed Classes

Often one wants to be able to place a Java primitive into a method of fields that only takes an object. The process of creating an object from a primitive is referred to as creating a “boxed” object. The resulting object is an immutable object which stores just that one primitive.

Java boxed types in JPype are wrapped with classes that inherit from Python int and float types as both are immutable in Python. This means that a boxed type regardless of whether produced as a return or created explicitly are treated as Python types. They will obey all the conversion rules corresponding to a Python type as implicit matches.

In addition, they produce an exact match with their corresponding Java type. The type conversion for this is somewhat looser than Java. While Java provides automatic unboxing of a Integer to a double primitive, JPype can implicitly convert Integer to a Double boxed.

To box a primitive into a specific type such as to place it into a java.util.List use JObject on the desired boxed type or call the constructor for the desired boxed type directly. For example:

lst = java.util.ArrayList()
lst.add(JObject(JInt(1)))      # Create a Java integer and box it
lst.add(java.lang.Integer(1))  # Explicitly create the desired boxed object

JPype boxed classes have some additional functionality. As they inherit from a mathematical type in Python they can be used in mathematical operations. But unlike Python numerical types they can take an addition state corresponding to being equal to a null pointer. The Python methods are not aware of this new state and will treat the boxed type as a zero if the value is a null.

To test for null, cast the boxed type to a Python type explicitly and the result will be checked. Casting null pointer will raise a TypeError.

b = JObject(None, java.lang.Integer)
a = b+0      # This succeeds and a gets the value of zero
a = int(b)+0 # This fails and raises a TypeError

Boxed objects have the following additional functionality over a normal object.

Convert to index
Integer boxed types can be used as Python indices for arrays and other indexing tasks. This method checks that the value of the boxed type is not null.
Convert to int
Integer and floating point boxed types can be cast into a Python integer using the int() method. The resulting object is always of type int. Casting a null pointer will raise a TypeError.
Convert to float
Integer and floating point boxed types can be cast into a Python float using the float() method. The resulting object is always of type float. Casting a null pointer will raise a TypeError.
Comparison
Integer and floating point types implement the Python rich comparison API. Comparisons for null pointers only succeed for == and != operations. Non-null boxed types act like ordinary numbers for the purposes of comparison.

Number Class

The Java class java.lang.Number is a special type in Java. All numerical Java primitives and Python number types can convert implicitly into a Java Number.

Input Result
None java.lang.Number(null)
Python int, float java.lang.Number
Java byte, NumPy int8 java.lang.Byte
Java short, NumPy int16 java.lang.Short
Java int, NumPy int32 java.lang.Integer
Java long, NumPy int64 java.lang.Long
Java float, NumPy float32 java.lang.Float
Java double, NumPy float64 java.lang.Double

Additional user defined conversion are also applied. The primitive types boolean and char and their corresponding boxed types are not considered to numbers in Java.

Object Class

Although all classes inherit from Object, the object class itself has special properties that are not inherited. All Java primitives will implicitly convert to their box type when placed in an Object. In addition, a number of Python types implicitly convert to a Java object. To convert to a different object type, explicitly cast the Python object prior to placing in a Java object.

Here a table of the conversions:

Input Result
None java.lang.Object(null)
Python str java.lang.String
Python bool java.lang.Boolean
Python int java.lang.Number
Python float java.lang.Number

In addition it inherits the conversions from java.lang.Number. Additional user defined conversion are also applied.

String Class

The String class in Java is a special representation often pointing either to a dynamically created string or to a constant pool item defined in the class. All Java strings are immutable just like Python strings and thus these are considered to be equivalent classes.

Because Java strings are in fact just pointers to blob of bytes they are actually slightly less than a full object in some JVM implementation. This is a violation of the Object Orients (OO) principle, never take something away by inheritance. Unfortunately, Java is a frequent violator of that rule, so this is just one of those exceptions you have to trip over. Therefore, certain operations such as using a string as a threading control with notify or wait may lead to unexpected results. If you are thinking about using a Java string in synchronized statement then remember it is not a real object.

Java strings have a number of additional functions beyond a normal object.

Length
Java strings have a length measured in the number of characters required to represent the string. Extended Unicode characters count for double for the purpose of counting characters. The string length can be determined using the Python len(str) function.
Indexing
Java strings can be used as a sequence of characters in Python and thus each character can be accessed as using the Python indexing operator [].
Hash
Java strings use a special hash function which matches the Python hash code. This ensures that they will always match the same dictionary keys as the corresponding string in Python. The Python hash can be determined using the Python hash(str) function. Null pointers are not currently handled. To get the actually Java hash, use s.hashCode()
Contains

Java strings implement the concept of in when using the Java method contains. The Java implementation is sufficiently similar that it will work fairly well on strings. For example, "I" in java.lang.String("team") would be equal to False.

Testing other types using the in operator will likely raise a TypeError if Java is unable to convert the other item into something that can be compared with a string.

Concatenation
Java strings can be appended to create a new string which contains the concatenation of the two strings. This is mapped to the Python operator +.
Comparison
Java strings are compared using the Java method compareTo. This method does not currently handle null and will raise an exception.
For each
Java strings are treated as sequences of characters and can be used with a for-loop construct and with list comprehension. To iterate through all of the characters, use the Python construct for c in str:.

Unfortunately, Java strings do not yet implement the complete list of requirements to act as Python sequences for the purposes of collections.abc.Sequence.

The somewhat outdated JString factory is a Python class that pretends to be a Java string type. It has the marginal advantage that it can be imported before the JVM is actually started. Once the JVM is started, its class representation is pointed to java.lang.String and can be used to construct a new string object or to test if an object is actually a Java string using isinstance. It does not implement any of the other string methods and just serves as convenience class. The more capable java.lang.String can be imported in place of JString, but only after the JVM is started.

String objects may optionally convert to Python strings when returned from Java methods, though this option is a performance issue and can lead to other difficulties. This setting is selected when the JVM is started. See String Conversions for details.

Java strings will cache the Python conversion so we only pay the conversion cost once per string.

Exception Classes

Both Python and Java treat exception classes differently from other objects. Only these types may be caught as part of a try block. Therefore, the exceptions have a special wrapper. Most of the mechanics of exceptions happen under the surface. The one difference between Python and Java is the behavior when the argument is queried. Java arguments can either be the string value, the exception itself, or the internal construction key depending on how the exception came into existence. Therefore, the arguments to a Java exception should never be used as their values are not guaranteed.

Java exception can report their stacktrace to Python in two different ways. If printed through the Python stack trace routine, Java exceptions are split between the Python code that raised and a phantom Java cause which contains the Java exception in Python order. If the debugging information for the Java source is enabled, Python may even print the Java source code lines where the error occurred. If you prefer Java style stack traces then print the result from the stacktrace() method. Unhandled exception that terminate the program will print the Python style stack trace information.

The base class JException is a special type located in jpype.types that can be imported prior to the start of the JVM. This serves as the equivalent of java.lang.Throwable and contains no additional methods. It is currently being phased out in favor of catching the Java type directly.

Using jpype.JException with a class name as a string was supported in previous JPype versions but is currently deprecated. For further information on dealing with exception, see the Exception Handling section. To create a Java exception use JClass or any of the other importing methods.

Anonymous Classes

Sometimes Java will produce an anonymous class which does to have any actual class representation. These classes are generated when a method implements a class directly as part of its body and they serve as a closure with access to some of the variables that were used to create it.

For the purpose of JPype these classes are treated as their parents. But this is somewhat problematic when the parent is simply an interface and not an actual object type.

Lambdas

The companion of anonymous classes are lambda classes. These are generated dynamically and their parent is always an interface. Lambdas are always Single Abstract Method (SAM) type interfaces. They can implement additional methods in the form of default methods but those are generally not accessible within JPype.

Inner Classes

For the most part, inner classes can be used like normal classes, with the following differences:

  • Inner classes in Java natively use $ to separate the outer class from the inner class. For example, inner class Foo defined inside class Bar is called Bar.Foo in Java, but its real native name is Bar$Foo.
  • Inner classes appear as member of the containing class. Thus to access them import the outer class and call them as members.
  • Non-static inner classes cannot be instantiated from Python code. Instances received from Java code can be used without problem.

Importing Java classes

As Java classes are remote from Python and can neither be created nor extended within Python, they must be imported. JPype provides three different methods for creating classes.

The highest level API is the use of the import system. To import a Java class, one must first import the optional module jpype.imports which has the effect of binding the Java package system to the Python module lookup. Once this is completed package or class can be imported using the standard Python import system. The import system offers a very rich error reporting system. All failed imports produce an ImportError with diagnostics as to what went wrong. Errors include unable to find the class, unable to find a required dependency, and incorrect Java version.

One important caveat when dealing with importing Java modules. Python always imports local directories as modules before calling the Java importer. So any directory named java, com, or org will hide corresponding Java package. We recommend against naming directories as java or top level domain.

The older method of importing a class is with the JPackage factory. This factory automatically loads classes as attributes as requested. If a class cannot be found it will produce an AttributeError. The symbols java and javax in the jpype module are both JPackage instances. Only public classes appear on JPackage but protected and even private classes can be accessed by name. Though most private classes don’t have any methods or fields that can be accessed.

The last mechanism for looking up a class is through the use of the JClass factory. This is a low level API allowing the loading of any class available using the forName mechanism in Java. The JClass method can take up to three arguments corresponding to arguments of the forName method and can be used with alternative class loaders. The majority of the JPype test bench uses JClass so that the tests are only evaluating the desired functionality and not the import system. But this does not imply that JClass is the preferred mechanic for importing classes. The first argument can be a string or a Java class instance. There are two keyword arguments loader and initialize. The loader can point to an alternative ClassLoader which is handy when loading custom classes through mechanisms such as over the web. A False initialize argument loads a class without loading dependencies nor populating static fields. This option is likely not useful for ordinary users. It was provided when calling forName was problematic due to caller sensitive issues.

Type Matching

This section provides tables documenting the JPype conversion rules. JPype defines different levels of “match” between Python objects and Java types. These levels are:

  • none, There is no way to convert.
  • explicit (E), JPype can convert the desired type, but only explicitly via casting. Explicit conversions are only execute automatically in the case of a return from a proxy.
  • implicit (I), JPype will convert as needed.
  • exact (X), Like implicit, but when deciding with method overload to use, one where all the parameters match “exact” will take precedence over “implicit” matches.

See the previous section on Java Conversions for details.

There are special conversion rules for java.lang.Object and java.lang.Number. (Object Class and Number Class)

Python\Java byte short int long float double boolean char String Array Object java.lang.Object java.lang.Class
int I [1] I [1] X I I [3] I [3] X [8]         I [11]  
long I [1] I [1] I [1] X I [3] I [3]           I [11]  
float         I [1] X           I [11]  
sequence                          
dictionary                          
string               I [2] X     I  
unicode               I [2] X     I  
JByte X                     I [9]  
JShort   X                   I [9]  
JInt     X                 I [9]  
JLong       X               I [9]  
JFloat         X             I [9]  
JDouble           X           I [9]  
JBoolean             X         I [9]  
JChar               X       I [9]  
JString                 X     I  
JArray                   I/X [4]   I  
JObject                   I/X [6] I/X [7] I/X [7]  
JClass                       I X
“Boxed”[10] I I I I I I I         I  
[1](1, 2, 3, 4, 5, 6) Conversion will occur if the Python value fits in the Java native type.
[2](1, 2) Conversion occurs if the Python string or unicode is of length 1.
[3](1, 2, 3, 4) Java defines conversions from integer types to floating point types as implicit conversion. Java’s conversion rules are based on the range and can be lossy. See (http://stackoverflow.com/questions/11908429/java-allows-implicit-conversion-of-int-to-float-why)
[4]Number of dimensions must match and the types must be compatible.
[6]Only if the specified type is a compatible array class.
[7](1, 2) The object class is an exact match, otherwise implicit.
[8]Only the values True and False are implicitly converted to booleans.
[9](1, 2, 3, 4, 5, 6, 7, 8) Primitives are boxed as per Java rules.
[10]Java boxed types are mapped to Python primitives, but will produce an implicit conversion even if the Python type is an exact match. This is to allow for resolution between methods that take both a Java primitve and a Java boxed type.
[11](1, 2, 3) Boxed to java.lang.Number

Exception Handling

Error handling is an important part of any non-trivial program. All Java exceptions occurring within Java code raise a jpype.JException which derives from Python Exception. These can be caught either using a specific Java exception or generically as a jpype.JException or java.lang.Throwable. You can then use the stacktrace(), str(), and args to access extended information.

Here is an example:

try :
      # Code that throws a java.lang.RuntimeException
except java.lang.RuntimeException as ex:
      print("Caught the runtime exception : ", str(ex))
      print(ex.stacktrace())

Multiple java exceptions can be caught together or separately:

try:
      # ...
except (java.lang.ClassCastException, java.lang.NullPointerException) as ex:
      print("Caught multiple exceptions : ", str(ex))
      print(ex.stacktrace())
except java.lang.RuntimeException as ex:
      print("Caught runtime exception : ", str(ex))
      print(ex.stacktrace())
except jpype.JException:
      print("Caught base exception : ", str(ex))
      print(ex.stacktrace())
except Exception as ex:
      print("Caught python exception :", str(ex))

Exceptions can be raised in proxies to throw an exception back to Java.

Exceptions within the JPype core are issued with the most appropriate Python exception type such as TypeError, ValueError, AttributeError, or OSError.

Exception aliasing

Certain exceptions in Java have a direct correspondence with existing Python exceptions. Rather than forcing JPype to translate these exceptions, or forcing the user to handle Java exception types throughout the code, we have “derived” these exceptions from their Python counter parts. Thus, rather than requiring special error handling for Java you can simple catch these exceptions using the standard Python exception types.

java.lang.IndexOutOfBoundsException
This exception is synonymous with the Python exception IndexError. As many slicing or array operations in Java can produce an IndexOutOfBoundsException but the Python contract for slicing of an array should raise an IndexError, this type has been customized to consider IndexError to be a base type.
java.lang.NullPointerException
This exception is derived from the Python exception ValueError. Numerous Java calls produce a NullPointerException and in all cases this would match a Python ValueError.

By deriving these exceptions from Python, the user is free to catch the exception either as a Java exception or as the more general Python exception. Remember that Python exceptions are evaluated in order from most specific to least.

Controlling the JVM

In this chapter, we will discuss how to control the JVM from within Python. For the most part, the JVM is invisible to Python. The only user controls needed are to start up and shutdown the JVM.

Starting the JVM

The first task is always to start the JVM. The settings to the JVM are immutable over the lifespan of the JVM. The user settings are: the JVM arguments, the class path used to find jars, and whether to convert Java strings to Python strings.

Class paths

JPype supports two styles of classpaths. The first is modeled after Matlab the second argument style uses a list to the startJVM function.

The Matlab style uses the functions jpype.addClassPath and getClassPath. The first function adds a directory or jar file to the search path. Wild cards are accepted in the search. Once all of the paths are added to internal class path, they can be retrieved using getClassPath which takes a keyword argument env which defaults to true. When set to false, JPype will ignore the environment variable CLASSPATH which is normally included in the default classpath.

To use the argument style, pass all of the class paths in a list as the keyword argument classpath to the startJVM. This classpath method does not include the environment CLASSPATH, but it does provide a quick method to pull in a specific set of classes. Wild cards are accepted as the end of the path to include all jars in a given directory.

One should note that the class path can only be set prior starting the JVM. Calls to set the class path after the JVM is started are silently ignored. If a jar must be loaded after the JVM is started, it may be loaded using java.net.URLClassLoader. Classes loaded using a URLClassloader are not visible to JPype imports nor to JPackage.

String conversions

The convertStrings argument defines how strings are returned by JPype. Early in the life of this project return types were often converted to Python types without regard to preserving the type information. Thus strings would automatically convert to a Python string effectively the data from Java to Python on each return. This was a violation of the Python philosophy that explicit is better than implicit. This also prohibited chaining of Java string operations as each operation would lose the Java representation and have to be transferred back and forth. The simple operation of trying to create a Java string was difficult as directly calling java.lang.String constructor would once again convert the result back to a Python string, hence the need to use the JString factory. There was an option to turn off the conversion of strings, but it was never operable. Therefore, all code written at the time would expect Java strings to convert to Python strings on return.

Recognizing this is both a performance issue and that it made certain types of programming prohibitive, JPype switched to having a setting requiring applications to chose a policy at the start of operation. This option is a keyword argument convertStrings. The default for 0.7 is to give the older broken behavior. If specified as False, Java strings will act as ordinary classes and return a Java string instance. This string instance can be converted by calling the Python str() function. Failure to specify a policy will issue a warning message.

You are strongly encouraged to set convertStrings false especially when are writing reusable Python modules with JPype. String in JPype 0.8, the default will to not convert strings.

Path to the JVM

In order the start the JVM, JPype requires the path to the Java shared library typically located in the JRE installation. This can either be specified manually as the first argument to jpype.startJVM or by automatic search.

The automatic search routine uses different mechanisms depending on the platform. Typically the first mechanism is the use the environment variable JAVA_HOME. If no suitable JVM is found there, it will then search common directories based on the platform. On windows it will consult the registry.

You can get the JVM found during the automatic search by calling jpype.getDefaultJVMPath().

In order to use the JVM, the architecture of the JVM must match the Python version. A 64 bit Python can only use a 64 bit JVM. If no suitable JVM can be found it should raise an error. In some cases so rare, it may lead to a crash depending on how the platform handles a failed shared library load.

Launching the JVM

Now that we have discussed the JVM options, lets show how to put it into practice. Suppose that the Python script at the top level of your working director, with a subdirectory holding all your working jars ./lib, and a second directory with bare classes ./classes. Java has been properly installed with the same architecture as Python (both 64 bit in this case).

To start JPype we would execute the following:

import jpype
jpype.startJVM("-ea", classpath=['lib/*', 'classes'], convertStrings=False)

Arguments that begin with a dash are passed to the JVM. Any unrecognized argument will raise an exception unless the keyword argument ignoreUnrecognized is set to True. Details of available arguments can be found in the vendor JVM documentation.

The most frequent problem encountered when starting JPype is the jars failing to be loaded. Java is unforgiving when loading jar files. To debug the failures, we will need to print the loaded classpath.

Java has a method to retrieve the classpath that was used during the loading process.

print(java.lang.System.getProperty('java.class.path'))

This command will print the absolute path to each of the jars that will be used by the JVM. Each of the jars are written out explicitly as the JVM does not permit wild-cards. JPype has expanded each of them using glob. If an expected jar file is missing the list, then it will not be accessable.

There is a flag to determine the current state of the JVM. Calling jpype.isJVMStarted() will return the current state of the JVM.

Once the JVM is started, we can find out the version of the JVM. The JVM can only load jars and classfiles compiled for the JVM version or older. Newer jar files will invariably fail to load. The JVM version can be determined using jpype.getJVMVersion().

Shutting down the JVM

At the other end of the process after all work has been performed, we will want to shutdown the JVM to terminate the program. This will happen automatically and no user intervention is required. If however, the user wants to continue execution of Python code after the JVM is finished they can explicitly call jpype.shutdownJVM(). This can only be called from the main Python thread. Any other thread will raise an exception.

The shutdown procedure of JPype and Java is fairly complicated.

  1. JPype requests that the JVM shutdown gracefully.
  2. Java waits until all non-daemon thread terminate. Thus if you did not send a termination to each non-daemon threads the shutdown will wait here until those threads complete their work.
  3. Once the all threads have completed except for the main thread, the JVM will begin the shutdown sequence. From this point on the JVM is in a crippled state limited what can happen to spawning the shutdown threads and completing them.
  4. The shutdown will first spawn the threads of cleanup routine that was attached to the JVM shutdown hook in arbitrary order. These routines can call back to Python and perform additional tasks.
  5. Once the last of these threads are completed, JPype then shuts down the reference queue which dereferences held all Python resources.
  6. Then JPype shuts down the type manager and frees all internal resources in the JPype module.
  7. Last, it unloads the JVM shared library returning the memory used by the JVM.
  8. Once that is complete, control is returned to Python.

All Java objects are now considered dead and cannot be reactivated. Any attempt to access their data field will raise an exception.

Attaching a shutdown hook

If you have resources that need to be closed when the JVM is shutdown these should be attached to the Java Runtime object. The following pattern is used:

@JImplements(Runnable)
class MyShutdownHook:
    @JOverride
    def run(self):
       # perform any required shutdown activities

java.lang.Runtime.getRuntime().addShutdownHook(Thread(MyShutdownHook()))

This thread will be executed in a new thread once the main thread is the only one remaining alive. Care should always be taken to complete work in a timely fashion and be aware the shutdown threads are inherently racing with each other to complete their work. Thus try to avoid expensive operations on shutdown..

Debugging shutdown

The most common failure during shutdown is the failure of an attached thread to terminate. There are specific patterns in Java that allow you to query for all currently attached threads.

Customization

JPype supports three different types of customizations.

The first is to adding a Python base class into a Java tree as was done with certain exceptions. This type of customization required private calls in JPype and is not currently exposed to the user.

Second a Python class can be used as a template when a Java class is first constructed to add additional functionality. This type of customization can be used to make a Java class appear as a native Python class. Many of the Java collection classes have been customized to match Python collections.

Last, Python class can be added to the implicit conversion list. This customizer is used to make Python types compatable with Java without requiring the user to manually case over and over.

All customization available to the users is done through class decorators added to Python classes or functions.

Class Customizers

Java wrappers can be customized to better match the expected behavior in Python. Customizers are defined using decorators. Applying the annotations @JImplementationFor and @JOverride to a regular Python class will transfer methods and properties to a Java class. @JImplementationFor requires the class name as a string, a Java class wrapper, or Java class instance. Only a string can be used prior to starting the JVM. @JOverride when applied to a Python method will hide the Java implementationallowing the Python method to replace the Java implementation. when a Java method is overridden, it is renamed with an proceeding underscore to appear as a private method. Optional arguments to @JOverride can be used to control the renaming and force the method override to apply to all classes that derive from a base class (“sticky”).

Generally speaking, a customizer should be defined before the first instance of a given class is created so that the class wrapper and all instances will have the customization.

Example taken from JPype java.util.Map customizer:

@_jcustomizer.JImplementationFor('java.util.Map')
class _JMap:
    def __jclass_init__(self):
       Mapping.register(self)

    def __len__(self):
       return self.size()

    def __iter__(self):
       return self.keySet().iterator()

    def __delitem__(self, i):
       return self.remove(i)

The name of the class does not matter for the purposes of customizer though it should be a private class so that it does not get used accidentally. The customizer code will steal from the prototype class rather than acting as a base class, thus, ensuring that the methods will appear on the most derived Python class and are not hidden by the java implementations. The customizer will copy methods, callable objects, __new__, class member strings, and properties.

Type Conversion Customizers

One can add a custom converter method which is called whenever a specified Python type is passed to a particular Java type. To specify a conversion method add @JConversion to an ordinary Python function with the name of Java class to be converted to and one keyword of exact or instanceof. The keyword controls how strictly the conversion will be applied. exact is restricted to Python objects whose type exactly matches the specified type. instanceof accepts anything that matches isinstance to the specified type or protocol. In some cases, the existing protocol definition will be overly broad. Adding the keyword argument excludes with a type or tuple of types can be used to prevent the conversion from being applied. Exclusions always apply first.

User supplied conversions are tested after all internal conversions have been exhausted and are always considered to be an implicit conversion.

@_jcustomizer.JConversion("java.util.Collection", instanceof=Sequence,
  excludes=str)
def _JSequenceConvert(jcls, obj):
    return _jclass.JClass('java.util.Arrays').asList(obj)

JPype supplies customizers for certain Python classes.

Python class Implicit Java Class
pathlib.Path java.io.File
pathlib.Path java.nio.file.Path
datetime.datetime java.time.Instant
collections.abc.Sequence java.util.Collection
collections.abs.Mapping java.util.Map

Collections

JPype uses customizers to augment Java collection classes to operate like Python collections. Enhanced objects include java.util.List, java.util.Set, java.util.Map, and java.util.Iterator. These classes generally comply with the Python API except in cases where there is a significant name conflict and thus no special treatment is required when handling these Java types. Details of customizing Java classes can be found in the previous chapter, Customization.

This section will detail the various customization that are to applied the Java collection classes.

Iterable

All Java classes that implement java.util.Iterable are customized to support Python iterator notation and thus can be used in Python for loops and in list comprehensions.

Iterators

All Java classes that implement java.util.Iterator act as Python iterators.

Collection

All Java classes that inherit from java.util.Collection have a defined length determined by the Python len(obj) function. As they also inherit from Iterable, they have iterator, forech traversal, and list comprehension.

In addition, methods that take a Java collection can convert a Python sequence into a collection implicitly if all of the elements have a conversion into Java. Otherwise a TypeError is raised.

Lists

Java List classes such as ArrayList and LinkedList can be used in Python for loops and list comprehensions directly. A Java list can be converted to a Python list or the reverse by calling the requested type as a copy constructor.

pylist = ['apple', 'orange', 'pears']

# Copy the Python list to Java.
jlist = java.util.ArrayList(pylist)

# Copy the Java list back to Python.
pylist2 = list(jlist)

Note that the individual list elements are still Java objects when converted to Python and thus a list comprehension would be required to force Python types if required. Converting to Java will attempt to convert each argument individually to Java. If there is no conversion it will produce a TypeError. The conversion can be forced by casting to the appropriate Java type with a list comprehension or by defining a new conversion customizer.

Lists also have iterable, length, item deletion, and indexing. Note that indexing of java.util.LinkedList is supported but can have a large performance penalty for large lists. Use of iteration is much for efficient.

Map

A Java classes that implement java.util.Map inherit the Python collections.abc.Mapping interface. As such they can be iterated, support the indexing operator for value lookups, item deletion, length, and support contains.

Here is a summary of their capabilities:

Action Python
Place a value in the map jmap[key]=value
Delete an entry del jmap[key]
Get the length len(jmap)
Lookup the value v=jmap[key]
Get the entries jmap.items()
Fetch the keys jmap.key()
Check for a key key in jmap

In addition, methods that take a Java map can implicitly convert a Python dict or a class that implements collections.abc.Mapping assuming that all of the map entries can be converted to Java. Otherwise a TypeError is raised.

MapEntry

Java map entries unpack into a two value tuple, thus supporting iterating through key value pairs. Thus is useful when iterating map entries in a for loop by pairs.

Set

All Java classes that implement java.util.Set implement delitem as well as the Java collection customizations.

Enumeration

All Java classes that implement java.util.Enumeration inherit Python iterator behavior and can be used in Python for loops and list comprehensions.

Working with NumPy

As one of the primary focuses of JPype is working with numerical codes such as NumPy, there are a number of NumPy specific enhancements. NumPy is a large binary package and therefore JPype cannot be compiled against NumPy directly without force it to be a requirement. Instead of compiling against NumPy directly, JPype implements interfaces that NumPy can recognize and use. The specific enhancements are the following: direct buffer transfers of primitive arrays and buffers, direct transfer of multi dimensional arrays, buffer backed NumPy arrays, and conversion of NumPy integer types to Java boxed types.

Transfers to Java

Memory from a NumPy array can be transferred Java in bulk. The transfer of a one dimensional NumPy array to Java can either be done at initialization or by use of the Python slice operator.

Assuming we have a single dimensional NumPy array npa, we can transfer it with initialization using

ja = JArray(JInt)(npa)

Or we can transfer it to Java as a slice assignment.

ja[:] = npa

The slice operator can transfer the entire array or just a portion of it.

Multidimensional transfers to Java

Multidimensional arrays can also be transferred at initialization time. To transfer a NumPy array to Java use the JArray.of function

z = np.zeros((5,10,20))
ja = JArray.of(z)

Transfers to NumPy

Java arrays can be in two forms. Java multidimensional arrays are not contiguous in memory. If all of the arrays in each dimension are the same, then the array is rectangular. If the size of the arrays within any dimension differ, then the array is jagged. Jagged arrays are an array of arrays rather than a rectangular block of memory.

NumPy arrays only hold rectangular arrays as multidimensional arrays of primitives. All other arrangements are a stored as a single dimensional array of objects. JPype can automatically transfer a rectangular array to NumPy as a bulk transfer. To do so JPype supports a memoryview on rectangular arrays. Whenever a memoryview is called on a multidimensional array of primitives, JPype verifies that it is rectangular and creates a buffer. If it is jagged, a BufferError is raised. When a Java array is used as an argument to initialize a NumPy array, it creates a memoryview so that all of the memory can be transferred in bulk.

Buffer backed NumPy arrays

Java direct buffers provide access between foreign memory and Java. This access bypasses the JNI layer entirely, permitting Java and Python to operate on a memory space with native speed. Java direct buffers are not under the control of the garbage collector and thus can result in memory leaks and memory exhaustion if not used carefully. This is used with Java libraries that support direct buffers. Direct buffers are part of the Java nio package and thus functionality for buffers is in jpype.nio.

To create a buffer backed NumPy array, the user must either create a direct memory buffer using the Java direct buffer API or create a Python bytearray and apply jpype.nio.convertToByteBuffer to map this memory into Java space. NumPy can then convert the direct buffer into an array using asarray.

To originate a direct buffer from Java, use:

jb = java.nio.ByteBuffer.allocateDirect(80)
db = jb.asDoubleBuffer()
a = np.asarray(db)

To origate a direct buffer from Python, use:

bb = bytearray(80)
jb = jpype.nio.convertToDirectBuffer(bb)
db = jb.asDoubleBuffer()
a = np.asarray(db)

Buffer backed arrays have one downside. Python and by extension NumPy have no way to tell when a buffer becomes invalid. Once the JVM is shutdown, all buffers become invalid and any access to NumPy arrays backed by Java risk crashing. To avoid this fate, either create the memory for the buffer from within Python and pass it to Java. Or use the Java java.lang.Runtime.exit which will terminate both the Java and Python process without leaving any opertunity to access a dangling buffer.

Buffer backed memory is not limited to use with NumPy. Buffer transfers are supported to provide shared memory between processes or memory mapped files. Anything that can be mapped to an address with as a flat array of primitives with machine native byte ordering can be mapped into Java.

NumPy Primitives

When converting a Python type to a boxed Java type, there is the difficulty that Java has no way to known the size of a Python numerical value. But when converting NumPy numerical types, this is not an issue. The following conversions apply to NumPy primitive types.

Numpy Type Java Boxed Type
np.int8 java.lang.Byte
np.int16 java.lang.Short
np.int32 java.lang.Integer
np.int64 java.lang.Long
np.float32 java.lang.Float
np.float64 java.lang.Double

Further, these NumPy types obey Java type conversion rules so that they act as the equivalent of the Java primitive type.

Implementing Java interfaces

Proxies in Java are foreign elements that pretend to implement a Java interface. We use this proxy API to allow Python to implement any Java interface. Of course, a proxy is not the same as subclassing Java classes in Python. However, most Java APIs are built so that sub-classing is not required. Good examples of this are AWT and SWING. Except for relatively advanced features, it is possible to build complete UIs without creating a single subclass.

For those cases where sub-classing is absolutely necessary (i.e. using Java’s SAXP classes), it is necessaryy to create an interface and a simple subclass in Java that delegates the calls to that interface. The interface can then be implemented in Python using a proxy.

There are two APIs for supporting of Java proxies. The new high-level interface uses decorators which features strong error checking and easy notation. The older low-level interface allows any Python object or dictionary to act as a proxy even if it does not provide the required methods for the interface.

Implements

The newer style of proxy works by decorating any ordinary Python class to designate it as a proxy. This is most effective when you control the Python class definition. If you don’t control the class definition you either need to encapsulate the Python object in another object or use the older style.

Implementing a proxy is simple. First construct an ordinary Python class with method names that match the Java interface to be implemented. Then add the @JImplements decorator to the class definition. The first argument to the decorator is the interface to implement. Then mark each method corresponding to a Java method in the interface with @JOverride. When the proxy class is declared, the methods will be checked against the Java interface. Any missing method will result in JPype raising an exception.

High-level proxies have one other important behavior. When a proxy created using the high-level API returns from Java it unpacks back to the original Python object complete with all of its attributes. This occurs whether the proxy is the self argument for a method or proxy is returned from a Java container such as a list. This is accomplished because the actually proxy is a temporary Java object with no substance, thus rather than returning a useless object, JPype unpacks the proxy to its original Python object.

Proxy Method Overloading

Overloaded methods will issue to a single method with the matching name. If they take different numbers of arguments then it is best to implement a method dispatch:

@JImplements(JavaInterface)
class MyImpl:
   @JOverride
   def callOverloaded(self, *args):
     # always use the wild card args when implementing a dispatch
     if len(args)==2:
        return self.callMethod1(*args)
     if len(args)==1 and isinstance(args[0], JString):
        return self.callMethod2(*args)
     raise RuntimeError("Incorrect arguments")

   def callMethod1(self, a1, a2):
     # ...
   def callMethod2(self, jstr):
     # ...

Multiple interfaces

Proxies can implement multiple interfaces as long as none of those interfaces have conflicting methods. To implement more than one interface, use a list as the argument to the JImplements decorator. Each interface must be implemented completely.

Deferred realization

Sometimes it is useful to implement proxies before the JVM is started. To achieve this, specify the interface using a string and add the keyword argument deferred with a value of True to the decorator.

@JImplements("org.foo.JavaInterface", deferred=True)
class MyImpl:
   # ...

Deferred proxies are not checked at declaration time, but instead at the time for the first usage. Because of this, when uses an deferred proxy the code must be able to handle initialization errors wherever the proxy is created.

Other than the raising of exceptions on creation, there is no penalty to deferring a proxy class. The implementation is checked once on the first usage and cached for the remaining life of the class.

Proxy Factory

When a foreign object from another module for which you do not control the class implementation needs to be passed into Java, the low level API is appropriate. In this API you manually create a JProxy object. The proxy object must either be a Python object instance or a Python dictionary. Low-level proxies use the JProxy API.

JProxy

The JProxy allows Python code to “implement” any number of Java interfaces, so as to receive callbacks through them. The JProxy factory has the signature:

JProxy(intr, [dict=obj | inst=obj] [, deferred=False])

The first argument is the interface to be implemented. This may be either a string with the name of the interface, a Java class, or a Java class instance. If multiple interfaces are to be implemented the first argument is replaced by a Python sequence. The next argument is a keyword argument specifying the object to receive methods. This can either be a dictionary dict which names the methods as keys or an object instance inst which will receive method calls. If more than one option is selected, a TypeError is raised. When Java calls the proxy the method is looked up in either the dictionary or the instance and the resulting method is called. Any exceptions generated in the proxy will be wrapped as a RuntimeException in Java. If that exception reaches back to Python it is unpacked to return the original Python exception.

Assume a Java interface like:

public interface ITestInterface2
{
        int testMethod();
        String testMethod2();
}

You can create a proxy implementing this interface in two ways. First, with an object:

class C :
        def testMethod(self) :
                return 42

        def testMethod2(self) :
                return "Bar"

c = C()  # create an instance
proxy = JProxy("ITestInterface2", inst=c) # Convert it into a proxy

or you can use a dictionary.

def _testMethod() :
   return 32

def _testMethod2() :
   return "Fooo!"

d = { 'testMethod' : _testMethod, 'testMethod2' : _testMethod2, }
proxy = JProxy("ITestInterface2", dict=d)

Proxying Python objects

Sometimes it is necessary to push a Python object into Java memory space as an opaque object. This can be achieved using be implementing a proxy for an interface which has no methods. For example, java.io.Serializable has no arguments and little functionality beyond declaring that an object can be serialized. As low-level proxies to not automatically convert back to Python upon returning to Java, the special keyword argument convert should be set to True.

For example, let’s place a generic Python object such as NumPy array into Java.

import numpy as np
u = np.array([[1,2],[3,4]])
ls = java.util.ArrayList()
ls.add(jpype.JProxy(java.io.Serializable, inst=u, convert=True))
u2 = ls.get(0)
print(u is u2)  # True!

We get the expected result of True. The Python has passed through Java unharmed. In future versions of JPype, this method will be extended to provide access to Python methods from within Java by implementing a Java interface that points to back to Python objects.

Reference Loops

It is strongly recommended that object used in proxies must never hold a reference to a Java container. If a Java container is asked to hold a Python object and the Python object holds a reference to the container, then a reference loop is formed. Both the Python and Java garbage collectors are aware of reference loops within themselves and have appropriate handling for them. But the memory space of the other machine is opaque and neither Java nor Python is aware of the reference loop. Therefore, unless you manually break the loop by either clearing the container, or removing the Java reference from Python these objects can never be collected. Once you lose the handle they will both become immortal.

Ordinarily the proxy by itself would form a reference loop. The Python object points to a Java invocation handler and the invocation handler points back to Python object to prevent the Python object from going away as long as Java is holding onto the proxy. This is resolved internally by making the Python weak reference the Java portion. If Java ever garbage collects the Java half, it is recreated again when the proxy is next used.

This does have some consequences for the use of proxies. Proxies must never be used as synchronization objects. Whenever they are garbage collected, they loss their identity. In addition, their hashCode and system id both are reissued whenever they are refreshed. Therefore, using a proxy as a Java map key can be problematic. So long as it remains in the Java map, it will maintain the same identify. But once it is removed, it is free to switch identities every time it is garbage collected.

AWT/Swing

Java GUI elements can be used from Python. To use Swing elements the event loop in Java must be started from a user thread. This will prevent the JVM from shutting down until the user thread is completed.

Here is a simple example which creates a hello world frame and launches it from within Python.

import jpype
import jpype.imports

jpype.startJVM()
import java
import javax
from javax.swing import *

def createAndShowGUI():
    frame = JFrame("HelloWorldSwing")
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
    label = JLabel("Hello World")
    frame.getContentPane().add(label)
    frame.pack()
    frame.setVisible(True)

# Start an event loop thread to handling gui events
@jpype.JImplements(java.lang.Runnable)
class Launch:
    @jpype.JOverride
    def run(self):
        createAndShowGUI()
javax.swing.SwingUtilities.invokeLater(Launch())

Concurrent Processing

This chapter covers the topic of threading, synchronization, and multiprocess. Much of this material depends on the use of Proxies covered in the prior chapter.

Threading

JPype supports all types of threading subject to the restrictions placed by Python. Java is inherently threaded and support a vast number of threading styles such as execution pools, futures, and ordinary thread. Python is somewhat more limited. At its heart Python is inherently single threaded and requires a master lock known as the GIL (Global Interpreter Lock) to be held every time a Python call is made. Python threads are thus more cooperative that Java threads.

To deal with this behavior, JPype releases the GIL every time it leaves from Python into Java to any user defined method. Shorter defined calls such as to get a string name from from a class may not release the GIL. Every time the GIL is released it is another opportunity for Python to switch to a different cooperative thread.

Python Threads

For the most part, Python threads based on OS level threads (i.e. POSIX threads) will work without problem. The only challenge is how Java sees threads. In order to operate on a Java method, the calling thread must be attached to Java. Failure to attach a thread will result in a segmentation fault. It used to be a requirement that users manually attach their thread to call a Java function, but as the user has no control over the spawning of threads by other applications such as an IDE, this inevitably lead to unexpected segmentation faults. Rather that crashing randomly, JPype automatically attachs any thread that invokes a Java method. These threads are attached automatically as daemon threads so that will not prevent the JVM from shutting down properly upon request. If a thread must be attached as a non-daemon, use the method java.lang.Thread.attach() from within the thread context. Once this is done the JVM will not shut down until that thread is completed.

There is a function called java.lang.Thread.isAttached() which will check if a thread is attached. As threads automatically attach to Java, the only way that a thread would not be attached is if it has never called a Java method.

The downside of automatic attachment is that each attachment allocates a small amount of resources in the JVM. For applications that spawn frequent dynamically allocated threads, these threads will need to be detached prior to completing the thread with java.lang.Thread.detach(). When implementing dynamic threading, one can detach the thread whenever Java is no longer needed. The thread will automatically reattach if Java is needed again. There is a performance penalty each time a thread is attached and detached.

Java Threads

To use Java threads, create a Java proxy implementins java.lang.Runnable. The Runnable can then be passed any Java threading mechanism to be executed. Each time that Java threads transfer control back to Python, the GIL is reacquired.

Other Threads

Some Python libraries offer other kinds of thread, (i.e. microthreads). How they interact with Java depends on their nature. As stated earlier, any OS- level threads will work without problem. Emulated threads, like microthreads, will appear as a single thread to Java, so special care will have to be taken for synchronization.

Synchronization

Java synchronization support can be split into two categories. The first is the synchronized keyword, both as prefix on a method and as a block inside a method. The second are the three methods available on the Object class (notify, notifyAll, wait).

To support the synchronized functionality, JPype defines a method called synchronized(obj) to be used with the Python with statement, where obj has to be a Java object. The return value is a monitor object that will keep the synchronization on as long as the object is kept alive. For example,

from jpype import synchronized

mySharedList = java.util.ArrayList()

# Give the list to another thread that will be adding items
otherThread,setList(mySharedList)

# Lock the list so that we can access it without interference
with synchronized(mySharedList):
   if not mySharedList.isEmpty():
     ... # process elements
# Resource is unlocked once we leave the block

The Python with statement is used to control the scope. Do not hold onto the monitor without a with statement. Monitors held outside of a with statement will not be released until they are broken when the monitor is garbage collected.

The other synchronization methods are available as-is on any Java object. However, as general rule one should not use synchronization methods on Java String as internal string representations may not be complete objects.

For synchronization that does not have to be shared with Java code, use Python’s support directly rather than Java’s synchronization to avoid unnecessary overhead.

Threading examples

Java provides a very rich set of threading tools. This can be used in Python code to extend many of the benefits of Java into Python. However, as Python has a global lock, the performance of Java threads while using Python is not as good as native Java code.

Limiting execution time

We can combine proxies and threads to produce achieve a number of interesting results. For example:

def limit(method, timeout):
""" Convert a Java method to asynchronous call with a specified timeout. """
    def f(*args):
         @jpype.JImplements(java.util.concurrent.Callable)
         class g:
             @jpype.JOverride
             def call(self):
                 return method(*args)
         future = java.util.concurrent.FutureTask(g())
         java.lang.Thread(future).start()
         try:
             timeunit = java.util.concurrent.TimeUnit.MILLISECONDS
             return future.get(int(timeout*1000), timeunit)
         except java.util.concurrent.TimeoutException as ex:
             future.cancel(True)
         raise RuntimeError("canceled", ex)
    return f

    print(limit(java.lang.Thread.sleep, timeout=1)(200))
    print(limit(java.lang.Thread.sleep, timeout=1)(20000))

Here we have limited the execution time of a Java call.

Multiprocessing

Because only one JVM can be started per process, JPype cannot be used with processes created with fork. Forks copy all memory including the JVM. The copied JVM usually will not function properly thus JPype cannot support multiprocessing using fork.

To use multiprocessing with JPype, processes must be created with “spawn”. As the multiprocessing context is usually selected at the start and the default for Unix is fork, this requires the creating the appropriate spawn context. To launch multiprocessing properly the following recipe can be used.

import multiprocessing as mp

ctx = mp.get_context("spawn")
process = ctx.Process(...)
queue = ctx.Queue()
# ...

When using multiprocessing, Java objects cannot be sent through the default Python Queue methods as calls pickle without any Java support. This can be overcome by wrapping Python Queue to first encode to a byte stream using the JPickle package. By wrapping a Queue with the Java pickler any serializable Java object can be transferred between processes.

In addition, a standard Queue will not produce an error if is unable to pickle a Java object. This can cause deadlocks when using multiprocessing IPC, thus wrapping any Queue is required.

Miscellaneous topics

This chapter contains all the stuff that did not fit nicely into the narrative about JPype. Topics include code completion, performance, debugging Java within JPype, debugging JNI and other JPype failures, how caller sensitive methods are dealt with, and finally limitations of JPype.

Javadoc

JPype can display javadoc in ReStructured Text as part of the Python documentation. To access the javadoc, the javadoc package must be located on the classpath. This includes the JDK package documentation.

For example to get the documentation for java.lang.Class, we start the JVM with the JDK documentation zip file on the classpath.

We can then access the java docs for the String with help(java.lang.String) or for the methods with help(java.lang.String.trim). To use the javadoc supplied by a third party include the both the jar and javadoc in the classpath.

The parser will ignore any javadoc which cannot be extracted. It has some robustness against tags that are not properly closed or closed twice. Javadoc with custom page layouts will likely not be extracted.

If javadoc for a class cannot be located or extracted properly, default documentation will be generated using Java reflection.

Autopep8

When Autopep8 is applied a Python script, it reorganizes the imports to conform to E402. This has the unfortunate side effect of moving the Java imports above the startJVM statement. This can be avoided by either passing in --ignore E402 or setting the ignore in .pep8.

Example:

import jpype
import jpype.imports

jpype.startJVM()

from gov.llnl.math import DoubleArray

Result without --ignore E402

from gov.llnl.math import DoubleArray  # Fails, no JVM running
import jpype
import jpype.imports

jpype.startJVM()

Performance

JPype uses JNI, which is well known in the Java world as not being the most efficient of interfaces. Further, JPype bridges two very different runtime environments, performing conversion back and forth as needed. Both of these can impose performance bottlenecks.

JNI is the standard native interface for most, if not all, JVMs, so there is no getting around it. Down the road, it is possible that interfacing with CNI (GCC’s Java native interface) may be used. Right now, the best way to reduce the JNI cost is to move time critical code over to Java.

Follow the regular Python philosophy : Write it all in Python, then write only those parts that need it in C. Except this time, it’s write the parts that need it in Java.

Everytime an object is passed back and forth, it will incure a conversion cost.. In cases where a given object (be it a string, an object, an array, etc …) is passed often into Java, the object should be converted once and cached. For most situations, this will address speed issues.

To improve speed issues, JPype has converted all of the base classes into CPython. This is a very significant speed up over the previous versions of the module. In addition, JPype provides a number of fast buffer transfer methods. These routines are triggered automatically working with any buffer aware class such as those in NumPy.

As a final note, while a JPype program will likely be slower than its pure Java counterpart, it has a good chance of being faster than the pure Python version of it. The JVM is a memory hog, but does a good job of optimizing code execution speeds.

Code completion

Python supports a number of different code completion engines that are integrated in different Python IDEs. JPype has been tested with both the IPython greedy completion engine and Jedi. Greedy has the disadvantage that is will execute code resulting potentially resulting in an undesirable result in Java.

JPype is Jedi aware and attempts to provide whatever type information that is available to Jedi to help with completion tasks. Overloaded methods are opaque to Jedi as the return type cannot be determined externally. If all of the overloads have the same return type, the JPype will add the return type annotation permitting Jedi to autocomplete through a method return.

For example:

JString("hello").substring.__annotations__
# Returns {'return': <java class 'java.lang.String'>}

Jedi can manually be tested using the following code.

js = JString("hello")
src = 'js.s'
script = jedi.Interpreter(src, [locals()])
compl = [i.name for i in script.completions()]

This will produce a list containing all method and field that begin with the letter “s”.

JPype has not been tested with other autocompletion engines such as Kite.

Garbage collection

Garbage collection (GC) is supposed to make life easier for the programmer by removing the need to manually handle memory. For the most part it is a good thing. However, just like running a kitchen with two chiefs is a bad idea, running with two garbage collections is also bad. In JPype we have to contend with the fact that both Java and Python provide garbage collection for their memory and neither provided hooks for interacting with an external garbage collector.

For example, Python is creating a bunch a handles to Java memory for a period of time but they are in a structure with a reference loop internal to Python. The structures and handles are small so Python doesn’t see an issue, but each of those handles is holding 1M of memory in Java space. As the heap fills up Java begins garbage collecting, but the resources can’t be freed because Python hasn’t cleanup up these structures. The reverse occurs if a proxy has any large NumPy arrays. Java doesn’t see a problem as it has plenty of space to work in but Python is running its GC like mad trying to free up space to work.

To deal with this issue, JPype links the two garbage collectors. Python is more aggressive in calling GC than Java and Java is much more costly than Python in terms of clean up costs. So JPype manages the balance. JPype installs a sentinel object in Java. Whenever that sentinel is collected Java is running out of space and Python is asked to clean up its space as well. The reverse case is more complicated as Python can’t just call Java’s expensive routine any time it wants. Instead JPype maintains a low-water and high-water mark on Python owned memory. Each time it nears a high-water mark during a Python collection, Java GC gets called. If the water level shrinks than Java was holding up Python memory and the low-water mark is reset. Depending on the amount of memory being exchanged the Java GC may trigger as few as once every 50 Python GC cycles or as often as every other. The sizing on this is dynamic so it should scale to the memory use of a process.

Using JPype for debugging Java code

One common use of JPype is to function as a Read-Eval-Print Loop for Java. When operating Java though Python as a method of developing or debugging Java there are a few tricks that can be used to simplify the job. Beyond being able to probe and plot the Java data structures interactively, these methods include:

  1. Attaching a debugger to the Java JVM being run under JPype.
  2. Attaching debugging information to a Java exception.
  3. Serializing the state of a Java process to be evaluated at a later point.

We will briefly discuss each of these methods.

Attaching a Debugger

Interacting with Java through a shell is great, but sometimes it is necessary to drop down to a debugger. To make this happen we need to start the JVM with options to support remote debugging.

We start the JVM with an agent which will provide a remote debugging port which can be used to attach your favorite Java debugging tool. As the agent is altering the Java code to create additional debugging hooks, this process can introduce additional errors or alter the flow of the code. Usually this is used by starting the JVM with the agent, placing a pause marker in the Python code so that developer can attach the Java debugger, executing the Python code until it hits the pause, attaching the debugger, setting break point in Java, and then asking Python to proceed.

So lets flesh out the details of how to accomplish this…

jpype.startJVM("-Xint", "-Xdebug", "-Xnoagent",
  "-Xrunjdwp:transport=dt_socket,server=y,address=12999,suspend=n")

Next, add a marker in the form of a pause statement at the location where the debugger should be attached.

input("pause to attach debugger")
myobj.callProblematicMethod()

When Python reaches that point during execution, switch to a Java IDE such as NetBeans and select Debug : Attach Debugger. This brings up a window (see example below). After attaching (and setting desired break points) go back to Python and hit enter to continue. NetBeans should come to the foreground when a breakpoint is hit.

_images/attach_debugger.png

Attach data to an Exception

Sometimes getting to the level of a debugger is challenging especially if the code is large and error occurs rarely. In this case, it is often beneficial to attach data to an exception. To achieve this, we need to write a small utility class. Java exceptions are not strictly speaking expandable, but they can be chained. Thus, it we create a dummy exception holding a java.util.Map and attach it to as the cause of the exception, it will be passed back down the call stack until it reaches Python. We can then use getCause() to retrieve the map containing the relevant data.

Capturing the state

If the program is not running in an interactive shell or the program run time is long, we may not want to deal with the problem during execution. In this case, we can serialize the state of the relevant classes and variables. To use this option, we mus make sure all of the classes in Java that we are using are Serializable. Then add a condition that detects the faulty algorithm state. When the fault occurs, create a java.util.HashMap and populate it with the values to be examined from within Python. Use serialization to write the entire structure to a file. Execute the program and collect all of the state files.

Once the state files have been collected, start Python with an interactive shell and launch JPype with a classpath for the jars. Finally, deserialize the state files to access the Java structures that have been recorded.

Getting additional diagnostics

For the most part JPype does what its told, but that does not mean that there are no bugs. With some many different interactions between Python and Java there is always some untested edge-cases.

JPype has a few diagnostic tools to help deal with these sorts of problems but each of them require accessing a “private” JPype symbol which may be altered, removed, folded, spindled, or mutilated in any future release. Thus none of the following should be used in production code.

Checking the type of a cast

Sometimes it is difficult to understand why a particular method overload is being selected by the method dispatch. To check the match type for a conversion call the private method Class._canConvertToJava. This will produce a string naming the type of conversion that will be performed as one of none, explicit, implicit, or exact..

To test the result of the conversion process, call Class._convertToJava. Unlike an explicit cast, this just attempts to perform the conversion without bypassing all of the other logic involved in casting. It replicates the exact process used when a method is called or a field is set.

C++ Exceptions in JPype

Internally JPype can generate C++ exception which is converted into Python exceptions for the user. To trace an error back to its C++ source, it is necessary to obtain the original C++ exception. As all sensitive block have function names compiled in to the try catch blocks, these C++ exception stack frames can be extracted as the “cause” of a Python exception. To enable C++ stack traces use the command _jpype.enableStacktraces(True). Once executed all C++ exceptions that fell through a C++ exception handling block will produce an augmented C++ stack trace. If the JPype source code is available to Python, it can even print out each line where the stack frame was caught. This is usually at the end of each function that was executed. JPype does not need to be recompiled to use this option.

Tracing

To debug a problem that resulted from a stateful interaction of elements the use of the JPype tracing mode may helpful. To enable tracing recompile JPype with the --enable-tracing mode set. When code is executed with tracing, every JNI call along with the object addresses and exceptions will be printed to the console. This is keyed to macros that appear at the start and end of each JPype function. These macros correspond to a try catch block.

This will often produce very large and verbose tracing logs. However, tracing is often the only way to observe a failure that originated in one JNI call but did not fail until many calls later.

Instrumentation

In order to support coverage tools, JPype can be compiled with a special instrumentation mode in which the private module command _jpype.fault can be used to trigger an error. The argument to the fault must be a function name as given in the JP_TRACE_IN macro at the start of each JPype function or a special trigger point defined in the code. When the fault point is encounter it will trigger a SystemError. This mode of operation can be used to replicate the path that a particular call took and verify that the error handling from that point back to Java is safe.

Because instrumentation uses the same control hooks as tracing, only one mode can be active at a time. Enabling instrumentation requires recompiling the JPype module with --enable-coverage option.

Using a debugger

If there is a crash in the JPype module, it may be necessary to get a backtrace using a debugger. Unfortunately Java makes this task a bit complicated. As part of its memory handling routine, Java takes over the segmentation fault handler. Whenever the fault is triggered, Java checks to see if it was the result the growth of an internal structure. If it was simply a need for additional space, Java handles the exception by allocating addition memory. On the other hand, if a fault was triggered by some external source, Java constructs a JVM fault report and then transfers control back to the usual segmentation fault handler. Java will often corrupt the stack frame. Any debugger attempting to unpack the corrupted core file will instead get random function addresses.

The alternative is for the user to start JPype with an interactive debugger and execute to the fault point. But this option also presents challenges. The first action after starting the JVM is a test to see if its segmentation fault handler was installed properly. Thus it will trigger an intentional segmentation fault. The debugger can not recognize the difference between an intentional test and an actual fault, so the test will stop the debugger. To avoid this problem debuggers such as gdb must be set to ignore the first segmentation fault. Further details on this can be found in the developer guide.

Caller sensitive methods

The Java security model tracks what caller requested the method as a means to determine the level of access to provide. Internal callers are provided privileged access to perform unsafe operations and external callers are given safer and more restricted access. To perform this task, the JVM seaches the call stack to obtain the calling methods module.

This presents a difficulty for method invoked from JNI. A method called from JNI lacks any call stack to unravel. Rather than relegating the call to a safer level of access, the security model would outright deny access to certain JPype calls. This resulted in a number of strange behaviors over the years that were forced to be worked around. This issue was finally solved with the release of Java 12 when they outright broken all calls to getMethod by throwing a NullPointer exception whenever the caller frame was not found. This inadvertent clued us into why Java would act so strangely for certain calls such as constructing a SQL database or attempting to call Class.forName. By creating an actual test case to work around we were able to resolve this limitation.

Once we identified the issue, the workaround is only call caller sensitive methods from within Java. But given that we call methods through JNI and the JNI interface defines no way to specify an origin for the call, the means we needed to develop an alternative calling mechanism. Instead of calling methods directly, we instead pass the method id and the list of desired arguments to the internal org.jpype Java package. This package unpacks the request and executes the desired method from within Java. The call stack will indicate the caller is an external jar and be given the safe and restricted level of access. The result is then passed back to through the JNI layer.

This special calling mechanism is slower and more indirect than the normal calling procedure, so its use is limited to only those methods that really require a caller sensitive procedure. The mechanism to determine which methods are caller sensitive depends on the internals of Java and have changed with Java versions. Older Java versions did not directly mark the caller sensitive methods and we must instead blanket bomb all methods belonging to java.lang.Class, java.lang.ClassLoader, and java.sql.DriverManager. Newer versions specifically annotate the methods requiring caller sensitive treatment, but for some reason this annotation is a package private and thus we must search through method annotations by name to find the caller sensitive annotation. Fortunately, this process is only performed once when the class is created, and very few methods have a large number of annotations so this isn’t a performance hit.

JPype Known limitations

This section lists those limitations that are unlikely to change, as they come from external sources.

Restarting the JVM

JPype caches many resources to the JVM. Those resource are still allocated after the JVM is shutdown as there are still Python objects that point to those resources. If the JVM is restarted, those stale Python objects will be in a broken state and the new JVM instance will obtain the references to these resulting in a memory leak. Thus it is not possible to start the JVM after it has been shut down with the current implementation.

Running multiple JVM

JPype uses the Python global import module dictionary, a global Python to Java class map, and global JNI TypeManager map. These resources are all tied to the JVM that is started or attached. Thus operating more than one JVM does not appear to be possible under the current implementation. Further, as of Java 1.2 there is no support for creating more than one JVM in the same process.

Difficulties that would need to be overcome to remove this limitation include:

  • Finding a JVM that supports multiple JVMs running in the same process. This can be achieved on some architectures by loading the same shared library multiple times with different names.
  • Alternatively as all available JVM implementations support on one JVM instance per process, a communication layer would have to proxy JNI class from JPype to another process. But this has the distinct problem that remote JVMs cannot register native methods nor share memory without considerable effort.
  • Which JVM would a static class method call. The class types would need to be JVM specific (ie. JClass('org.MyObject', jvm=JVM1))
  • How would a wrapper from two different JVM coexist in the jpype._jclass module with the same name if different class is required for each JVM.
  • How would the user specify which JVM a class resource is created in when importing a module.
  • How would objects in one JVM be passed to another.
  • How can boxed and String types hold which JVM they will box to on type conversion.

Thus it appears prohibitive to support multiple JVMs in the JPype class model.

Errors reported by Python fault handler

The JVM takes over the standard fault handlers resulting in unusual behavior if Python handlers are installed. As part of normal operations the JVM will trigger a segmentation fault when starting and when interrupting threads. Pythons fault handler can intercept these operations and interpret these as real faults. The Python fault handler with then reporting extraneous fault messages or prevent normal JVM operations. When operating with JPype, Python fault handler module should be disabled.

This is particularly a problem for running under pytest as the first action it performs is to take over the error handlers. This can be disabled by adding this block as a fixture at the start of the test suite.

try:
    import faulthandler
    faulthandler.enable()
    faulthandler.disable()
except:
   pass

This code enables fault handling and then returns the default handlers which will point back to those set by Java.

Unsupported Python versions

Python 3.4 and earlier

The oldest version of Python that we currently support is Python 3.5. Before Python 3.5 there were a number of structural difficulties in the object model and the buffering API. In principle, those features could be excised from JPype to extend support to older Python 3 series version, but that is unlikely to happen without a significant effort.

Python 2

CPython 2 support was removed starting in 2020. Please do not report to us that Python 2 is not supported. Python 2 was a major drag on this project for years. Its object model is grossly outdated and thus providing for it greatly impeded progress. When the life support was finally pulled on that beast, I like many others breathed a great sigh of relief and gladly cut out the Python 2 code. Since that time JPype operating speed has improved anywhere from 300% to 10000% as we can now implement everything back in CPython rather than band-aiding it with interpreted Python code.

PyPy

The GC routine in PyPy 3 does not play well with Java. It runs when it thinks that Python is running out of resources. Thus a code that allocates a lot of Java memory and deletes the Python objects will still be holding the Java memory until Python is garbage collected. This means that out of memory failures can be issued during heavy operation. We have addressed linking the garbage collectors between CPython and Java, but PyPy would require a modified strategy.

Further, when we moved to a completely Python 3 object model we unfortunately broke some of the features that are different between CPython and PyPy. The errors make absolutely no sense to me. So unless a PyPy developer generously volunteering time for this project, this one is unlikely to happen.

Jython Python

If for some reason you wandered here to figure out how to use Java from Jython using JPype, you are clearly in the wrong place. On the other hand, if you happen to be a Jython developer who is looking for inspiration on how to support a more JPype like API that perhaps we can assist you. Jython aware Python modules often mistake JPype for Jython at least up until the point that differences in the API triggers an error.

Unsupported Java virtual machines

The open JVM implementations Cacao and JamVM are known not to work with JPype.

Unsupported Platforms

Some platforms are problematic for JPype due to interactions between the Python libraries and the JVM implementation.

Cygwin

Cygwin was usable with previous versions of JPype, but there were numerous issues for which there is was not good solution solution.

Cygwin does not appear to pass environment variables to the JVM properly resulting in unusual behavior with certain windows calls. The path separator for Cygwin does not match that of the Java DLL, thus specification of class paths must account for this. Threading between the Cygwin libraries and the JVM was often unstable.