Jython Journeys » jython

Transforming with XSLT on Google AppEngine and jython.

alan.kennedy — Tue, 01 Dec 2009 18:42:26 +0000

I’m writing this post in response to a challenge. I wrote a post entitled Jython on Google AppEngine: Why bother?, which was a reply to a statement made by a Google engineer about not seeing the point of running jython on AppEngine. And a similar statement was repeated in a comment on that blog post (i.e. “the gist of what I intended to say was that while it was possible to run Jython on App Engine, I couldn’t think why you’d want to.”)

So rather than get into a back-and-forth of yes-it-is-no-it-isnt-yes-it-is, I thought I’d reply with some simple code that demonstrates something that cannot (currently) be done in cpython on AppEngine, but is easy with jython on AppEngine: XSLT transforms.

So, without further ado, here is the jython source code for running an XSLT transform on Google AppEngine. It is implemented as a jython WSGI application, using xhaus.com’s modjy WSGI gateway for java servlets that was contributed to the jython project a few months back.

In order to get this code running, I suggest you download the modjy on google appengine demo from our downloads site. In that download, there is a jython file called “demo_app.py”. Simply replace the entire contents of that file with the code below, and you’re good to go. If you need help with getting the modjy demo application working on Google AppEngine, see the documentation on the modjy wiki: Running jython WSGI applications on Google AppEngine, with modjy.

__doc__ = """
A demonstration of how to run XSLT transforms on Google AppEngine with jython (not cpython).
Please feel free to use this code in whatever way you wish.
"""
from java.io import StringReader, StringWriter
from javax.xml.transform import Transformer, TransformerFactory
from javax.xml.transform.stream import StreamResult, StreamSource
 
xml_data = """

  
    2001 - A Space Odyssey
    Arthur C Clarke
  
  
    I, Robot
    Isaac Asimov
  
  
    Kil'n people
    David Brin
  

"""
 
xslt_source = """

 


  A list of books, transformed into html from xml using xslt and jython.
  
    
      
      A list of books, transformed into html from xml using xslt and jython.
      Title Author
    
    
    
  


 

"""
 
def generate_html(xml_data, xslt_source):
  transformer = TransformerFactory.newInstance().newTransformer(StreamSource(StringReader(xslt_source)))
  output_buffer = StringWriter()
  transformer.transform(StreamSource(StringReader(xml_data)), StreamResult(output_buffer))
  return output_buffer.buffer.toString()
 
def handler(environ, start_response):
  result = generate_html(xml_data, xslt_source)
  start_response("200 Hoopy", [ ('content-type', 'text/html') ])
  return [result]

Happy Transforming!

Jython on Google AppEngine: Why bother?

alan.kennedy — Mon, 30 Nov 2009 14:40:09 +0000

Why on earth would you want to run jython applications on Google Appengine?

First off, I must apologise for the flippant nature of the attention grabbing title. I’m a jython committer, and a jython user, so I believe that there are very good reasons for running jython on Google AppEngine. (You can read how to do so on the modjy wiki: how to run jython applications on Google AppEngine).

The title is essentially a paraphrase of a statement made by a Google AppEngine Evangelist, at a talk about AppEngine that I attended here in Dublin. The speaker was of the opinion that there was no valid reason for running jython on AppEngine, a statement which I couldn’t let pass.

I think that the core of his reasoning for making that statement was a simple misunderstanding, but a misunderstanding that is common among users who are familiar with python, or users of cpython, but who are not familiar with jython’s use as a JVM scripting language. The misunderstanding is that the only reason for running jython is so that one can run python applications on the JVM, using python’s wealth of "batteries included" APIs. If that is the case, then why bother running jython, when one can simply use the cpython support on AppEngine directly?

To explain, when writing a cpython application that processes email, for example, one can make use of the python’s email module, which provides a wealth of features for dealing with the many complexities of parsing, processing and generating email messages, MIME, etc. Users of jython can also use the python email modules, since those modules are written in pure python. But since jython runs on the Java Virtual Machine(JVM) and permits seamless integration of java and python code, jython users writing email processing also have the option of using the excellent JavaMail APIs, which are just as robust as the pure python modules, and arguably more efficient. So jython users have the best of both worlds: they can use either python APIs or java APIs, with equal ease.

Processing email is perhaps too straightforward an example, in that both python and java have excellent email processing support. But due to the more widespread use of java in the development community in general, java generally has far more support for any given technology need, since there are often multiple options available. A good example of this is cryptography: there is a wealth of java cryptography libraries available, including the excellent Bouncy Castle, Cryptix, Flexiprovider, etc. The choice of crypto libraries in python is generally narrower, supports less functionality, and is sometimes abandon-ware.

So if one’s understanding of jython is simply that it is a way to run pure python modules on the JVM, then the above mentioned mis-understanding is forgivable. But it is simply not the case that the primary usage of jython is to run pure python modules: Jython excels as a dynamic/scripting language for the JVM, and permits the use of java APIs, without having to write java code.

In discussion with the Google engineer after the talk, he conceded that there was one possible use case for running jython on AppEngine, namely that of prototyping java applications. In this scenario, applications would

Be developed in jython, taking advantage of the enormous productivity increases gained by developing in python
Be translated to java once the functionality of the application is complete

My reply to this is: “Why bother with step 2″?

If you’ve got a running application that is fully functional and written in jython, then you don’t need to translate it to java. You should only consider translating it, or parts of it, into java if you really have to. For example, after profiling your jython application, you may find one or two core pieces that are performance bottlenecks: these are prime candidates for translation to java, to improve performance. But there’s no reason to translate the rest of the application: maintainability and productivity is best served by simply leaving the jython code exactly as is.

I suppose a part of the reason why this mis-perception of jython has come about is because python is such an excellent language that it is in the unique position of having two completely independent implementations of the language that run on Google AppEngine. But just because both cpython and jython run on AppEngine doesn’t mean that their usages should be identical. Jython’s place in the AppEngine ecosystem is primarily as a JVM dynamic language.

So, in summary, jython is more than just a mechanism for running pure python code on the JVM: it is an excellent JVM development platform in it’s own right.

I hope I’ve given you some good reasons to consider developing your next Google AppEngine project on jython.

Jython 2.5; it’s a beautiful day!

alan.kennedy — Tue, 16 Jun 2009 18:39:23 +0000

A fantastic thing happened today: Jython 2.5 was released! I’ve anticipated this day a long time, as have many others; it’s great that it’s finally happened.

I’ve been using jython since 1999, when it was known as jpython. That was back in the Finn Bock Errata days; Finn had been kind of maintaining jpython after Jim Hugunin left CNRI, and produced a series of upgrade patches and fixes he called Errata. For legal reasons relating to CNRI’s IP, the project name had to be changed to jython; after that Finn, Samuele Pedroni and Barry Warsaw powered forward, bringing jython through cpython 2.0 and 2.1 compatibility. The result was a fantastically robust product, which had completely seamless Java integration, and which went on to become the JVM scripting language for a wide range of applications.

However, after 2.1 had settled down, Finn moved on. Cpython moved onto version 2.2, which brought big changes to the language: metaclasses, descriptors, properties, iterators, generators and nested scopes, to mention a few. All of the new features required large changes in the sometimes complex interactions between cpython and java class designs.

Samuele continued as jython’s maintainer, and a jython 2.2 alpha was produced that provided many of the required new features, and much of which still remains in jython 2.5. But Samuele moved on to the greener pastures of the amazing PyPy project, which is developing a self-hosting python-in-python interpreter. Although that is where his main energies lie, Samuele continues to be a jython contributor and provide valued guidance to the other developers.

The project languished for a couple of years, and the version number did not advance; 2.2 alpha was the state of jython for a long time, principally because of the enormous work involved in implementing a full 2.2 version. Brian Zimmer was project lead for a while, but much of the time there was no clear leadership. Jython 2.1 continued and continues to be used in a lot of places; it’s stability was and is fantastic, and it always did the job. But there was some angst about it’s future, especially with the advance of other JVM scripting languages such as JRuby, and with changes in Java itself, such as Annotations, Generics, etc.

Then the project sprang back to life, with a complete 2.2 being published; Charlie Groves was the key person in that achievement. Charlie did, and still does, an enormous amount of work on jython, and I believe that his dedication to releasing a complete and robust 2.2 version was key to revitalising jython, both the language and the community.

And then came Frank Wierzbicki, the current BDFL, and the person who has done most to bring jython to the modern state it exists in today.

Initially, there was hesitation about which version of the python language to target; cpython had reached version 2.5, and ironpython was at version 2.4. Upgrading jython to 2.3, then 2.4 and then 2.5 seemed like an awful lot of work. Frank adopted a bold strategy: aim to get jython as up-to-date as possible, and bring jython directly to 2.5. And today we’ve seen that that strategy worked!

Congratulations Frank! Superb job!

And well done to a great group of people who’ve contributed fantastic work; Jim Baker, Charlie Groves, Otmar Humbel, Tobias Ivarsson, Philip Jenvey, Nicholas Riley, Leo Soto, among others.

Well done to all the jython development team; ye’ve achieved something wonderful!

How to run jython WSGI applications on Google AppEngine with modjy.

alan.kennedy — Sun, 12 Apr 2009 21:36:01 +0000

As you can see from our last blog post, we were delighted to get jython WSGI applications running on Google AppEngine for java using modjy. We promised in that we’d write a tutorial on how to do do that.

Well we’ve done that: you can read the results on the modjy wiki: How to run jython WSGI applications on Google AppEngine with modjy.

If you successfully upload the mody demo application, you should see results like this: modjy demo application running on AppEngine

We’d be happy to know of any successful reports of jython WSGI applications running on AppEngine: please feel free to leave a comment about it.

Modjy running on Google AppEngine.

alan.kennedy — Sun, 12 Apr 2009 08:25:26 +0000

So, after the announcement by Google that their AppEngine could run other JVM languages, including jython, I had to have a go at getting modjy, our jython WSGI to java servlet bridge, running on it.

It wasn’t a complex job, but did require a number of tweaks to the standard jython deployment. I will write up these instructions and make them available very soon.

You can see the results here: modjy demo application running on Google AppEngine.

Jython runs on Google AppEngine!

alan.kennedy — Wed, 08 Apr 2009 13:05:40 +0000

It is now possible to run Jython on Google AppEngine!

If you’re not yet aware of it, Google AppEngine is one of the foremost Cloud Computing offerings currently available. It is also arguably one of the purest options available, since it truly removes the need for the application administrator to consider physical resources (apart from paying for them that is!). The Google AppEngine cloud will automatically scale up the resources as and when required.

This is in contrast to the Amazon EC2 offering, where the application architect has to specially design the application to distribute over the Amazon cloud, including allocating and de-allocating machine instances dynamically.

Yesterday, Google annnounced that they are providing Java on their AppEngine!

And because they’re supporting java, they also get automatic support for other languages that run on the JVM, with jython 2.2 and jython 2.5 being explicitly supported on AppEngine!

This is fantastic news!

The jython team are currently ramping up for a jython 2.5 Release Candidate, which should appear very soon. It’s very likely that we’ll get the Google patches included in jython 2.5 before then.

Discussions are ongoing how about applying the patch, and the issue of contributor agreements.

Watch this space!

Announce: jyson, a pure java JSON codec for jython.

alan.kennedy — Tue, 17 Mar 2009 22:19:15 +0000

I’m delighted to announce the publication of jyson, a fast codec for JavaScript Object Notation (JSON), a lightweight and easy-to-use data interchange format.

As you may know, JSON is used extensively in modern web development, particularly to communicate data between servers and clients in Rich Internet Applications, or Web 2.0 applications. JSON is commonly the preferred choice for data interchange between clients and servers which use a Respresentational State Transfer (REST) based architecture.

JSON is often the preferred choice because it is a fat free alternative to XML. Although we use XML all the time, we fully recognise that it be both complex to work with and inefficient to generate and parse. JSON, on the other hand, is both simple and fast.

Jyson is written in pure java, and should be highly performant. So, if you’re using jython and JSON, we recommend you check out jyson.

New xhaus.com open source software portal.

alan.kennedy — Sun, 15 Mar 2009 18:05:50 +0000

I’m delighted to announce the opening of the xhaus.com open source software portal.

On the portal you will find source code, documentation, issue trackers, etc, for open source software that we have published.

To run our open source portal, we use the excellent Redmine, an open source project management system written with Ruby on Rails. One thing we particularly like is that the Ruby community uses Redmine extensively, meaning that they subscribe to the philosophy of eating their own dog food, an admirable trait in the open source world.

We host our instance of Redmine on jruby, running on Apache Tomcat 6.0.

We chose Redmine because it is a highly functional and well architected piece of software. Browsing the source code for redmine shows that it is well designed and highly extensible, with a range of plugins for different tasks. Moreover, it has a rich feature set, including

Multiple projects
Hierarchical projects
Forums
Issue trackers
Wiki
File and Document areas
News
lots of other great features

Many thanks to the Redmine developers for their great software!

Being avid pythonistas, we normally like to use python or jython software when possible. In the case of project management, the obvious choice would be Trac, which was the first software of it’s type in the market, and in many ways defined the space that Redmine occupies.

But Trac has many problems, the most restrictive being that it can only support single projects (unless one is willing to do a fair amount of hacking). Furthermore, the Trac code base is getting a little crufty, a problem which is exacerbated by the fact that authors focus on changing the technologies on which Trac is built, rather than on adding new functionality that is useful to the end user.

So we invite you to visit the xhaus.com open source software portal.

Modjy now fully integrated into jython.

alan.kennedy — Sat, 07 Mar 2009 15:40:06 +0000

I’m delighted to announce that modjy, the java servlet to python WSGI gateway that I originally wrote to track the WSGI spec, has now finally been fully integrated into the jython source base.

Modjy has been distributed with jython for several months now, in the form of a third party library, as a zip file; I used to maintain the modjy source code in a private subversion repo. But now all of the java and jython source code for modjy has been checked into the jython source base. I did the integration work on a dedicated modjy branch. The jython BDFL, Frank Wierzbicki, then kindly exercised his considerable subversion fu to integrate that branch with the jython trunk, from whence the first jython 2.5 release candidate will be cut.

Jython 2.5 is a fantastic achievement, a tribute to Frank and all of the jython developers!

Socket shutdown versus socket close on cpython, jython and java.

alan.kennedy — Thu, 29 Jan 2009 18:16:48 +0000

So, here we are again with translating cpython socket semantics to java socket semantics, in order to correctly implement the cpython socket API on jython.

The latest issue is socket shutdown.

Cpython sockets have two separate methods related to terminating a socket connection; the close method and the shutdown method.

Before we get into the difference between the two, we should note that when a TCP socket connection is to be terminated, a network packet containing a FIN is sent to the peer, informing them to tear down the connection (although sometimes an RST packet can be sent instead). It is a sometimes used practice to shutdown only one side of a connection. For example, when a client knows that its request has been fully transmitted, it shuts down its transmit stream to the server, but leaves the receive stream open for the response. This makes the clients intention for future communications more explicit, and may lighten the load on the server. However, this is controversial topic, because that can also lead asynchronous servers to mistakenly think that a socket is readable when it is not. You can find a detailed and thorough discussion of this topic in ActiveState Recipe 408997: When to not just use socket.close(); make sure to read all of the comments.

The close method of sockets closes the file descriptor to which the socket is connected. When the last file descriptor connected to the socket is closed, then the operating system also closes the underlying socket, which results in the transmission of the FIN packet. That wording is important: “when the last file descriptor .. is closed” has specific meaning in many unix network servers architectures. The archetypal design for a network server on unix is to accept incoming socket connections in one process, and then spawn a subprocess to service the connection. Since the child process inherits the file descriptor table of its parent, it also inherits a file descriptor for the socket. Which means that there can be multiple file descriptors open on a socket. Which means that closing a socket is not guaranteed to close the connection; that will not happen until all file descriptors for the socket are closed.

The shutdown method, on the other hand only sends the FIN packet, regardless of how many file descriptors are open to the socket; moreover, it doesn’t affect any of those file descriptors.

There are some problems for implementing shutdown on java and thus on jython. Java servers are almost always multi-threaded, and process incoming requests in spawned threads, not processes. For this reason, and because of the fact that java does not expose C file descriptors, there are no socket methods that differentiate between the socket and file descriptor; the two are one and the same in java. This applies to all types of socket.

Implementing shutdown will be discussed below under the different socket types

TCP client sockets are fine: java.net.Socket objects have the shutdownInput and shutdownOutput methods, which shut down the receive stream and the transmit stream respectively.
TCP server sockets are different: they don’t have transmit and receive streams: they exist only to accept incoming socket connections. So when a TCP server socket is shutdown, the result of the call should be to close the listen queue for the socket, and discard all queued incoming connections. In java, server sockets do not have a shutdown method (as you can see from their javadocs: java.net.ServerSocket and java.nio.channels.ServerSocketChannels: remember, both classes are required to fully implement all cpython socket semantics in jython). So, how to implement the shutdown for TCP server sockets? In java, the same effect is achieved is issuing the close method. So the choice is whether or not to make shutdown call close? Since production code that uses the shutdown method will also call the close method immediately, or soon afterwards, that would make that series of calls fail. Therefore, the right choice (IMHO) is to have the shutdown method as a no-op, i.e. do nothing, and document the fact that the user must call close on the socket in the jython socket module documentation.
UDP sockets, similarly don’t have input and output streams; calling shutdown on the socket should stop the acceptance of incoming packets. But in java, neither java.net.DatagramSockets nor java.nio.channels.DatagramSocketChannels have shutdown methods; instead the same effect is achieved in by calling the close method. So, the right choice for the shutdown of UDP sockets is to make it a no-op, and document that fact.

I have checked in an implementation of the above into the jython SVN; in the release 2.2 maintenance branch at revision 6000 and into trunk at revision 6001. I have put together this page to record my reasoning for implementing things this way, and the research that I’ve done in trying to arrive at the right implementation. Or at least, an implementation that is as right as possible for the jython programmer.

On a related note, I see that the good people over in Twisted Matrix are making great progress with getting twisted running on jython. It will be fantastic to see one of the premier python asynchronous frameworks working on jython!

Lastly, thanks to Glyph Lefkowitz of twisted for reporting this situation in the jython bug tracker: listening socket shutdown expects the wrong kind of socket.