The debate was held at Google’s European HQ at Barrow St, Dublin.

Here are my slides from the python vs. ruby v. node.js debate.

The video from the debate is available on GTUGs youtube channel: the python segment begins after 20 minutes.

In May 2010, the Internet Committee for Assigned Names and Numbers (ICANN) permitted the first non-ASCII Internationalized Top-Level Domains to be made available.

An Internationalized Domain Name (IDN) is a domain name that includes characters from outside the ASCII character set. IDNs are specified in RFC 3490, and may contain any character from the Unicode character repertoire, meaning that domain names may now contain any writing symbol in use by human-beings anywhere on the planet.

Potential Issues

There are lots of interesting things and potential issues to note about IDNs.

They open up a whole new frontiers for cybersquatters and phishers, by opening up the possibility of Homograph Attacks, where the user is intentionally misguided by the visual similarity of character glyphs from different character sets. For example, is this character β a greek lower case beta or a german eszett?

Will companies like Société Générale find themselves held hostage by cybersquatters who have registered the accented version of their brand name before they even realise that it was possible to register it? In the case of the SocGen, the answer is no.

Solutions

Doubtless, various solutions will be adopted to solve these problems, as national registrars from around the world deal with how these problems relate specifically to the characters in use in their country. For example, the Russian national registrar Coordination Centre for RU TLD only permits Cyrillic characters to be used in the new .рф top-level domain.

Jython support

Unfortunately, jython idna encoding is currently broken, due to shortcomings in the unicodedata module and lack of stringprep support.

But IDNs are now supported when you run jython on a Java 6 JVM, since that version of the JVM has built-in IDN support: I checked in the changes at revision 7198.

There is also a workaround for java 5, where you can use GNU LibIDN to convert domain and host names to punycode before passing them to domain names that expect to receive ASCII parameters. This workaround is documented on the jython socket wiki.

If you’re interested in playing with IDNs, you can obviously download the latest jython. And there are plenty of Online IDN converters.

IPV4 exhaustion

As you are no doubt aware by now (since it’s been covered in all mainstream media), the reign of Internet Protocol Version 4 (IPV4) is coming to an end: the IPV4 address space will run out before the end of 2011.

The problem is a simple one: IPV4 addresses are made up of only 32 bits, meaning that they can only represent a maximum of 2³² different addresses (nearly 4.3 billion). But the internet is so successful and growing so fast that this address space is now too small: there are over 6 billion people on this planet, over 4 billion mobile phones, and over 1 billion PCs. Multi-device ownership is commonplace and growing in the rich world, and networking device ownership in the developing world is growing rapidly.

The decision to use 32-bit addresses was the right decision to make back in the 1960s. At the birth of the Internet, there were only several dozen machines connected to it. Rather than go into this subject in detail, I recommend reading this article: Vint Cerf on the Origins of 32-bit IP Addressing.

But as the Internet expanded and more and more machines connected to it, it became clear in the 1990s that IPV4 addresses would eventually run out, and that something would be needed to replace it: Internet Protocol Version 6 (IPV6) was born.

IPV6

IPV6 solves the address space problem permanently, by introducing a mind-bogglingly large address space: 128 bits, or 2¹²⁸ possible addresses. That’s 340,282,366,920,938,463,463,374,607,431,768,211,456 possible addresses! (Although see below for some caveats).

My favourite way to describe the size of this address space is as follows. If we were to divide the theoretical IPV6 address space between every star in the known universe (estimated to be 10²², or 10,000,000,000,000,000,000,000 stars), every star would have nearly 2⁵⁵ addresses to itself, i.e. an address space that is 8.4 million (2²³) times bigger than our existing IPV4 planetary internet (since 2²³ * 2³² == 2⁵⁵).

That’s a lot of addresses, and once IPV6 is fully in service across the internet, we won’t be running out of addresses for a very long time.

(Actually, the number of addresses available under IPV6 will be less than the theoretical maximum, because various sub-sets of the IPV6 address space have been reserved for special purposes. But the set of addresses available will remain practically unlimited for the foreseeable future).

Java vs. python

As readers of this blog will be aware, jython is an implementation of the python language that runs on the java platform.

Both python and java have been moving towards IPV6 for a number of years. Both are multi-platform languages, and attempt to provide APIs that work seamlessly across all operating systems.

Cpython, the most well known version of the python language, that is written in C, first provided IPV6 facilities back in version 2.2, released in December 2001. This support, although complete in design, did not initially work on all platforms on which cpython is used, because underlying operating system support was variable and patchy.

Simiarly, java introduced IPV6 support in version 1.4, released in February 2002. But this support suffered the same problems as with cpython: underlying operating system support was variable, particularly on older OSes such as Windows 2000, XP and 2003.

Fortunately, most operating systems have caught up with the present, and provide excellent IPV6 support, including most open-source operating systems, MacOS and Windows since Windows Vista.

Jython and IPV6

As is always true when using jython, there are two different approaches one can take to using IPV6 on jython: 1. Use the Java socket API or 2. Use the python socket API.

The java socket API

Users of jython have always had access to IPV6 though the java socket API, because of the seamless integration between java and jython. However, jython users often avoid using java APIs, particularly the more over-complex ones. And the java socket API is probably one of the worst examples of over-complexity, since it comprises two entirely different APIs, with different designs, to provide the full range of socket functionality: the java.net APIs and the java.nio APIs.

The python socket API

Long term users of jython networking libraries, which rely on the socket module, will be aware that the jython socket module has a particularly difficult task implementing the python socket API, since it must implement the cpython API, which has a C language philosophy, on top of the java API facilities, which have a different, more object-oriented philosophy. And as described above, the java socket API is really two completely different APIs, both of which must be used if you want the full range of functionality, e.g. asynchronous networking.

Implementing IPV6 has been problematic, for several reasons, including that operating system support has been patchy, thus making it difficult to reliably test on a cross-platform basis. Also, the cpython socket API is not without problems when it comes to IPV6.

Still, it has always been my stated intent that any python code written which uses the socket module should behave identically on jython and cpython, wherever possible. And now that intent extends to IPV6, final support for which I checked in recently (revisions 7191 and 7193), and which should be included in the next release of jython, due any day now.

I am particularly satisfied with this step forward, since I believe that IPV6 opens up a whole new world of networking possibilities.

Reliability

I won’t yet make the claim that the new IPV6 support is 100% defect free, since there are so many different java-version and operating-system combinations on which jython is used. There will always be some combination of operations on some JVM on some operating systems where it just won’t work, for example this java IPV6 defect on Windows. But the jython socket module has a very large set of unit-tests, currently 133 separate tests, and growing. As and when bugs are uncovered, I will document them, provide work-arounds, and unit-tests for the work-arounds.

Happy IPV6 networking!

Our conference venue, the Dublin School of English in Temple Bar, was a great success. Their classroom layouts proved ideal for talks and tutorials. The rooms for our three tracks were all accessed from a large foyer, where our sponsors set up their tables (and beanbags!). Open spaces rooms were also close, so the amount of traffic through the foyer gave a sociable and affable atmosphere.

Local restaurateurs Botticelli’s did a great job on the daytime catering, supplying tasty food and drink, and excellent coffee: we had a dedicated coffee machine and barista all day!

Just as I’ve experienced with other python conferences, there was a significant presence from the third-level education, both researchers and students. There were computer scientists interested in python as a language for teaching (the original purpose for which python was designed). There were research groups who were either already using or learning python in order to apply it to their programming needs, for analysing and visualising data, controlling equipment, etc. Work areas of third-level conference participants included genetics, electronics, and space hardware.

The venue for Saturday evening was the very modern Radisson Blu hotel on Golden Lane. The hotel staff were very professional and helpful: they made our entire experience there seamless and relaxed. What better way to end the evening than to retire to the bar for informal chats, the odd beer and the fine artistry of 3 piece blues band, Left, Right and Centre.

For the sprints on Sunday, the most remarkable project was certainly the project to code a solution finder for the fiendish Bedlam cube puzzles that sponsors Google had brought along. The effort even continued after the sprint was over, with the crew retiring to the TOG Hackerspace to work further on it.

One very successful outcome from the conference was that there is now a Python Ireland meetup in Cork city, which will be meeting on the first Thursday of each month. With the Galway Python meetups happening on the first Wednesday of the month, and the Dublin meetups on the second Wednesday, it is clear that Python Ireland is an active and growing community. Watch this space!

Many of the presentations given and media recorded at the conference are now available: See the following links

• Talks, slides, sourece code, etc
• Videos of talks
• General conference information

An enormous amount of organisation goes into making an event like a python conference work, and all of the committee are delighted that their work has all been worthwhile. One person deserves very special mention, Python Ireland’s Chairman, Vicky Lee, who was dedicated, untiring and endlessly patient through the whole process and put in more work than the rest of us put together: Massive kudos to Vicky!

Looking forward to PyCon Ireland 2011!

So rather than get into a back-and-forth of yes-it-is-no-it-isnt-yes-it-is, I thought I’d reply with some simple code that demonstrates something that cannot (currently) be done in cpython on AppEngine, but is easy with jython on AppEngine: XSLT transforms.

So, without further ado, here is the jython source code for running an XSLT transform on Google AppEngine. It is implemented as a jython WSGI application, using xhaus.com’s modjy WSGI gateway for java servlets that was contributed to the jython project a few months back.

In order to get this code running, I suggest you download the modjy on google appengine demo from our downloads site. In that download, there is a jython file called “demo_app.py”. Simply replace the entire contents of that file with the code below, and you’re good to go. If you need help with getting the modjy demo application working on Google AppEngine, see the documentation on the modjy wiki: Running jython WSGI applications on Google AppEngine, with modjy.

__doc__ = """
A demonstration of how to run XSLT transforms on Google AppEngine with jython (not cpython).
Please feel free to use this code in whatever way you wish.
"""
from java.io import StringReader, StringWriter
from javax.xml.transform import Transformer, TransformerFactory
from javax.xml.transform.stream import StreamResult, StreamSource
 
xml_data = """
<books>
  <book>
    <title>2001 - A Space Odyssey</title>
    <author>Arthur C Clarke</author>
  </book>
  <book>
    <title>I, Robot</title>
    <author>Isaac Asimov</author>
  </book>
  <book>
    <title>Kil'n people</title>
    <author>David Brin</author>
  </book>
</books>
"""
 
xslt_source = """
<xsl:stylesheet
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
>
 
<xsl:template match="books">
<html>
  <head><title>A list of books, transformed into html from xml using xslt and jython.</title></head>
  <body>
    <table border="1" width="50%" align="center">
      <caption>A list of books, transformed into html from xml using xslt and jython.</caption>
      <tr><th>Title</th><th>Author</th></tr>
    <xsl:for-each select="book">
      <tr><td><xsl:value-of select="title/text()"/></td><td><xsl:value-of select="author/text()"/></td></tr>
    </xsl:for-each>
    </table>
  </body>
</html>
</xsl:template>
 
</xsl:stylesheet>
"""
 
def generate_html(xml_data, xslt_source):
  transformer = TransformerFactory.newInstance().newTransformer(StreamSource(StringReader(xslt_source)))
  output_buffer = StringWriter()
  transformer.transform(StreamSource(StringReader(xml_data)), StreamResult(output_buffer))
  return output_buffer.buffer.toString()
 
def handler(environ, start_response):
  result = generate_html(xml_data, xslt_source)
  start_response("200 Hoopy", [ ('content-type', 'text/html') ])
  return [result]

Happy Transforming!

First off, I must apologise for the flippant nature of the attention grabbing title. I’m a jython committer, and a jython user, so I believe that there are very good reasons for running jython on Google AppEngine. (You can read how to do so on the modjy wiki: how to run jython applications on Google AppEngine).

The title is essentially a paraphrase of a statement made by a Google AppEngine Evangelist, at a talk about AppEngine that I attended here in Dublin. The speaker was of the opinion that there was no valid reason for running jython on AppEngine, a statement which I couldn’t let pass.

I think that the core of his reasoning for making that statement was a simple misunderstanding, but a misunderstanding that is common among users who are familiar with python, or users of cpython, but who are not familiar with jython’s use as a JVM scripting language. The misunderstanding is that the only reason for running jython is so that one can run python applications on the JVM, using python’s wealth of "batteries included" APIs. If that is the case, then why bother running jython, when one can simply use the cpython support on AppEngine directly?

To explain, when writing a cpython application that processes email, for example, one can make use of the python’s email module, which provides a wealth of features for dealing with the many complexities of parsing, processing and generating email messages, MIME, etc. Users of jython can also use the python email modules, since those modules are written in pure python. But since jython runs on the Java Virtual Machine(JVM) and permits seamless integration of java and python code, jython users writing email processing also have the option of using the excellent JavaMail APIs, which are just as robust as the pure python modules, and arguably more efficient. So jython users have the best of both worlds: they can use either python APIs or java APIs, with equal ease.

Processing email is perhaps too straightforward an example, in that both python and java have excellent email processing support. But due to the more widespread use of java in the development community in general, java generally has far more support for any given technology need, since there are often multiple options available. A good example of this is cryptography: there is a wealth of java cryptography libraries available, including the excellent Bouncy Castle, Cryptix, Flexiprovider, etc. The choice of crypto libraries in python is generally narrower, supports less functionality, and is sometimes abandon-ware.

So if one’s understanding of jython is simply that it is a way to run pure python modules on the JVM, then the above mentioned mis-understanding is forgivable. But it is simply not the case that the primary usage of jython is to run pure python modules: Jython excels as a dynamic/scripting language for the JVM, and permits the use of java APIs, without having to write java code.

In discussion with the Google engineer after the talk, he conceded that there was one possible use case for running jython on AppEngine, namely that of prototyping java applications. In this scenario, applications would

1. Be developed in jython, taking advantage of the enormous productivity increases gained by developing in python
2. Be translated to java once the functionality of the application is complete

My reply to this is: “Why bother with step 2″?

If you’ve got a running application that is fully functional and written in jython, then you don’t need to translate it to java. You should only consider translating it, or parts of it, into java if you really have to. For example, after profiling your jython application, you may find one or two core pieces that are performance bottlenecks: these are prime candidates for translation to java, to improve performance. But there’s no reason to translate the rest of the application: maintainability and productivity is best served by simply leaving the jython code exactly as is.

I suppose a part of the reason why this mis-perception of jython has come about is because python is such an excellent language that it is in the unique position of having two completely independent implementations of the language that run on Google AppEngine. But just because both cpython and jython run on AppEngine doesn’t mean that their usages should be identical. Jython’s place in the AppEngine ecosystem is primarily as a JVM dynamic language.

So, in summary, jython is more than just a mechanism for running pure python code on the JVM: it is an excellent JVM development platform in it’s own right.

I hope I’ve given you some good reasons to consider developing your next Google AppEngine project on jython.

So I’ve decided to expand the database to encompass the other language that I regularly use, java, and publish a database of open source HTTP proxies written in java.

While carrying out a web development project a few years ago, I needed a HTTP proxy to help with development and debugging. Rather than write my own, I decided to research what open source options were already available.

However, Python is such an easy language to write web servers and proxies in that a lot people had written and puiblished HTTP proxies, so making the decision wasn’t easy. It required a lot of research, which basically meant a code and architectural review of all of the open source python proxies available.

Since most web programmers use a HTTP proxy at some stage, I thought it would be a useful exercise to compile a complete list of the available options, with a comparison table to enable a feature by feature comparison.

The result of that was this comparison table of open source HTTP proxies written in python, which became quite popular, because it saved python programmers going through the same complex comparison and review process I had undertaken.

Since I’m currently working on another proxy-related project, I needed to review available open source HTTP proxies written in java. And I thought I’d save someone some research time by publishing a similar comparison table of open source HTTP proxies in java.

I hope you find it useful.

I’ve been using jython since 1999, when it was known as jpython. That was back in the Finn Bock Errata days; Finn had been kind of maintaining jpython after Jim Hugunin left CNRI, and produced a series of upgrade patches and fixes he called Errata. For legal reasons relating to CNRI’s IP, the project name had to be changed to jython; after that Finn, Samuele Pedroni and Barry Warsaw powered forward, bringing jython through cpython 2.0 and 2.1 compatibility. The result was a fantastically robust product, which had completely seamless Java integration, and which went on to become the JVM scripting language for a wide range of applications.

However, after 2.1 had settled down, Finn moved on. Cpython moved onto version 2.2, which brought big changes to the language: metaclasses, descriptors, properties, iterators, generators and nested scopes, to mention a few. All of the new features required large changes in the sometimes complex interactions between cpython and java class designs.

Samuele continued as jython’s maintainer, and a jython 2.2 alpha was produced that provided many of the required new features, and much of which still remains in jython 2.5. But Samuele moved on to the greener pastures of the amazing PyPy project, which is developing a self-hosting python-in-python interpreter. Although that is where his main energies lie, Samuele continues to be a jython contributor and provide valued guidance to the other developers.

The project languished for a couple of years, and the version number did not advance; 2.2 alpha was the state of jython for a long time, principally because of the enormous work involved in implementing a full 2.2 version. Brian Zimmer was project lead for a while, but much of the time there was no clear leadership. Jython 2.1 continued and continues to be used in a lot of places; it’s stability was and is fantastic, and it always did the job. But there was some angst about it’s future, especially with the advance of other JVM scripting languages such as JRuby, and with changes in Java itself, such as Annotations, Generics, etc.

Then the project sprang back to life, with a complete 2.2 being published; Charlie Groves was the key person in that achievement. Charlie did, and still does, an enormous amount of work on jython, and I believe that his dedication to releasing a complete and robust 2.2 version was key to revitalising jython, both the language and the community.

And then came Frank Wierzbicki, the current BDFL, and the person who has done most to bring jython to the modern state it exists in today.

Initially, there was hesitation about which version of the python language to target; cpython had reached version 2.5, and ironpython was at version 2.4. Upgrading jython to 2.3, then 2.4 and then 2.5 seemed like an awful lot of work. Frank adopted a bold strategy: aim to get jython as up-to-date as possible, and bring jython directly to 2.5. And today we’ve seen that that strategy worked!

Congratulations Frank! Superb job!

And well done to a great group of people who’ve contributed fantastic work; Jim Baker, Charlie Groves, Otmar Humbel, Tobias Ivarsson, Philip Jenvey, Nicholas Riley, Leo Soto, among others.

Well done to all the jython development team; ye’ve achieved something wonderful!

Well we’ve done that: you can read the results on the modjy wiki: How to run jython WSGI applications on Google AppEngine with modjy.

If you successfully upload the mody demo application, you should see results like this: modjy demo application running on AppEngine

We’d be happy to know of any successful reports of jython WSGI applications running on AppEngine: please feel free to leave a comment about it.

It wasn’t a complex job, but did require a number of tweaks to the standard jython deployment. I will write up these instructions and make them available very soon.

You can see the results here: modjy demo application running on Google AppEngine.