PyCon 2009 report

2009-04-15T00:00:00-07:00

PyCon 2009 report

15 Apr 2009 – Antwerp

This year was my first visit to PyCon in the United States. Here’s the full schedule for the conference.

In this post, I will give some notes and remarks about the sessions that I followed. For each session I included a link to the session details. Most of them have videos and slides of the talks.

How to Give a Python talk

A lot of talk about presentations using slides. I don’t like slides. The author didn’t like live coding because you could make a lot of typos, which would frustrate an audience. Later on, somebody gave a lightning talk about player piano, a project that types automatically for you and is very useful for presentations.

» How to Give a Python talk

Introduction to CherryPy

CherryPy is a lightweight HTTP framework. It is very much unlike Django: it doesn’t have an ORM or administrative interface, but just provides the perfect balance for creating web applications that are different from the standard CRUD.

You can install “tools” that can, for example, automatically encode/decode JSON. CherryPy-guy gave a nice example that I can’t find, but here’s the webpage.
You can use other dispatchers than the default. MethodDispatcher looks interesting, because it allows you to do REST-style systems.
There’s a tool to serve static content.

» Introduction to CherryPy

Introduction to Python Profiling

Tools for how to write fast Python code.

You need to profile your code to find out where it spends its time. This is really important: you really don’t know where your code is slow.
Use cProfile
KCacheGrind can help you with interpreting Python profiles.
RunSnakeRun allows you to view cProfile dumps in a GUI. This tool was created by the guy giving the talk.
Caching function references can speed up deeply nested code. But profile first.
Property lookups look just like an attribute lookup, but can be much slower.
Another interesting tool is line profiler, by Robert Kern of Enthought.

» Introduction to Python Profiling

Panel: Python VMs

A discussion about the different implementations of Python:

CPython, the “official” Python implementation. This gets confused with Python, because a few years ago there was only one Python language and implementation. Now there are many implementations of the same language.
Jython, Python on the JVM.
IronPython, Python on .NET and Mono.
PyPy, Python in Python. This is actually more useful than it sounds.
Unladen swallow, an effort by Google to make Python 10x faster.

What I found interesting in the discussion was that the “official” Python guys really welcome and support the other implementations. I had a chance to have lunch together with all the different VM guys, and they are really sincere about this.

Just as with JavaScript implementations, there is a lot of buzz and activity around making Python run faster. Python and JavaScript are very different languages however, and optimization techniques used in JavaScript cannot be used in Python. Still, everybody has their own idea of how to make Python faster (check out “PyPy status”).

» Panel: Python VMs

Python in a sandbox

A talk by the makers of PyPy about the use of PyPy as a virtualization tool. PyPy has a secure sandbox for running untrusted Python code, where you can bound IO calls, CPU and RAM resources. Really interesting for running other peoples’ Python code on your own server. PyPy is “only” 2-3 times slower than regular Python.

There is no official release, but here are the steps for running it yourself:

Download the sandbox.
Translate pypy with—sandbox flag (this takes a while).
Run the script using pypy interact.py

» Python in a sandbox

A better Python for the JVM

A very technical talk about the Jython compiler. A bit over my head. However the conclusions were:

Currently Jython is a bit slower than Python.
The new Jython compiler should make things a bit faster.
However, currently the new compiler isn’t faster.
But it provides better opportunities for optimization.

» A better Python for the JVM

Behind the scenes of EveryBlock.com

Adrian Holovaty, maker of Django, talked about Everyblock, his project to catalog everything that’s happening at the hyper-local level (in your neighborhood).

They have a lot of different data-types, and wanted to store all of them in the same table. This meant that date types had to be opaque, since the columns would be different for the various datatypes. The main table has columns named varchar1, varchar2, int1, int2, etc. A schema table defines what these various columns means for a given schema.

This really looked like a hack, which was surprising coming from Adrian, someone who has written such a clean web framework. It seems like an other database system (such as key-value stores or even a graph database, such as neo4j) would fit the job better.

One remark also struck me as odd: Adrian stated that he wrote everything from scratch, basically because “I don’t trust other people’s code”. It was unclear whether he was entirely serious or not, but the fact remains that no external code was used in his website.

» Behind the scenes of EveryBlock.com

Jython Progress

The reason for coming to PyCon: a very interesting talk by Frank Wierzbicki, who works at Sun on Jython.

The focus for the Jython 2.5 release was on compatibility. Through our use with Jython we found that a lot of stuff that worked in Python really worked in Jython as well, which was nice.

The focus for Jython 2.6 will be on performance and integration with Java. It should show up quicker than the 2.2 > 2.5 release, which took years to complete.

At the end he also showed Field, which is a Processing/NodeBox-like environment in Java using Jython. It has only very recently been open-sourced. It only runs on Mac OS X 10.5.

Jython Progress

Pinax: a platform for rapidly developing websites

Pinax has been getting some exposure lately because it provides ready-made components for Django. I had some troubles getting it to work, and was hoping this talk would provide some answers.

There seemed to be some problems with how to distribute this, and how to version all of it. They make a lot of use of svn:externals, which isn’t bad per se. It really requires a lot of commitment to get started with the framework, but once you have it installed, it provides a lot of the boilerplate functionality of sites, and more specifically, social networks: user-to-user messaging, twitter clone, tagging, photo management, interest groups, ... .

After the talk, I’m still not sure if it’s worth the trouble, or if I’d rather be writing a lot of that from scratch. I feel the framework is still a bit too young to be used without to much configuration hassle. Also, the remark of Adrian during his EveryBlock talk about writing everything himself rings true here.

» Pinax: a platform for rapidly developing websites

Class Decorators: Radically Simple

A nice introduction to “better metaclasses”. The functionality is only available in Python 2.6, but the talk is interesting (and short) enough to watch in its entirety.

» Class Decorators: Radically Simple

PyPy status talk

After the talk about Python sandboxing using PyPy, I was interested in what the status of PyPy was as a project. PyPy was started to be able to generate Python interpreters with more flexibility than a fixed C implementation. By defining a Python Interpreter on a higher level, you can experiment with different VM features quickly.

The PyPy interpreter is slower but consumes a lot less memory: some objects are 50% the size of CPython. They don’t support libraries written in C, but they do support CTypes as the official way to have bindings for PyPy.

» PyPy status talk

Drop ACID and think about data

Bob Ippolito talked about the various alternative data storage implementations that have come up recently. He covered both closed- and open-source implementations. Most of the talk was about alternative key-value stores and other non-relational models, but he didn’t cover graph databases.

There are a lot of different kinds of databases out there, but none of them seemed particularly stable or usable for my purposes. Afterwards, I checked out the open space about Cassandra, which seemed like the most stable one.

» Drop ACID and think about data

Concurrency and Distributed Computing with Python Today

This talk was not so much about the multiprocessing module in Python 2.6, but more about the tools available today.

He covered Jython as a solution for the global interpreter lock (GIL), because Jython uses Java threading which does not have the GIL. Also, the usage of java.util.concurrent solves a lot of problems. Actually, this seemed the most interesting approach for my projects.

He also talked about Stackless Python, which offers lightweight threads and cooperative multitasking.

There are different approaches to multitasking:

Real threads: hard to use because concurrency requires a lot of thinking ahead. They are hampered by the GIL in CPython.
Coroutines: light-weight threads are not truly parallel, but simplify the threading problem.
Actors: isolated, self reliant components that communicate via messages. They are truly parallel, and are a good model to overcome the threading fallacies. Erlang and Scala are two programming languages that use the actors idiom for concurrency.

All of these approaches have libraries implementing them in Python. There are a lot of them, and most are alpha quality. The two good ones are Twisted and Kamaelia. The rest is a mish-mash of technologies. (“Concurrency is hard, let’s go shopping!”)

» Concurrency and Distributed Computing with Python Today

Abstraction as Leverage

An abstract talk about the tower of abstractions we use every as programmers, and how to produce and consume abstraction layers.

The starting point for the talk was that abstractions “leak”. This is something Joe Spolsky has written about, and an interesting read. He also pointed out a quote from Jason Fried of 37 signals: “Prefer action over abstr-action”.

It’s hard to sum up the talk without paraphrasing it (badly), so I suggest you go see it.

» Abstraction as Leverage

Paver: easy build and deployment automation for Python projects

Paver is sort of like Ant for Python projects, but instead of using a bizarre XML syntax, just using Python. It is also very similar to Ruby’s Rake. It is really interesting and looks mature. It requires you to make a pavement.py file in your project directory that gets picked up and executed by Paver.

The basic building blocks in Paver are tasks. Tasks can be given command options (using optparse -like syntax).

Paver can be used for deploying servers, generating documentation, building a distribution (using distutils/setuptools), working with files, etc. Paver doesn’t replace distutils, but embraces and extends it in a non-evil way.

A good introduction is Getting Starting with Paver.

» Paver: easy build and deployment automation for Python projects

Making games in Python – Tools and techniques at CCP

I wanted to see this talk to see how a gaming company used Python almost everywhere in a high-profile MMORPG. They use Python for networking, the web server, content authoring tools (using wxPython) and game logic. Some performance-critical parts were they use C+. They did not talk about how they wrapped C+ code, only that they didn’t use Boost.Python.

They use embedded stackless Python, which they actively maintain. It allows them to do micro-threading (every object in the world runs in its own light-weight thread).

They use their own code-reloading system called livecoding because the reload built-in was deemed unsuitable. Their custom code reloading system allows them to reload game code while server and client are running. They also integrated unit testing so that the tests run before the code is reloaded to see if it passes.

» Making games in Python – Tools and techniques at CCP

Stackless python in EVE, pt. 2

This talk went more in depth about how they used Stackless Python in EVE. It covered StacklessIO, a unified framework for blocking operations in Stackless Python. He also talked about how they deployed StacklessIO at CCP, using “Cowboy mode”, which short-circuited normal QA procedures. Although interesting, I could not see myself using any of the technologies or working methods introduced.

» Stackless python in EVE, pt. 2

Designing a web framework: Django’s design decisions

I love to see Jacob Kaplan-Moss give a talk, and this one was no exception. This talk was high-level overview of some of the design decisions they made in the framework, and how to keep ego out of the equation.

He talked about the difference between academic frameworks (“architecture astronauts”) and the real world.

What I found interesting was the decision in Django to make a full stack framework instead of glueing together existing components. Part of the reason for this was that there were not much existing components available, but also that they could provide one consistent API dialect that felt they same, whether you were working in the templating language, the database API or some other part of the stack.

He also pointed that out that having users pick from existing components because it would allow for full flexibility only makes sense for expert users, and not for beginners that are just starting out with your framework. Having them pick from a list of components that they have no knowledge over is silly.

He also pointed out the talk by Cal Henderson of Flickr, Why I Hate Django which is really interesting and funny.

» Designing a web framework: Django’s design decisions

Open Space: Cassandra

I followed an open space talk by Jonathan Ellis of Rackspace Managed Hosting, one of the maintainers of Cassandra. Cassandra is a distributed database somewhere between BigTable and Dynamo, which are both closed source. The project was started by two ex-Googlers at Facebook, and has now been open-sourced. The talk went into depth about the implementation of Cassandra.

The system looks interesting but way too big for my needs. As long as we don’t really need high-performance distributed databases, I’d prefer my ACID using MySQL or Postgresql (or Neo4j).

Neo4j

Turns out the maker of the new Jython compiler also works on an open-source graph database called neo4j. His name is Tobias Ivarsson. I talked to him and showed him Perception. He was convinced that neo4j would be perfect for this: super-fast, O(1) lookups, lightweight, Python bindings. It is a mature graph database that runs in production for over 5 years.

There are components available for indexing, graph algorithms and much more.

Answered questions

I came to the conference to hear talks about the progress and future of Jython. I am now convinced that Jython is an excellent choice for Python development, given that there are some very motivated people working on it, and they have the full support of the CPython guys.

Unanswered questions

One thing I’ve been struggling with in NodeBox is how to do proper packaging of NodeBox packages. I need a system that can version packages, do dependency management, can run several version of the same package at the same time, and do live loading/unloading of packages. I recently discovered OSGi recently, which seems to solve this problem for Java. However, Python doesn’t seem to have a system like that available. There was a lot of talk about virtualenv, pip and even zc.buildout, but none of those seems to provide a complete answer. (By the way, Jacob’s recent blog post about zc.buildout is very interesting.) The whole environment seems very much in flux, and I think we’ll have to wait for the next PyCon to have an answer available.

Personally, I would love to see integration between the VM’s native package management system and the Python implementation (for Jython, that would be OSGi). However, even OSGi seems to come under attack, as Sun is rolling its own solution.

Overall, package management is a hairy business, and one that needs some serious thought. As for now, I think I’m better off taking a good hard look at OSGi and then rolling my own solution.

My Favourite Python hack

2009-03-27T00:00:00-07:00

My Favourite Python hack

27 Mar 2009 – Chicago

In NodeBox, we have this cool feature that we call “the throttle”. It allows you to interactively drag any number in the (Python) script and see the results immediately. Whether it’s and X/Y coordinate, the number in “for i in range(10)”, or a number in a complicated formula, just dragging the number helps you visualize the effect of it on the overall composition. Here’s a movie that demonstrates how this works:

Today I’d like to talk about the implementation, which works like this:

Find the character under the cursor.
Expand until you get the full number (include digits, periods and minus signs)
Replace it with magic_var and add this variable to the locals dictionary
When the user “drags” the number, replace the value of magic_var with an updated value and run the script again.
When the user releases the mouse, replace the old number in the source file with the new number.

Sounds simple, right? While this approach worked well in 90% of the cases, it broke down when the signs of the number changes from positive to negative or vice versa. Here are the three use cases:

* If an addition goes from positive to negative the sign should change:

random(alpha+8) -> random(alpha-8)

* If a subtraction goes from negative to positive the sign should change as well:

random(alpha-8) -> random(alpha+8)

* But if a negative number turns positive, the sign should disappear:

random(-8) -> random(8)

The difficulty was in finding the difference between the first case and the third case. After a lot of dabbling around with regular expressions, I suddenly had an insight. There is actually a Python module that handles this problem: the Python parser itself.

When Python parses the code, it builds an abstract syntax tree (AST), a tree-like representation of the code. Different nodes are represented by classes in the compiler.ast module. The use cases above are represented by instance of Sub (a two-part subtraction), UnarySub (a negative number), and Addition (a two-part addition).

Note that we replaced the original number with the name magic_var in our original source code, and compiled that. After compilation succeeds and returns an AST, a function called _checkSigns recursively checks the AST for special sign cases. First, it tries to find the magic variable. Once it has found that, it checks the parent node to see if it encounters any of the special cases, and basically prepares the object for string replacement afterwards.

What I like about this hack is that if it works correctly, you don’t even notice it’s there.

If you’d like to view the code, it’s in the NodeBox source as ValueLadder.py.

Revealing movie credits

2009-03-12T00:00:00-07:00

Revealing movie credits

12 Mar 2009 – Antwerp

I think it is funny how some movies reveal everything there is to know about them, just by looking at the movie credits. Not even the actors, just the roles they play:

Crack head 1
Elegant woman
Crackhead 2
Mass Appeal Editor
Woman in closet
Crackwhore
Shower Girl
Dealer
Tom
High-school girl
Perfect Urban Gentleman
Pool Player’s Homie
High-school girl 1
Junkyard slut
Junkyard stud
Woman on couch
Porn Model
Hooker
Shitkicker

The movie, if you’re still interested, is called Bad Biology. I haven’t seen it yet, but judging by the credits and the synopsis talking about the woman with 9 clitorises (clitorii?), I probably should.

Frederik De Bleser

PyCon 2009 report

PyCon 2009 report

How to Give a Python talk

Introduction to CherryPy

Introduction to Python Profiling

Panel: Python VMs

Python in a sandbox

A better Python for the JVM

Behind the scenes of EveryBlock.com

Jython Progress

Pinax: a platform for rapidly developing websites

Class Decorators: Radically Simple

PyPy status talk

Drop ACID and think about data

Concurrency and Distributed Computing with Python Today

Abstraction as Leverage

Paver: easy build and deployment automation for Python projects

Making games in Python – Tools and techniques at CCP

Stackless python in EVE, pt. 2

Designing a web framework: Django’s design decisions

Open Space: Cassandra

Neo4j

Answered questions

Unanswered questions

My Favourite Python hack

My Favourite Python hack

Revealing movie credits

Revealing movie credits