Coder Who Says Py

Asynchronous XML-RPC in Python

2012-01-26T09:58:00.002-08:00

Do you use XML-RPC (and specifically the xmlrpclib/xmlrpc.client from Python's stdlib)? Do you like multi-calls? Wish you could construct your XML-RPC multi-calls in a way so that you could make them asynchronous by constructing the call from scratch? Then you're in luck because I already did the hard work of figuring out the details for you! =)

In my case I needed to communicate with an XML-RPC server that supported multi-calls, but in an asynchronous fashion with a non-standard communication object (i.e. not sockets but App Engine's urlfetch). So I had to piece together how to make a XML-RPC client call that used multi-calls so that I could do it asynchronously myself.

The first thing to know is that a multi-call is a system.multicall function call to the server. It takes a sequence of arguments that specify what each individual function call should be. Each sequence item contains a dict with two keys: "methodName" and "params". The "methodName" key has a string value of the function name to call on the XML-RPC server. The "params" key holds the tuple of arguments (and it must be the tuple type and you must use a tuple even if it is just a single argument --think of what it would take for the call to work with *args) to pass to the function named in "methodName".

Knowing what function you are calling on the other end (system.multicall) and having a sequence of dicts specifying what you are calling in hand, you can make your call to the server. First you need to encode everything into XML: xmlrpc.client.dumps((seq_of_args,), 'system.multicall') (notice how I passed the sequence of dicts in a tuple -- and I mean a tuple, not a list). With the XML now in hand, you make the actual call to the server by making a POST with the XML as the body and set the Content-Type header to text/xml using whatever asynchronous mechanism you want to use to make the call.

What you get back from the server is XML that you can load with xmlrpc.client.loads(data, use_datetime=True)(chances are you will want the [0][0] value as that contains a sequence of results from each function call you requested). Do realize, though, that there is no indication of what function returned what, so you will need to correlate the index of the return value with what call you made (zip() makes this trivial).

And that's how you construct an XML-RPC multi-call from scratch in Python so you can make an asynchronous call. If you want to see an example of all this, see this code.

Grab bag of tips when working with App Engine unit testing

2012-01-24T15:44:00.000-08:00

[update: App Engine doc bug is being fixed and should be publicly visible in 1.6.2 or 1.6.3]

First, an announcement: register for PyCon! It's a great conference and tons of fun. Early bird ends on the 25th.

With that out of the way, this blog post is going to be testing an App Engine app. This can be, liking testing any complex system, is painful. And unfortunately the documentation on this subject is somewhat lacking for App Engine. But hopefully this post about random things to be aware of when writing unit tests for App Engine can help prevent someone else from having to learn the hard way like I did.

First thing you should do when testing an App Engine app is set up a testbed. This will allow you to stub out various parts of App Engine so your tests run in isolation without having stale state carrying over. You basically create a testbed which holds state. You then activate the system by setting some core environment variables with defaults. After that you do your own tweaks to env vars. From there you activate various stubs. And then finally you deactivate the whole thing to clean up...

But you need to realize there is an apparent error in the docs (which will be fixed no later than 1.6.3, possibly 1.6.2). The stated order of calling activate() and setup_env() are wrong; you should call activate() before calling setup_env() in order to set the default env vars before you begin to set your own.

Another thing you should know about is that setup_env() has an overwrite argument. This tells the method to overwrite any pre-existing env var; otherwise the call will skip changing any pre-existing env vars. This can call issues if your app presets some env vars that should be overridden during testing. I would honestly consider using that argument every time to make sure things work the way you expect.

And the last tip specific to setup_env()is make sure to pass all of your env var changes in a single call. Otherwise you can end up with some settings be changed as they have a default value.

In relation to handling users, to specify who the logged in user is you should set the USER_EMAIL env var. That will cause App Engine to return a user with the specified email address when calling users.get_current_user().

Random thing about email testing is that mail.send_mail_to_admins() does not write to the mail stub to verify any email was sent. This is because while testing there is no way to specify who admins are, so there is no way to know who to say the emails are sent to. So in order to test the function you will need to mock out the call yourself (personally I'm a fan of mock if you don't already have a preferred mocking library).

And finally, you can mock our the request and response objects for a handler. You can call a handler's initialize() method with the object to use -- for the request and response, respectively -- which allows you to mock and test how the handler does things.

Bonus for those of you using the Python 2.7 runtime: I would consider using unittest's discovery features to drive your unit tests. You can start from my driver code and add what you need. But it's nice to just be able to plop down a new test module and have it automatically picked up.

Working with App Engine backends

2012-01-17T15:53:00.000-08:00

This post is all about backends in App Engine, but first I want to make two personal points.

One is that I'm sorry I have not been updating this blog (or really contributing to Python and its great community) more this past year. It was a rather insane 2011 for me personally: I finished my Ph.D., got married, moved to SF (w/o my wife thanks to immigration), started working at Google full-time, moved to Toronto (yes, I moved internationally within the span of 6 months), transferred to the Google Waterloo office and started on a new team. In other words I have been stressed out and busy and busy continuously for what feels like ages. But now that I am back with my wife and I am done with the crazy international moving I don't expect 2012 to be anywhere near as nuts (nor any other year in the near future for that matter if I can help it).

Two, I am no longer on the App Engine team, so all that I am about to say (and will also say in the future as I have a couple of posts to write on App Engine) is from me as just another person using the service. I have been working on a redesign of a website of mine in my spare time and these next couple of posts are based on that work (if you following me on Google+ you know what it is, but I'm not ready for a full-blown Internet debut so I'm not going to link here yet). I don't have any insider knowledge here nor speak for the team.

With all of that out of the way, let's talk about backends! First you need to understand what an instance is to App Engine. When you hit a website hosted on App Engine, an instance is what is serving that request. Now an App Engine app can have anywhere from 0 to thousands of instances up and running at any one time to handle traffic. So think of an instance as a copy of your app that is running on App Engine serving traffic.

Normally you have some time restrictions on how long any of your instances can take to service a request. If it is an HTTP request from the general Internet you have 60 seconds. If it is from a cron job or push queue (i.e. from within your own app) you have 10 minutes. Now both time restrictions are actually rather long and you can accomplish a lot within those restrictions. And in the case of push queues, you can break your work up into multiple tasks so that you really can take as much time as you need as long as you can slice your work into 10 minute time chunks.

But there are just those times when you want something that will take more than 10 minutes, such as a process that is constantly polling some other service to get data. In those instances you want a backend since they are instances that run for as long as you want (assuming App Engine doesn't kill it for misbehaving or some other sysadmin reason). You can have a resident backend which never stops running or a dynamic backend that will stop running when it is done executing, but either type can take about 24 hours to run if it wants.

The way you use instances is to send requests to the backend's subdomain. You name your backends in your backend.yaml file which becomes the subdomain off of your appspot.com subdomain for addressing your backend. So, assuming you have a backend name "forever" for an app named "spam", the domain for the instance will be forever.spam.appspot.com. You can then create URL handlers in your instance and hit those URLs to cause your instance to do it's long-running thing.

Now for those of you who have been using App Engine for a while you might notice that the subdomain for a backend looks just like the specific version ID URL for your app, and that's on purpose. Thanks to it having the same format you can specify a backend name anywhere you can specify a version ID for your app and have it use your backend. E.g. to have a cron job that hits your backend you specify your backend's name as the "version" of your app to hit in your cron.yaml.

And before anyone worries, your backend defaults to being private to your app. If you really want your backend reachable from outside of your app (i.e. the public) you can specify that in your backend.yaml.

While backends sound fantastic and address an old complaint against App Engine and its lack of background processes, I have two warnings/tips about using them.

First, the tip: you can't use the OS X AppEngineLauncher to use backends. When you launch a dev_appserver you need to specify the --backends flag. You can do that manually in the AppEngineLauncher, but you need the flag for appcfg.py as well which you can't specify. So if you want to use backends (and thus want to specify the --backends flag) you will need to use the command-line version of the tools. Obviously this is a minor thing, but it took me quite a while to realize that was why backends were not working for me.

And my warning is that don't expect to use backends on the free quota. While you get 9 free hours of backend usage, you can blow through that surprisingly quickly. But the real issue is all of the other quotas that you have on App Engine. For instance, I was hitting another server to get data to process. What ended up happening is that I was maxing our my incoming bandwidth quota for urlfetch. So then I started to write code to throttle my requests. But even then I occasionally hit the quota. To properly handle that I would need to grab the quota exception and then retry or heavily throttle myself so that I never hit the issue. At that point I was writing code to throttle my requests and to retry when I managed to request to quickly; that's just silly when I have push queues to do that for me automatically. So I personally ended up not using backends in my case (although ironically after I refactored the code I started hitting another free quota limit which led to me turning on billing, so now I'm using $0.05/day in quota, so the app is just costing me $9/month).

But when you are willing to pay for backends they are great for what they provide, but you just have to realize they are just another tool and are not always going to be the best solution for everything (especially if you don't want to pay to use App Engine).

PyCon 2012 CFP ends in a month!

2011-09-12T11:32:00.002-07:00

The title says it all: the CFP for PyCon 2012 ends in a month, so you should seriously start thinking about getting those proposals in! As I say every year, you do not need to be a Python celebrity or have presented previously in order to get a talk accepted. Having been on the PC from the beginning of PyCon I can tell you that we very happily give slots to people we have never heard of. As long as your proposal shows you are organized and your topic is interesting your talk will be given a serious chance for acceptance.

And sorry about the lack of posts. My Python free time is limited at the moment as most free time is spent talking with my wife over a G+ Hangout thanks to US immigration perpetually dragging their feet, keeping her in Toronto and me in SF and causing general misery spread across the continent. That means what little time I do have I spend on trying to get importlib bootstrapped (which you can follow in my sandbox/bcannon repo in the bootstrap_importlib branch) instead of blogging. But once the bootstrapping is done (or I come up with some rather clever thing) I will do a technical post.

How to import a module from just a file path

2011-07-16T15:11:00.001-07:00

Dr. Brown asked on Twitter whether there was a single expression (e.g., no semi-colons or abuses of and or or) that could import a module from a file path, either through a stdlib function call or just constructed from scratch; mod = import_from_path('some_file.txt'). Because it involved import, I got cc'ed on the tweet while various people tried to come up with a solution. In the end people realized that it's not possible in Python 2 (but it is in Python 3). But how hard could it be, right? Right?!?

Well, the devil is always in the details. While there are some functions in the stdlib that come close (e.g., execfile() and runpy.run_path()), none of them return the module, but the globals dict. Other people tried to come up with something using 'exec', but it doesn't return anything (in Python 2) so you have to assign what you planned to use as the module ahead of time (that's two lines total if you weren't counting along).

In Python 3, though, this is all totally doable thanks to 'exec' becoming a function instead of a keyword:

mod = exec(open('path', 'r').read(), __import__('sys').modules.setdefault('name', __import__('imp').new_module(name)).__dict__)

This is obviously a complete hack and not something you should use in the real world. But it is a single expression (meeting Titus' restriction) and it even puts the module into sys.modules so that future imports will not have to re-import the module. This hack, though, does assume that you are only importing modules and never packages.

So what would it take to do a proper function that took (at least) a file path and ended up importing that file? First thing is whether you wanted any arbitrary path to be used, or only something for properly named Python modules (e.g., ending in .py, packages being named __init__.py). If you go with the former you need to specify the module name and whether it is a package (and if it is what the value of __path__ should be), else you can just gleam all of that info from the path itself.

After that the next step is to create the module object and populate it. The bare bones thing to do is create the object and specify the __name__ and __file__ attributes, along with __path__ if it applies. This has to be done as a module might do something with those values as a side-effect of importation (and remember folks, never launch a thread as a side-effect of an import! Deadlock awaits those who don't heed that warning).

Third, you create the code object. Now you could technically just read the file and pass it to 'exec', but that won't set up the file path, etc. in the code object for easier traceback issues that may arise when you do the actual executing of the module code.

Fourth, you actually execute the module code and finish initializing your module.

Now all of this ignores storing the module in sys.modules or even checking if it even exists in there. It also doesn't add the module to sys.modules so that future imports can use the module instead of having to do the same thing again. But if you are honestly wanting to import a module by specifying a path, you probably don't care about any of this anyway. And because you ignore sys.modules, you get to ignore the import lock.

So, taking all of this together, here is a basically untested function that does what Titus basically wanted (works in Python 3, but changing it for Python 2 is literally just using the 'exec' statement instead of function):

import imp

# Step 1
def import_from_path(path, name='', package_path=None):
    """Import a module from a specified file path.

    If the module is a package, set package_path to a list of directories that
    is to become __path__.

    """
    # Step 2
    mod = imp.new_module(name)
    mod.__file__ = path
    if package_path is not None:
        mod.__path__ = package_path
    # Step 3
    with open(path, 'r') as file:
        code = compile(file.read(), path, 'exec', dont_inherit=True)
    # Step 4
    exec(code, mod.__dict__)
    return mod

My personal plans for Python 3.3

2011-06-28T21:34:00.000-07:00

Now that my life has hit another purgatory moment (waiting to here when my wife's permanent residency interview will be and still in the ramp-up period at work), I have had some time to think about what my personal goals will be for Python 3.3. This list is in no particular order and I make no promises to actually do any of it as I only have roughly 249 days until alpha1. =)

First, I want to bootstrap in importlib as the implementation of __import__. I think I can simply freeze importlib._bootstrap, do some imports of built-in modules, and then have it almost work. The reason it is "almost" is that importlib relies on _io, which itself relies on importing the os module. That's a problem when you are trying to essentially import what will act as import. I need to have a look still at the _io code to see if the importing of os can be postponed until after importlib is up and running.

Second, I need to finish PEP 399. Before the whole work/immigration whirlwind started I submitted this PEP to python-dev and got a bunch of feedback on it. The usual push-and-shove led to what seemed to be a general consensus of what the PEP should be striving for, so I think I can wrap it up and get it accepted once I have time to incorporate the feedback (mainly watering down the wording so it is less technical).

Third, I should try to finally get PEP 362 accepted (tracking bug). The code has been up on the Cheeseshop as pep362 for over 4 years now without any complaints. Since I know there are users out there (Michael Foord keeps asking me to finally get the code into the stdlib) I figure the API is generally acceptable and there are no secret bugs creeping in there. Basically I just need to rip out the silly use of inspect that is left since I dropped Python 2 support in the development version, make sure I have proper code coverage, and write docs.

Fourth, I need to get an Amazon or Rackspace instance up that generates a coverage report for the stdlib on some reasonable schedule. This first requires me to finally review and commit Brandon Rhodes' script which gets coverage reporting much earlier in the interpreter startup than it does by default. Once that is done I need to get a VM going somewhere that can freshly check out Python, coverage.py, run the test suite under coverage, and then make the coverage report available for viewing. This will make the test coverage not only more visible, but easier for people to help increase it as a starter project when contributing to Python.

So that is everything I am hoping to do before March 3, 2012. I plan on making the coverage and importlib bootstrapping my priorities, although I will finish PEP 399 first since I hate leaving open PEPs lying about. And I am definitely going to try to not take on any more Python projects until I get through this list as this is already plenty for me to do. =)

Framerates for movies and the debacle that they are

2011-04-22T15:15:00.000-07:00

One of my great weak points is technology, especially those involving movies. While I wouldn't go as far as calling myself a cinephile (e.g., I don't know every random fact about my favorite movies), I do love watching films which means I like trying to make my viewing experience at home rather nice (which honestly has not been that great thanks to my long-lasting student budget).

One of the latest things being debated about the quality of cinema is the frame rate of films. Currently movies project at 24 fps. ATSC (digital broadcasting version of NTSC) is typically 30 fps at 1080p although it can also do 24 fps and the funky rates film and old NTSC video show at (PAL is 25 fps but I'm North American, so I am going to stick with what I know in this blog post). Blu-ray can do 24 fps at 1080p. So everything these days can handle the native projection rate of 24 fps of 35 mm film.

But really, 24 fps?!? Any hardcore gamer is going to tell you that frame rate is unacceptable. You want at least 30, if not 60 fps. As it turns out, so do people watching movies. This is why most decent TVs these days at least output 120Hz video which helps to artificially raise the frame rate and lead to a smoother picture, no matter if the original material is 24 or 30 fps (or even 60, but that is typically reserved for video game output). And even newer TVs can do 240Hz, which leads to an even smoother experience (supposedly; have not experienced it myself).

But what about in the theater? Everything is still filmed and made for 24 fps and that is what projectors (digital and analog) use. Well, two of the more cutting edge directors currently working want to change this. James Cameron has talked about filming Avatar 2 & 3 at 48 or 60 fps. Peter Jackson has gone beyond saying and is actually doing by filming the Hobbit at 48 fps. According to Jackson the higher frame rate not only looks better, but it makes 3D something that is actually bearable to watch (i.e., the nasty flicker goes away). And it turns out most digital projectors can have their firmware upgraded to project films that use these higher frame rates. And with 48 fps being a multiple of 24, these directors could easily film at 48 fps and make 35 mm prints where every other frame is dropped so old projectors can still show at 24 fps.

But what about watching at home? While it's great that seeing those select films I want to see in the theaters will be projected at a higher frame rate (e.g., the Hobbit), what about when I want to watch at home? This is where that 48 fps gets sticky. Both blu-ray and ATSC do not support 48 fps at any resolution. Both do, however, support 60 fps at 720p. And I am willing to bet that Netflix streaming won't handle either 48 or 60 fps until we all have fiber connections. Another issue for the 48 fps is that it is not a common divisor of 120 (Hz), making it potentially nasty to artificially raise the frame rate; luckily 48 is a divisor for 240 (Hz).

So the question becomes, how does Hollywood decide to handle this theater/home frame rate discrepancy that is coming our way? Well, with this being Hollywood, I am willing to bet they do it in the way that screws over the home user the most, forcing you to go to the theater for the best experience. That would support the idea of going with 48 fps in theater projection. That way they still get a higher frame rate that helps out 3D projection, they can easily cut 24 fps versions for 35 mm prints, and they can stick with blu-ray and such at 24 fps as well. All of this while getting to keep a nice hook to get people who care about video quality to come to the theaters to pay for your expensive movie ticket. They can then come out with a new home video standard that supports 48 fps and force us to all buy our movie collections yet again (you know George would be happy to sell you Star Wars for the bazillionth time on a new format, although unless he lets Han shoot first I ain't going for it). If they go 60 fps they can at least do 720p60, but then again do we really want the drop in resolution?

importlib: doing it, and doing it, and doing it well

2011-03-23T18:43:00.000-07:00

Earlier today I pushed a change to CPython which allows importlib to pass Python's test suite as the implementation of __import__ (sans failures because nothing expects __loader__ to be universally set on modules). This means that there is no known compatibility issues standing between me and making importlib CPython's implementation of __import__.

So what was the final hurdle? It basically boils down to code objects and their immutability. When you import from bytecode you are essentially loading a code object which represents the module (basically using marshal.loads()). The issue is that code objects embed what file they were created from (the co_filename attribute). But what happens if you relocate a .pyc file? Should co_filename point to the place the file was originally created at or at its current location? Turns out Python thinks it should be the latter.

The issue with updating the co_filename attribute on a code object is that it can only be done by C code; attributes on a code object are immutable from Python. My original plan was to modify marshal.loads() to take a filename argument which represented where the marshal data came from. That way marshal could fix co_filename while keeping the attribute immutable. But this didn't work.

It turns out that the attribute used by __import__ for fixing co_filename is not as thorough as the recursive solution I came up with for marshal.loads(). There is a loop that looks for objects to fix, but as soon as one of them is accurate the search is stopped by __import__. Rather than replicate this somewhat odd solution for marshal, I simply exposed a private API in imp for me to use in order to mutate a code object's co_filename attribute. Not an elegant solution, but since I am taking a PyPy view to import (i.e., it's my job to make importlib as compatible as possible and not tweak __import__ to fit my needs) I didn't have much of a choice.

Regardless, it feels very satisfying to have an implementation of import that passes the test suite now. This means I can work towards bootstrapping importlib as the implementation of import. I am planning to go with some bytecode-freezing solution for CPython where there is a build rule which recreates the bytecode any time importlib._bootstrap is mutated. That will remove the worry of having to rely upon some external file for import to work (otherwise I could come up with some import shim that did enough to import importlib directly). And with performance of importlib already being acceptable I feel fairly confident that this bootstrapping will happen in Python 3.3 (I have started work in the bootstrap_importlib branch of my personal repo).

PyCon 2011 wrap-up

2011-03-20T17:56:00.000-07:00

tl;dr version: it was awesome! Read on to find out what I did at the sprints and two themes that I found at the conference.

Since I have not had a chance to blog until now, people like Nick Coghlan and Brian Curtin have beaten me to writing good summaries of the VM summit, language summit, conference, and sprints so I won't worry about doing an overall summary of everything.

At the sprints I almost exclusively focused on my Python 3 on PyPI website. After squashing a bunch of bugs I added two more popularity metrics. The first metric is download rate. This is to help balance out against total download count which favours older projects thanks to simply having been available longer. I also added the poll results as found on the front page of www.python.org (look in the upper-right corner of the page). The download rate has about 50% of those projects converted which is great, while the poll results is at 16% which is also very good considering people are explicitly asking for those projects to be converted because they were not at the time of voting. And because people seem to have some odd desire to see the crap code I wrote for this website, you can find it here.

As for the two themes I see gaining more exposure in the upcoming year, one was obviously Python 3 support from projects. There is now a book out by Lennart Regebro [Amazon] (which I wrote the foreword for), a good number of projects have already switched, Django will be switching this summer, and at the sprints Jim Fulton started porting zc.buildout to Python 3 (which happens to be the largest Python 3 holdout according to my website at the moment). In other words the momentum is there for Python 3 and I continue to not worry about it.

The other theme which gained more traction is separating Python the language from CPython the interpreter. PyPy's performance is now good enough to make it a serious consideration for anyone to use. And once they release a version supporting Python 2.7 (IronPython beat them to Python 2.7 compatibility) then they will be in a position to become the de-facto interpreter people use when they don't have specific requirements (e.g., embedding Python in a Java program and thus requiring Jython). What this means (at least to me) is that python-dev is going to need to start to shift towards a stance where the Python language takes precedence over CPython the interpreter. While CPython will continue to be the reference implementation (don't ask if PyPy will become it; I don't think they want that responsibility honestly), certain considerations are going to be needed in order to make sure that things we do in the Hg repository do not end up hurting the other VMs.

To help with this goal of making sure the Python language does not play favourites, I am planning on doing three things. One is a PEP I am drafting up dictating that C extension code in the stdlib must be kept fully compatible with any pure Python equivalent code. It will also say that any extension module with no matching pure Python version must get an exemption to be included in the stdlib. This is to prevent all the VMs from having to re-implement every extension module on their own, essentially leading to four versions of the same module.

The second thing is getting importlib to become the de-facto implementation of import in Python. Every single VM hates implementing import. It's subtle and complicated, making it very tough to implement in a way that is conformant. By moving it to pure Python, importlib will allow all of the VMs to simply use importlib and thus not have to re-implement import.

Lastly, a very long-term goal is to break out Python's stdlib into its own repository. This will make it even more obvious that CPython is the reference implementation of Python and that it is not special in any other way. It will also allow the other VMs to use the same code base to prevent duplicated work (the key contributors on the other VMs are already receiving commit privileges on CPython so they can start fixing bugs now). This is a very long-term project of mine and won't be started until my other two goals mentioned above are finished. In other words my goal is to start at PyCon 2012 with this.

Secret site goes live: Python 3 Support on PyPI

2011-03-04T14:23:00.000-08:00

When the Python 3 Wall of Shame launched a couple weeks ago, I (along with others) noticed discrepancies (discussed in the comments of the announcement blog post) between the real world and what the list said (e.g., doctutils was listed as not supporting Python 3). A similar issue came from On Python 3 Yet? Both websites seemed to be to accidentally spreading FUD about Python 3 support on PyPI thanks to the projects on PyPI not listing Python 3 support properly. While in no way intentional, it does cause issues when people use these sites as "proof" Python 3 still lacks support from the community.

So how did this happen? Well, projects on PyPI are listed as supporting Python 3 when they set the proper Python 3 trove classifier (instructions on how to fix this are in the Python 3 porting HOWTO). What I suspect has happened is that projects which do support Python 3 but don't have the proper classifier either (a) are not aware they should set it or (b) don't care/too lazy. Either way it leads to a mis-representation of what projects do and do not support Python 3.

To deal with this, I created a website to measure Python 3 support on PyPI. What differentiates this website from the others is that I am personally curating the list of projects that support Python 3. So while PyPI says docutils does not support Python 3, my website does. Same goes for projects which have forks that support Python 3 (e.g., setuptools "supports" Python 3 through distribute). I have even begun to flag projects which have working support in their version control system as "maybe" supporting Python 3. I completely understand why the creators of the other websites don't do this; it takes work. But since I have more invested in Python 3 than most people I am willing to put the work in.

If you go to the home page you will notice that I list a project as either supporting, not supporting, or maybe supporting Python 3. The latter case is there because a ton of projects do not specify at all if they even support Python 2! My hope is that projects begin to properly set their trove classifiers. This not only benefits the community by letting people know if they do or do not support Python 3, but because it can be used to know how far back they support Python. Take Django 1.2.5 as an example. It does not have trove classifiers set to let me know that it supports Python 2.4 through Python 2.7. Heck, I don't even know if they have tested against Python 2.7 yet. But if they set their trove classifiers this would be known both visibly and programmatically.

I am asking the community for help combating this problem of projects not listing their Python 3 support properly. If you know of a project which supports Python 3 somehow but that fact is not known on this new website, please either list it over on the Convore discussion on this topic, leave a comment here, or leave a comment on Google Buzz along with proof of the Python 3 support (e.g., a link to an official page stating the support). If you tack on a project's PyPI name on to http://py3ksupport.appspot.com/pypi/ (e.g., http://py3ksupport.appspot.com/pypi/docutils) you can see if the project's support has been picked up yet or not. As I said, forks do qualify for support as well as do projects which have functioning support in version control but have just not done a release yet. This is all rather important for the top projects as listed on the home page since that is what people will really pay attention to in order to notice Python 3 uptake.

To give people some scope of how much of a difference this curation makes, consider the top 50 projects by downloads for any release. 14 of the top 50 (i.e., 28%) support Python 3 somehow. But if you went by only trove classifiers you would think only 9 project (18%) supported Python 3. Even hard coding modules absorbed into Python's stdlib would leave off 2 of the projects. If you look at latest release downloads you have even bigger support and discrepancies: 16 out of 50 projects support Python 3. But of those, 8 I had to manually flag (4 of which are not from the stdlib). Percentage-wise this manual effort makes a difference and gives a much better indication of how much support there already is for Python 3.

So this is what I have been working on for the past two weeks. I learned a lot about App Engine and creating a scalable website. The whole thing updates every 20 minutes from PyPI in terms of project metdata changes. I also update download totals every day. Long term I hope to add more metrics to gauge what the top projects are (e.g., listed dependencies, Google Code searches, etc.). I also want to develop a Chrome extension which will let a user know when a project supports Python 3 when viewing PyPI even if it doesn't say so (maybe even getting to the point that people can notify me through a button click that the project actually does support Python 3). I also want an API so that people on other websites that use different metrics can have access to the same data I use to mark a project as supported or not. But as of right now, I'm ready to take the weekend off. =)

PSF core grant, the last days: the website works (so far)!

2011-03-03T23:18:00.000-08:00

The last couple of days I have been working hard to hack together (and I mean hack; it is not the most elegant code or solution I have ever written) the remaining bits of the website. I am currently waiting for the task queues to finish doing their thing so that I have a complete data set before I go "public" by asking the community for some help with something. But so far things are working out and hopefully I will wake up in the morning to no errors and the data set complete!

The biggest challenge I have run into over the last couple of days is App Engine's 10K limit on task queue data. that ain't much data. So I had to do some reworking of my data syncing workflow to use the datastore as a temporary storage solution. It worked out, but I'm sure I am doing something silly. At least I made sure everything happens in a transaction so that if anything fails the task queue will simply try again until it works (hopefully it's a transient failure and doesn't require me to fix some code =) .

Introducing mnfy

2011-03-01T18:13:00.000-08:00

Remember back in November when I did a blog post about an AST-based minifier for Python code? Remember how I said I gave up on it out of frustration over how the AST represented things in a way that didn't mirror the syntax? Yeah, that "giving up" part didn't sit well with me, so I created mnfy to help minify/obfuscate Python 3 code.

It's basically a little project that I created which helps in minifying code. I got over my issues with the AST by simply thinking it through and figuring out exactly which AST nodes needed special treatment (e.g., how to detect an 'elif' clause). Once I did that I was able to work on some of the transforms which is where the fun is.

The code currently doesn't do anything fancy that human beings who like to muck about with minifying/obfuscating Python don't already do:

Eliminates unneeded whitespace
Minimal parenthesis usage (mostly)
Uses hexadecimal numbers when it will save on characters
Combines imports into a single line that are defined in a row
Drop unused constants (including docstrings)
Functions into lambda definitions

As I said, nothing nuts, but at least takes some of the tedium out. The transformations of the syntax are also separated into safe and unsafe transforms. I consider safe transforms things that are semantically equivalent to what was changed in 99% of the cases (i.e., if you muck with execution frames you are on your own). Unsafe transforms will work in 90% of the cases, but are not exactly semantically equivalent (e.g., lambdas are not entirely equivalent to a defined function). Thus users can choose their exposure to transforms which may or may not break their code. You can also skip transforms all together and simply go with the elimination of whitespace and unneeded formatting characters (e.g., parentheses). In the latter case I am able to minify the entire standard library and still get back the exact same AST.

Anyway, this was just a fun little project that I don't plan to take that seriously (i.e., don't expect massive churn on the code any time soon =). If either minification/obfuscation or working with Python's AST interests you, there is a talk on using Python's ast module and another on obfuscating Python code at PyCon you can attend.

PSF core grant, day 39: darn scalability

2011-03-01T09:35:00.000-08:00

So the server component for the secret website is nearly complete, when I started to test the site through App Engine's dev appserver, some issues arose which were not triggered by unit tests. So I spent Sunday and yesterday (counting as one day thanks to working on bursts on-and-off) working on fixing those issues.

Otherwise the only other news is that the Python-Dev In a Box code is up. Please realize that it will not work for you! It is currently geared towards using the Hg test repo which is not ready for general use (e.g., it's missing some commits that landed in svn). I also am going on an assumption a patch I submitted to coverage.py about a __main__.py file for the repo directory gets accepted, and so I have a file I manually copy over in that instance until that happens. In other words, if you decide to play with the code realize that it probably won't work and so bug reports about it not running are probably premature. =)

PSF core grant, days 37 & 38: making Python-Dev In a Box a reality

2011-02-26T22:32:00.000-08:00

I got enough positive feedback on the Python-Dev In a Box idea from my last blog post that I decided to implement it. Over yesterday and today I have been writing some scripts to help bootstrap people into contributing. I will make the code and repository public as soon as I can get the remote repository at hg.python.org created.

So the goal of this project was to make it easy for people to jump in and start contributing at a sprint or on their way home from a conference. That meant providing the code, tools, and information needed to contribute. It also meant doing as much upfront for people as possible so as to not waste time for what everyone would end up doing along with not requiring a network connection. And all of this, at max, needs to fit in 700 MB on a CD.

First, there is a script called make_a_box.py which creates a Python-Dev In a Box instance. It will prompt you for what you want to include in the box:

CPython (from an Hg repository at the moment, but I might patch it to be from svn if Hg is not ready by PyCon)
Devguide
PEPs
coverage.py
Mercurial and TortoiseHg
Visual C++ Express (the Web installer)

When reasonable it will build what is provided. In the case of CPython and the various bits of documentation, the docs get built. For coverage.py it runs it so you have a basic coverage report for people to work off of.

For users, I included two extra scripts. One simply builds CPython (on UNIX). While building is very straightfoward (and documented in the devguide), I needed the functionality for generating the coverage report. Since I needed the code anyway I figured I would make it a stand-alone script. About the only real perk to it is that it uses multiprocessing.cpu_count() to run the Makefile with as many cores as you have (although CPython compiles in a fairly linear fashion so extra cores doesn't do that much).

I also provided a script that runs the test suite in the fastest, most rigorous way possible. Once again, while documented in the devguide, people can get a little lazy when it comes to running the entire test suite, so hopefully having a script that uses the "best" options will lead to more rigorous testing overall.

My plan is to create a Box for the PyCon sprints so that people attending the sprint can simply copy from a CD or flash drive everything they need to get going. This should speed up people getting started, leading to more productivity.

PSF core grant, day 36: maintaining a website is a pain, Hg, and Python-Dev In a Box

2011-02-24T19:20:00.000-08:00

I'm this close to having the data maintenance aspect of the "secret" website done, but came up short when I forgot that entities in App Engine cannot be stored in a transaction unless they are in the same entity group. Since I have tens of thousands of entities, putting them all in a single entity group seems stupid, so I will simply have to break out another worker to be called from a task queue in order to process each entity one by one (at least I can still batch the external network calls). Bah. So close!

While nothing to do directly with me, on the behalf of himself and Georg Brandl, Antoine Pitrou announced a test Mercurial repository for (C)Python. I'm obviously excited about this since I spearheaded this whole ~~debacle~~ movement to switch from svn to hg back in late 2008, making the decision of hg with Guido at PyCon 2009. It looks like there is a slight chance we might actually get this done in the near future (since Antoine and Georg are pushing this the last little distance I can't speak to whether they will get it done by PyCon 2011, but it would be neat if they did).

A rather nice side-effect of switching to Mercurial is the smaller amount of space that a Mercurial repo of Python takes up compared to Subversion. This led to a slight discussion #python-dev about creating a Python-Dev In a Box. The idea is that to help spur contributors one could put the Mercurial repo, the devguide (pre-built, but also along with the hg repo), Mercurial itself (including TortoiseHg), pre-build the docs (so that people have a copy of Sphinx and such which can also be used to build , and anything else I could think of (PEPs checkout? Visual Studio Express installer? coverage.py along with coverage results?) on a CD or flash drive so that people have everything they need to start hacking on Python. And by putting it on physical media it makes it easy to bootstrap everyone in the room quickly without slamming the network. It also provides everyone even the ability to get going while offline (e.g., "Can't stay for the sprint? Take a CD and start on the plane home (if you are not on Windows as that requires downloading Visual Studio Express)!"). Hell, think if this became a bag stuffer at PyCon! I wonder if that would increase contributions?

PSF core grant, days 33, 34, & 35: Python 3.2.0, fighting personal feature creep

2011-02-23T17:52:00.000-08:00

First off, for those of you who don't know, Python 3.2.0 was released on February 20th (which serendipitously happens to correspond to the 20 year anniversary of Python 0.9, the first release of Python). To celebrate Python being a score old, why don't you help port a project to Python 3.2.0? =)

With Python 3.2.0 out the door, the development branch of Python was open to commits again. I was able to commit all but one of my patch queue I have been sitting on for over a month. This means that the in-development branch of Python can now run the test suite under coverage.py without any special patches. I also fixed a bunch of static analysis warnings found by Clang. I am ignoring the time burned on trying to fix test_zlib failing under OS X as people much wiser than I at mmap solved that one. I also fixed the new crypt module changes to be PEP 8 compliant and simplified the API a little.

As for the secret project, I am working on not letting feature creep get the better of me. I scaled it back slightly so that it won't start out life with as much stuff since it didn't need it everything. But then I also thought of a cool way to get more people to participate with the website through a Chrome extension. Out with one feature, in with another.

But I am reaching the ability to launch in private beta in about a week. I don't see why I won't have this ready to go for public consumption (in a rather rough state) by PyCon.

PSF core grant, day 30, 31, and 32: lots of testing, moving on to data maintenance

2011-02-20T20:13:00.000-08:00

In a previous blog post I mentioned my refactoring/testing strategy for the website I am developing. Friday was all about implementing that strategy. Even without touching the network, my tests were good enough to lead to only two very shallow bugs that needed fixing. At that point I am basically caught up in my testing regimen.

That means new features! But yesterday and today have been a back-and-forth battle between trying to do a good job and worrying too much about rare edge cases and features that are not needed. Since the data only needs to portray details in a rough level and only for a subset of the available data, I realized I need to relax and not sweat every single little detail that could come up. It's just as easy to run an occasional sanity check to rectify any data drift that may come up.

And a tip to those using nose-GAE, if you get an odd error about not being able to delete a datastore file, it probably means you have a syntax error somewhere.

And a tip for those using Google Chrome: be aware that if you are manually triggering things through a GET request, Chrome's prefetching of pages can cause you to accidentally request twice. This was a problem for me as I was doubling the stuff I was putting on task queues and it took me a while to realize why.

Register for PyCon NOW if you actually plan to attend

2011-02-17T22:18:00.000-08:00

A trusted chair of PyCon US 2011, Van Lindberg, did a post explaining some of the behind-the-scenes issues that must be worked out when organizing the conference. It's an interesting post that's worth reading.

But one of the key take-aways from that post is that the hotel space is going quickly. We have already taken 105% of our allotted space. It's already projected that we will sell out PyCon this year. Make sure you register if you have not already else you may very well find yourself unable to attend no matter how badly you want to come.

PSF core grant, days 28 & 29: learning the hard way that you need tests

2011-02-17T20:04:00.000-08:00

My little skunkworks website got cleared to be worked on, so I have shifted my focus to it. I think I am going to keep it a secret until PyCon and announce it during a lightning talk for fun. But just because the subject of my work is a secret doesn't mean the process of creating the website needs to be kept from the public (even if it means I have to admit to what I am doing early; being secretive is not exactly critical).

I must admit that most of my experience in creating websites is relegated to either simple stuff or more JavaScript-heavy work. This is turning out to be the first website where there is a serious back-end that has offline workers, constantly updating data, etc. In other words it is a learning experience for me.

The first lesson is that I can't be sloppy with this website. Back on Sunday I started to slap together the website on Google App Engine on my machine, simply coding away until I thought I reached a point where I could actually examine some data. I was using task queues for the first time and trying to coordinate all of this communication with various queues running at various rates with different workers, etc. In other words I over-engineered.

But even worse is that I was not doing any proper testing. I figured I might just do some hand-driven tests on a subset of data to make sure things were working out. But my slap-dash coding quickly became a hindrance as I ran into several import errors thanks to me not paying attention and making sure to add the proper import statement. With the turn-around of launching a page which populated a task queue being long, I knew that I just couldn't keep going like this.

And so I knew I had to recode for testing. But how the heck do you test a website, let alone an App Engine website? I mean I know how to write proper unit tests, functional tests, etc., but I didn't know the best way to handle that for a website that has to run under some heavy infrastructure that has to be available.

This called for some research to even get tests executing. I stumbled across nose-gae which seemed to give me what I needed to get the tests executed if I was willing to use nose (which I was). I even tried to be a good little developer and work under virtualenv, but I ran up against odd path issues involving App Engine that I simply did not want to try to diagnose after an attempt to use gaeunit left me wanting (not enough traceback info to debug well). Luckily nose and nose-gae were the only things I felt the need to actually install so I didn't totally screw up my Python 2.5 install.

With tests runnable, I then had to decide how the heck I was going to do this. And that's when I decided I needed to refactor the hell out of my code. Being an App Engine app that is (at least currently) using no web framework, every URL resolves to a request handler class. That means there is at least a get() or post() method on each class. So I made the decision that all request handling (e.g., getting what is in some POST argument but not even decoding the JSON) and all response handling (e.g., writing back out any HTML) would be handled in the get/post method, but nothing more; essentially they are gutted to the point of simply stitching together method calls, as one typically should when there is I/O involved. I then broke code down into helper methods as necessary in order to think about the functionality in the way I needed to. I also put all transaction code in its own method since I am purposefully keeping that work to a computational minimum. This allows me to mock out things like task queue usage in order to verify what I want to happen is going on without worrying about App Engine actually using some task queue and having to clean that up manually later (handling actual datastore work is no big deal so I am happy to let that actually occur with proper test cleanup).

All of this leads to me writing abstracted, modular code where I can test non-network, non-App Engine code quickly and easily and then purposefully cleaning up or mock out as needed when I do need to work with App Engine's assets. At this point I have not written tests for the actual website page requests as the get()/post() methods as they are dirt-simple and they are just being called by App Engine task queue events (but eventually I will once I decide how I want to handle that specific case; webtest?).

What is the lesson in all of this? Don't be sloppy. Assume you will have to test your code eventually, forcing you to write code in a fashion that makes the code easy to test. And then actually bother to test your code. Dealing with the network or APIs that have no way to undo or cancel an event programmatically (e.g., something in a task queue) should not stop you from writing tests as there are ways to isolate these things such that they are not a burden.

PSF core grant, day 27: little details, a secret, and a progress report

2011-02-14T21:56:00.000-08:00

Started out the day creating a patch to link from the issue tracker to the devguide. There have been a couple people who have asked me to have the issue tracker fields link to the devguide to give a better explanation. Plus updating the docs is easier than the tracker.

But what really took up most of my time today (and yesterday, but I had errands throughout today so I am counting this as a day's work) is a little skunkworks project I started working on related to Python 3 and combating some (accidental) FUD that seems to keep coming up that really needs to get squashed.

Anyway, I also figured I would summarize what the heck I have done as part of my grant as I have finished day 27 of 41 (and I need to summarize for the PSF board this week). I completed the devguide, which I have wanted to do for literally years. There is also a branch of it already prepared for when the switch to Hg occurs. This also led to various patches that make sure the stdlib can have its test coverage measured, fixed a bunch of static analysis warnings from Clang, and have import failures not implicitly represent a skipped test.

I wrote a HOWTO on how to port a Python 2 project to Python 3. Now the community finally has a single document to read to at least get started in porting instead of having to pull up various blog posts on the subject.

All of this and I am three days behind "schedule" of ending my grant come March 1 (unless my math is wrong and this really isn't the 30th weekday since January 4th, inclusive). Overall I'm happy with what I have done and I am hoping the remaining 14 days are just as productive.

PSF core grant, days 25 & 26: unexpected skips & python.org/dev/ no more

2011-02-11T18:35:00.000-08:00

The theme for the past two days has been about removing things. Yesterday was about issue 10966 and finishing the patch to remove the concept of (un)expected skipped tests. Today was about gutting http://www.python.org/dev/ to properly represent the fact that almost all of its content has been subsumed by http://docs.python.org/devguide/. Once the website rebuilds proper redirects will be in place to redirect from the old content to the devguide.

PSF core grant, days 23 & 24: prepping for hg, cleaning up test skipping

2011-02-09T18:06:00.000-08:00

Sorry for being behind on updates. Monday was final paperwork on the PhD (unless the university tells me otherwise I am now done with everything), yesterday it just got too late to worry about posting, and then this morning I just got in a groove and didn't want to stop to post. So today's post covers work done both today and yesterday.

For preparing for the transition from svn to hg, me and several other core developers have a preliminary draft of the devguide ported over. It currently suggests using mq, but that may change to feature clones if it turns out that the mq instructions are too hard to follow.

For cleaning up test skipping, I have begun work on issue 10966. I am basically trying to make it an error for extension modules to fail when they are not compiled on platforms they are expected for. This has been an issue in the past as compilation errors have gone unnoticed as tests were flagged as skipped instead of as a failure simply because the import failed. This will also do away with the idea of expected and unexpected test skips as the test code itself will state on what platforms a test is required/optional on.

PSF core grant, day 22: Python 2/3 porting HOWTO goes live, starting to doc for Hg transition

2011-02-05T19:14:00.000-08:00

As promised the other day, the Python 2/3 porting HOWTO is now live. Yesterday I fixed some things based on feedback from people that I committed today (as well as others contributing their own fixes).

Working on the HOWTO in svn has reminded me how much I prefer hg thanks to all of my work on the devguide. Related to that, I started a branch of the devguide which outlines using hg instead of svn once the transition occurs. I have done all of the easy updates so far, leaving the tough one of outlining a basic workflow which includes back/forward-porting changes.

This whole bit of work stemmed from a massive python-committers thread on the topic. Basically people are trying to decide how best to structure the workflow of python-dev. In svn the way things work is that everything is committed in py3k unless the branch is frozen for cutting a release. If we are in an RC then you have to get approval to commit changes, otherwise you sit on your change until py3k opens up again. As for porting changes between versions, you commit in py3k and then use svnmerge to backport to e.g., release-31maint.

For hg the thinking is to tweak this. It's quite possible that when an RC is reached, a branch will be created that no one but the release manager can touch. When there is a fix in py3k that should be in the RC, the RM (release manager) can cherry-pick the commit and pull it in. While this might seem to create more work for the RM by having them need to execute some cherry-picking command, it does mean they don't need to watch commits that need to be reverted and should also make committers think twice about what they want to bug the RM about.

As for porting, we will most likely switch to a forward-porting strategy. So people would apply a fix in release-31maint and then pull into py3k. The amount of work should be no worse or better than it is with svnmerge, but the DAG from hg should allow for making more sense of what exactly is going on in the history. Plus it makes doing a blanket pull much easier. Also, people will have to make the decision upfront as to whether something needs backporting. Granted, some people might still refuse to put the effort in to backport, but at least for those of us who do it should be at least equal, if not better than, using svnmerge.

But I need to write all of this up and get the resident hg experts to agree on all of this first.

PSF core grant, day 21: new Python 2/3 porting HOWTO, fixing static analysis warnings

2011-02-03T17:33:00.000-08:00

Big news today (at least for me) is that Georg let me check in the Python 2/3 porting HOWTO into py3k so it will be included with Python 3.2. Unfortunately I have no link to give out at the moment as the docs have not done their daily rebuild yet.

The other thing that I did was re-run Clang's static analyzer over CPython with certain function annotated as 'noreturn' and use their newest release. In the end this led to a bunch of dead code being removed and one bug being found in sqlite3.

PSF core grant, day 20: writin' them words, killin' them assignments

2011-02-02T20:09:00.000-08:00

Since my dissertation was formally accepted by UBC yesterday, I spent part of yesterday just trying to grasp the fact that I finished my PhD, and part of today dealing with paperwork from my department to prove that I don't owe them anything.

But between the two days I managed to get some stuff done. I finished the initial draft of my Python 2/3 porting guide. I am waiting to hear whether I can sneak it into Python 3.2 or not.

I also managed to fix all of the dead assignments detected by clang's static analyzer.