Coder Who Says Py

My (very shallow) thoughts on Dart

2012-05-13T21:11:00.002-07:00

Being the language nerd that I am, I actually find it fun to learn new programming languages. Now typically this is nothing more than me reading all of the official documentation and writing some toy examples that give me a very shallow, quick-and-dirty feel for a language. Since I have been involved in language design for nearly a decade (started participating on python-dev in June 2002) and have done toy examples now in 18 languages (17 actually still run; I have never bothered to get Forth to work again after a gforth change broke my code), this is actually usually enough for me to grasp the inspirations for a language and thus understand its essence.

At work I have been doing some JavaScript work for an internal Chrome extension and dashboard and so that led me to want to look into what Dart had to offer over JavaScript. I know the language is only at version 0.09 (and still changing weekly), but the fundamentals are there so I wanted to see what the general feel of the language is (and will continue to be).

I also know Dart is somewhat controversial for some people. Personally, I fall on the "competition is good" side of the argument, not the "OMG fragmentation" side. I want ECMAScript Harmony to still happen and give me a cleaner, tighter, more functional JavaScript, but that doesn't mean Dart doesn't have a place in the world as a cleaner OO language for the web. Besides, me thinking otherwise would make me a massive hypocrite as I began working on Python before it was cool (I feel like I need a hipster meme for that statement, but I digress) and I have worked hard to convert people to Python from other languages. Hell, I have tried to foster competition between the Python VMs to get them to push each other to perform better and be ever more interoperable. IOW I don't totally buy this fragmentation argument.

Going into learning Dart I knew who was involved with the language which is what will inherently define how a language feels. I knew Lars Bak of V8 helped design the language, which meant it would have some design restrictions put on it to make it have a damn fast VM. Josh Bloch has been helping to design Dart's library which meant some JDK feel to it. I also know Jim Hugunin is involved which should also help with the VM speed. So fast with an API designed like the JDK.

What did I find? A language with a damn fast VM and a standard library that felt like the JDK. =) Take OO as a Python programmer would expect (e.g. pure OO where everything is an object, not dogmatic OO like Java where everything has to be in an class definition), make types entirely optional for testing and tooling purposes but enough support to use interfaces and generics, and then toss in abilities based on what JavaScript allows and then you have a good idea of what Dart offers.

So, Dart has optional typing. In case you have not heard, Dart does not use type information at runtime for performance and only throws any form of fit if a type doesn't match what is specified unless you run in checked mode. If you do that then you get warnings about possible type issues. But Dart's type system is unsound so don't expect typing to catch every error that a more strict type system might even when you run in checked mode. Dart views types as helpful documentation and a way to help tools assist with things, period. I actually find it rather refreshing to have a language that treats types as just documentation since that is really what they are for the programmer (VMs can use it for performance, but it isn't required for good performance and type safety only saves you from a minor set of bugs which every Python programmer probably realizes eventually =).

But that's even if you bother with types! You can write all of your code without types and everything will run without issue. Even generics are optional, so you can declare a function accepts a List or List; Dart doesn't care either way and it alleviates covariance/contravariance headaches by not caring if you don't care either. It's actually rather nice to have non-library code be written quickly using dynamic typing and only add in the type information for library code where you care about what interface is expected. IOW I think Dart strike a nice balance with how it does typing and I actually feel fine using types when I know what I expect to accept in my own code that I don't expect anyone else to rely upon.

Dart is OO, not prototypical like JavaScript. It's single-inheritance, which I'm fine with. It does have interfaces as one would expect in a statically typed language, but it softens their expense by allowing one to define a default implementation of an interface. What this means is that the Map interface will also give you a HashMap instance if you call new Map(). I suspect they snagged the idea from Scala where you have the Map class which hides HashMap from the user if you simply don't care about what Map implementation you use.

It does have a modicum of privacy by using a leading underscore for signaling something is private, much like Python. But the privacy is enforced at the library-level or is public, period. Every field automatically has a getter and setter defined for them, so there is no way to force a private field (which I think is a good thing since I find private privacy bloody annoying). I also like that getters and setters are directly supported by the language with automatic generation show you don't ever have to see a setSomething()/getSomething() function call just to read/write a field, but you can do something like Python's properties very easily.

The standard libraries are fine and just feel like the JDK. Things are very much LBYL rather than EAFP. I am willing to bet (although I have not tested this) that exceptions are a little expensive in Dart (since exceptions are hard to optimize) and so they would rather go the LBYL way. But they still went a little overboard in my opinion on some things (e.g. the list interface has a last() method instead of supporting negative indexes). But there is nothing there that is making me run away screaming.

One place I do think Dart could use some improvement is simplifying their constructor rules. Upfront Dart has some nice syntactic sugar for a construction where you directly specify how a constructor's arguments map to instance fields, avoiding having to declare the constructor parameters and then also write an assignment. OK, I like that.

Dart also has initializer lists which let you initialize final fields. OK, that's cool and a nice idea taken from C++.

Constructors are not inherited. OK, that's fine since you probably want to be explicit about how you tweak stuff. But there is an exception about the default, no-argument constructor calling the superclass' no-argument constructor. So while not technically inherited, it might as well be in that single instance. And all defined constructors will automatically call the default constructor, which if it isn't defined you must explicitly call a constructor somehow (probably in the initializer list of your constructor). Um, OK...

And you have named constructors. This gets you around from the lack of type-based method overloading for constructors. OK, I can go with that.

You also have constant constructors since fields can only be initialized to compile-constant values. Fine, that's for performance and determinism in instance creation, so I can grasp the desire for that.

And then you have factory constructors. OK, this is where I go "WTF people". This is so that you can have a constructor that actually doesn't create a new instance but instead can return something else other than a new instance (think of Python's __new__() or any of Java's static factory methods). But this lets you use the new keyword on a factory constructor instead of using a static method. And that to me seems unneeded.

So lets recap what constructor options we have. We have regular constructors, default and defined, which supports initialize lists. You have named constructors. There are constant constructors. And you also have factory constructors. If you don't count the default constructor as special that means Dart has four types of constructors. WTF!?! I realize that Java's FactoryFactoryOfFactories crap has probably spooked the crap out of the Dart designers, all the while having Java influences making them think they need the new keyword for anything that would return an instance of a class, but this seems a bit much. Dart's function definitions are rich enough to allow for optional arguments, etc. which would suggest that the typical constructor can do the job of named constructors with static methods picking up the slack where absolutely necessary where factory constructors are used. Maybe I'm missing something here, but I think they tried to design for everything that is bad about Java's constructor mess without stopping to think what their function definitions already buy them, all while making sure the new keyword was used.

Luckily that is the only bit of Dart that I found poorly designed. Everything else is reasonable and something any JavaScript programmer will be somewhat familiar with or quickly grasp.

Now as I said, I only did toy examples in Dart beyond reading the docs from beginning to end. If I had more time this weekend I may have done one more coding example that was more involved, but I ran out of time. But based on what I have read and what I learned, I am happy with Dart and would be content in using it for programming for the Internet. I would also be totally happy being asked to use it in a situation where others wanted to use types (e.g. I would be fine ditching Java for Dart if people really felt the need to hold on to their types).

Thoughts on using function signatures as a DSL for CLI parsers

2012-05-12T13:02:00.002-07:00

I have no idea why, but this morning I thought about a decorator for delineating what function should be treated as the main function (e.g. using a decorator instead of the traditional if __name__ == '__main__' idiom). Now I solved it in my head on the spot, and then immediately realized someone had to have solved this already. Turns out various people have done things as nuts as examine stack levels to detect the __main__ name, but the most straight-forward solution I found doesn't do anything nearly as nuts or CPython-specific and is basically what I came up with. There was a red herring, though, in everyone's solution where they claim the decorator has to be on the last function in your module. While technically true when using the decorator as a decorator only, you can also just as easily not decorate the function and instead, at the end of your module, do something like main(func) since that is the same as decorating func with main.

A really simple expansion of this idea of helping out with defining what function is the main function, is to pass in sys.argv and to return a value to signify exit status: sys.exit(func(sys.argv[1:])). So now you have made the decorator more useful than replacing the old __name__ idiom.

But while that is nice and helps deal with the very common case, I wanted more. Why can't you introspect on the arguments the function takes and use that to automatically generate a command-line parser? I did a search and the best I could find is entrypoint, but it doesn't go far enough for me. What I want is to use the full expressiveness of function parameters in Python to express as much about what should/could be given on the command-line along with passing in as little as possible to the decorator in order to replicate the common case of command-line parsing; think just as easy as getopt but more powerful by using as much of argparse as you can without coming up with complicated rules about how things should work (since once you pass a certain complexity threshold you should just build the argument parser using argparse's API directly and stop trying to optimize for it like I'm suggesting).

So what do we have at our disposal to build such a decorator? We have positional arguments so we know how many arguments are required without some specific qualifier. We have variable positional arguments (e.g. *args) to take an optional number of extra arguments at the end of the command-line. We have keyword arguments which are optional flags that one can specify. You could even have variable keyword arguments for major flexibility, but that just seems like a total lack of structure the CLIs just don't typically provide. With all of that you can reproduce getopt without any issue for long-form names. For short names, I would say you need to pass in a mapping of short names to long names into the decorator. Same goes for long names to help string (you can use the function's docstring for the main help for the app itself).

But where things get really interesting is when you take into consideration function annotations. That opens up the possibility of going beyond getopt and potentially supporting argparse's action, nargs, and type options. Take the type option as an example. You could say limit:int=10 to have a command-line option called --limit which only accepted an integer and defaulted to 10. This obviously could also work with float or any other type where you can just pass in a string to the constructor to get back an instance of the type. So you have a general case which can be useful, but you can you potentially special-case some things to get enhanced functionality where it doesn't make sense to simply take in a string?

Lists pose an interesting option as argparse provides both nargs for specifying the number of arguments to a single option, or the append action for accepting multiple instances of the same option and accumulating them. In my mind both can be expressed in a way that I think makes sense but some might view as too magical. If you specify names:list=[], then that supports the append action, e.g. --names Brett --names Andrea leads to names being set to ['Brett', 'Andrea']. But if you were to do names:['+']=[], then that would get the same result from --names Brett Andrea. In other words, the list type specifies the append action while a list instance specifies using the nargs option with the single item in the list acting as the value to set to nargs.

For booleans, I would want the use of the bool type to mean use either the store_true or store_false action based on what the default argument was. So turn_on:bool=True would use the store_false action since the argument is meant to be a boolean and it's default value is True, meaning that if the option was specified it represents the reverse.

Finally, the tricky bit is for files since that is a common command-line argument and you might as well open the file and close it for the function. The solution argparse uses is a specific FileType class where you can pass specific arguments to use when opening the file. The problem is that it doesn't support everything open() does, e.g. encoding. So what I would want to do instead is provide a partial function that took everything but the file path and then when it came time to call the main function, passed in the file path to the partial function, passed the returned file to contextlib.closing(), and then passed it on to the main function. You could even generalize a lot of this and simply say that whatever is specified as the function annotation, if it isn't a special-case like lists, then you call the annotation with what came from the command-line and if it provides a context manager it is used before calling the main function.

So those are my thoughts on using function parameters as a DSL for getopt++/argparse-- functionality on a Saturday morning. Honestly the most complicated bit would be constructing the arguments to pass to the main function in the right order, otherwise it's just introspecting on a function's parameters and making the proper call to argparse. But then again the real question is whether anyone thinks this at all sounds reasonable enough to code it up.

Playing with the Ninja build system

2012-04-28T20:21:00.001-07:00

Whenever I learn a new programming language I end up writing some toy examples to try to get a feel for what the language is about. This leads to the need to build code using many different compilers with their own flags, quirks, etc. Up until today I had used SCons for my build setup. But honestly, it always seemed like overkill to me. Because I only had about 5 programs to build per language with at most two files used to produce the program, a full-blown build system was never really needed. Add to the fact that I am building for languages that no build system would have built-in support for, it led me to always have a wandering eye for another build system I could use.

This past week someone on Google+ shared a post comparing configure+make, cmake+make, and cmake+ninja. I had never heard of Ninja, so I decided to have a look. It turns out someone had written a build tool whose only explicit job was to take a DAG, figure out what needed to be built, and then execute the commands for the build. No crazy metadata checks like Make, or fanciful features, just bare-bones building. Ninja was actually designed to be a target for other higher-level build systems like cmake which can do the pre-computation of what the DAG should be, leaving it to Ninja to drive the needed compilation.

What attracted me to it was that it was fast and the syntax was simple. I have code examples for 16 languages, of which 10 have build rules (one happens to be Python 2.7 as I pre-compile the .pyo files). Turned out to be a pretty straight-forward process to take my custom SCons commands and just translate them to the subsequent shell commands that Ninja would execute for me. They are a tad verbose in order to make sure that the ninja -t clean command would clean up all intermediary files (I'm looking at you OCaml, Haskell, Java, and Scala). But as I said, I typically never have more than 5 programs to build per language, so it wasn't that much of a burden. And if I really cared I could have written a Python script to auto-generate the Ninja files for me, but I decided the effort of writing the code would be just as much as writing the build files by hand.

I realize I could have used Make, but I honestly am not enamoured with that tool; requiring tabs just rubs me the wrong way. Plus it's rather slow in the common case of only changing a file or two compared to a complete build from scratch.

Overall, for my weird case Ninja worked out. For something more complex, though, I will consider looking at cmake+ninja as a build solution.

The re-launch of py3ksupport!

2012-02-14T15:29:00.001-08:00

The reason the past few blog posts I have written have been App Engine-themed is because I have re-launched py3ksupport! I did a complete rewrite of the code to make it more efficient (since I'm paying $9/month for the site) and at the same time moved over to HRD so as to guarantee the site is always up.

Before I discuss some unique features of py3ksupport, I want to point out that right now that 56 - 60% of the top 50 projects based on downloads of their latest PyPI release support Python 3. The reason for the range is that some projects had in-development support last time I looked and since that can change underneath me I wanted to cover the possibility the data was stale. But the key point is that 8 of the top 10 projects support Python 3 and one of them has support under development along with over half of the top 50 projects.

So I'm sure the site explains itself (and the FAQ fills in gaps), but I figured I should explain some of the more unique features of the site. One is the metadata rating given to each project. Basically I wanted to shame project owners into updating their project metadata. LOTS of projects don't bother to specify the Python support metadata for their projects which makes my life difficult and is unfortunate for users. For instance, of projects bothered to specify the exact versions of Python they support then users could easily tell from the Cheeseshop (nee PyPI) whether they could use the project based on what version of Python they are tied to. I might have to do some public shaming at PyCon if the situation doesn't improve. =)

The other key point is that I personally keep the front page up to date. If a new project shows up on the front page that does not have the proper metadata specified then I personally search online to find out the status of the Python 3 support. Since I get emailed each day when this happens the situation tends to get fixed that day and will be noticed within an hour of me fixing it (the length of time I have things cached).

I'm hoping to eventually move to measuring an project's popularity based on the download rate for the lifetime of the application (instead of just the latest release) to give a more accurate reflection of how popular a project is.

How I bootstrapped importlib

2012-02-06T16:44:00.000-08:00

If you have been reading this blog over the past five years I am sure you have read a post or five about my desire to bootstrap importlib into Python as the implementation of __import__. Well, as of today I'm willing to say that the difficult technological hurdles have been scaled! At this point the only thing holding me back from taking my code from https://hg.python.org/sandbox/bcannon#bootstrap_importlib and making importlib drive import statements are some small compatibility issues, integrating into the build process better, a code review, and python-dev sign-off. In other words all of the interesting problems have been solved, so I'm finally ready to write a blog post discussing how I pulled off what I have.

So how exactly do you import __import__? To begin, as with any bootstrap challenge, you need to figure out what is available to you so you know what your design parameters are. In my case I knew I couldn't import anything that required filesystem access since half of import is handling the search for a module (the other half is the actual importing); if I wanted to import a file I would need to essentially write half of import in C to work properly. This restriction also has unexpected side-effects, e.g. you can't rely on open() because that is part of the io module which is a Python module.

That meant I could only rely on built-in modules. If you run sys.builtin_module_names you will discover what is available directly within the CPython binary. The question then becomes if that is enough? It turns out that yes, those built-in modules are enough to perform an import. OK, so you know you have the bare minimum modules required to do an import, but how the heck do you get the built-in modules into the global scope of the module that imports module since you can't use an import statements?

This is when Python's dynamism comes in handy. Since the import statement doesn't do much more than pull in the module object and assign it to a variable at the global scope of the module, I just needed to get the module object for importlib and assign to its __dict__ the built-in modules I needed. Turns out that sys and imp are enough to allow importlib to handle the import of the rest of the built-in modules needed for import to work, so that kept this bit of code short.

But this brings up the next quandry: how do I create a module object of importlib? If I end up searching for importlib on sys.modules then I would have ended up implementing a decent chunk of import itself. So how could I get the module object? This is when frozen modules comes into play.

A frozen module is just a C array containing the marshaled code for a module (which is what a .pyc file is sans magic number, timestamp, and now file size of the source). Since marshal is a built-in module then frozen modules can be loaded without issue. That means you can load a frozen module without using import (much like importing built-in modules).

And that is all of the parts needed to import importlib w/o import. =) To summarize, you get importlib set as __import__ by doing the following:

Import the frozen module (i.e. read in a C array of a marshaled module object and unmarshal it)
Import sys and imp (built-in modules, so done in C code by calling key C functions which return module objects) and set it on the module object
Call Python code to import the rest of the built-in modules using sys and imp
Set Python-based __import__ on the builtins module

And voila! __import__ ends up implemented in pure Python code. Now I just need to clean up the code, fix the compatibility issues, rip out the old C code, and get python-dev to sign off. =) Hopefully I will get far enough I will have a lightning talk at PyCon with benchmark numbers to show this is actually all a good thing (including ripping out a ton of C code, especially if I can re-implement chunks of imp in pure Python =).

Asynchronous XML-RPC in Python

2012-01-26T09:58:00.002-08:00

Do you use XML-RPC (and specifically the xmlrpclib/xmlrpc.client from Python's stdlib)? Do you like multi-calls? Wish you could construct your XML-RPC multi-calls in a way so that you could make them asynchronous by constructing the call from scratch? Then you're in luck because I already did the hard work of figuring out the details for you! =)

In my case I needed to communicate with an XML-RPC server that supported multi-calls, but in an asynchronous fashion with a non-standard communication object (i.e. not sockets but App Engine's urlfetch). So I had to piece together how to make a XML-RPC client call that used multi-calls so that I could do it asynchronously myself.

The first thing to know is that a multi-call is a system.multicall function call to the server. It takes a sequence of arguments that specify what each individual function call should be. Each sequence item contains a dict with two keys: "methodName" and "params". The "methodName" key has a string value of the function name to call on the XML-RPC server. The "params" key holds the tuple of arguments (and it must be the tuple type and you must use a tuple even if it is just a single argument --think of what it would take for the call to work with *args) to pass to the function named in "methodName".

Knowing what function you are calling on the other end (system.multicall) and having a sequence of dicts specifying what you are calling in hand, you can make your call to the server. First you need to encode everything into XML: xmlrpc.client.dumps((seq_of_args,), 'system.multicall') (notice how I passed the sequence of dicts in a tuple -- and I mean a tuple, not a list). With the XML now in hand, you make the actual call to the server by making a POST with the XML as the body and set the Content-Type header to text/xml using whatever asynchronous mechanism you want to use to make the call.

What you get back from the server is XML that you can load with xmlrpc.client.loads(data, use_datetime=True)(chances are you will want the [0][0] value as that contains a sequence of results from each function call you requested). Do realize, though, that there is no indication of what function returned what, so you will need to correlate the index of the return value with what call you made (zip() makes this trivial).

And that's how you construct an XML-RPC multi-call from scratch in Python so you can make an asynchronous call. If you want to see an example of all this, see this code.

Grab bag of tips when working with App Engine unit testing

2012-01-24T15:44:00.000-08:00

[update: App Engine doc bug is being fixed and should be publicly visible in 1.6.2 or 1.6.3]

First, an announcement: register for PyCon! It's a great conference and tons of fun. Early bird ends on the 25th.

With that out of the way, this blog post is going to be testing an App Engine app. This can be, liking testing any complex system, is painful. And unfortunately the documentation on this subject is somewhat lacking for App Engine. But hopefully this post about random things to be aware of when writing unit tests for App Engine can help prevent someone else from having to learn the hard way like I did.

First thing you should do when testing an App Engine app is set up a testbed. This will allow you to stub out various parts of App Engine so your tests run in isolation without having stale state carrying over. You basically create a testbed which holds state. You then activate the system by setting some core environment variables with defaults. After that you do your own tweaks to env vars. From there you activate various stubs. And then finally you deactivate the whole thing to clean up...

But you need to realize there is an apparent error in the docs (which will be fixed no later than 1.6.3, possibly 1.6.2). The stated order of calling activate() and setup_env() are wrong; you should call activate() before calling setup_env() in order to set the default env vars before you begin to set your own.

Another thing you should know about is that setup_env() has an overwrite argument. This tells the method to overwrite any pre-existing env var; otherwise the call will skip changing any pre-existing env vars. This can call issues if your app presets some env vars that should be overridden during testing. I would honestly consider using that argument every time to make sure things work the way you expect.

And the last tip specific to setup_env()is make sure to pass all of your env var changes in a single call. Otherwise you can end up with some settings be changed as they have a default value.

In relation to handling users, to specify who the logged in user is you should set the USER_EMAIL env var. That will cause App Engine to return a user with the specified email address when calling users.get_current_user().

Random thing about email testing is that mail.send_mail_to_admins() does not write to the mail stub to verify any email was sent. This is because while testing there is no way to specify who admins are, so there is no way to know who to say the emails are sent to. So in order to test the function you will need to mock out the call yourself (personally I'm a fan of mock if you don't already have a preferred mocking library).

And finally, you can mock our the request and response objects for a handler. You can call a handler's initialize() method with the object to use -- for the request and response, respectively -- which allows you to mock and test how the handler does things.

Bonus for those of you using the Python 2.7 runtime: I would consider using unittest's discovery features to drive your unit tests. You can start from my driver code and add what you need. But it's nice to just be able to plop down a new test module and have it automatically picked up.

Working with App Engine backends

2012-01-17T15:53:00.000-08:00

This post is all about backends in App Engine, but first I want to make two personal points.

One is that I'm sorry I have not been updating this blog (or really contributing to Python and its great community) more this past year. It was a rather insane 2011 for me personally: I finished my Ph.D., got married, moved to SF (w/o my wife thanks to immigration), started working at Google full-time, moved to Toronto (yes, I moved internationally within the span of 6 months), transferred to the Google Waterloo office and started on a new team. In other words I have been stressed out and busy and busy continuously for what feels like ages. But now that I am back with my wife and I am done with the crazy international moving I don't expect 2012 to be anywhere near as nuts (nor any other year in the near future for that matter if I can help it).

Two, I am no longer on the App Engine team, so all that I am about to say (and will also say in the future as I have a couple of posts to write on App Engine) is from me as just another person using the service. I have been working on a redesign of a website of mine in my spare time and these next couple of posts are based on that work (if you following me on Google+ you know what it is, but I'm not ready for a full-blown Internet debut so I'm not going to link here yet). I don't have any insider knowledge here nor speak for the team.

With all of that out of the way, let's talk about backends! First you need to understand what an instance is to App Engine. When you hit a website hosted on App Engine, an instance is what is serving that request. Now an App Engine app can have anywhere from 0 to thousands of instances up and running at any one time to handle traffic. So think of an instance as a copy of your app that is running on App Engine serving traffic.

Normally you have some time restrictions on how long any of your instances can take to service a request. If it is an HTTP request from the general Internet you have 60 seconds. If it is from a cron job or push queue (i.e. from within your own app) you have 10 minutes. Now both time restrictions are actually rather long and you can accomplish a lot within those restrictions. And in the case of push queues, you can break your work up into multiple tasks so that you really can take as much time as you need as long as you can slice your work into 10 minute time chunks.

But there are just those times when you want something that will take more than 10 minutes, such as a process that is constantly polling some other service to get data. In those instances you want a backend since they are instances that run for as long as you want (assuming App Engine doesn't kill it for misbehaving or some other sysadmin reason). You can have a resident backend which never stops running or a dynamic backend that will stop running when it is done executing, but either type can take about 24 hours to run if it wants.

The way you use instances is to send requests to the backend's subdomain. You name your backends in your backend.yaml file which becomes the subdomain off of your appspot.com subdomain for addressing your backend. So, assuming you have a backend name "forever" for an app named "spam", the domain for the instance will be forever.spam.appspot.com. You can then create URL handlers in your instance and hit those URLs to cause your instance to do it's long-running thing.

Now for those of you who have been using App Engine for a while you might notice that the subdomain for a backend looks just like the specific version ID URL for your app, and that's on purpose. Thanks to it having the same format you can specify a backend name anywhere you can specify a version ID for your app and have it use your backend. E.g. to have a cron job that hits your backend you specify your backend's name as the "version" of your app to hit in your cron.yaml.

And before anyone worries, your backend defaults to being private to your app. If you really want your backend reachable from outside of your app (i.e. the public) you can specify that in your backend.yaml.

While backends sound fantastic and address an old complaint against App Engine and its lack of background processes, I have two warnings/tips about using them.

First, the tip: you can't use the OS X AppEngineLauncher to use backends. When you launch a dev_appserver you need to specify the --backends flag. You can do that manually in the AppEngineLauncher, but you need the flag for appcfg.py as well which you can't specify. So if you want to use backends (and thus want to specify the --backends flag) you will need to use the command-line version of the tools. Obviously this is a minor thing, but it took me quite a while to realize that was why backends were not working for me.

And my warning is that don't expect to use backends on the free quota. While you get 9 free hours of backend usage, you can blow through that surprisingly quickly. But the real issue is all of the other quotas that you have on App Engine. For instance, I was hitting another server to get data to process. What ended up happening is that I was maxing our my incoming bandwidth quota for urlfetch. So then I started to write code to throttle my requests. But even then I occasionally hit the quota. To properly handle that I would need to grab the quota exception and then retry or heavily throttle myself so that I never hit the issue. At that point I was writing code to throttle my requests and to retry when I managed to request to quickly; that's just silly when I have push queues to do that for me automatically. So I personally ended up not using backends in my case (although ironically after I refactored the code I started hitting another free quota limit which led to me turning on billing, so now I'm using $0.05/day in quota, so the app is just costing me $9/month).

But when you are willing to pay for backends they are great for what they provide, but you just have to realize they are just another tool and are not always going to be the best solution for everything (especially if you don't want to pay to use App Engine).

PyCon 2012 CFP ends in a month!

2011-09-12T11:32:00.002-07:00

The title says it all: the CFP for PyCon 2012 ends in a month, so you should seriously start thinking about getting those proposals in! As I say every year, you do not need to be a Python celebrity or have presented previously in order to get a talk accepted. Having been on the PC from the beginning of PyCon I can tell you that we very happily give slots to people we have never heard of. As long as your proposal shows you are organized and your topic is interesting your talk will be given a serious chance for acceptance.

And sorry about the lack of posts. My Python free time is limited at the moment as most free time is spent talking with my wife over a G+ Hangout thanks to US immigration perpetually dragging their feet, keeping her in Toronto and me in SF and causing general misery spread across the continent. That means what little time I do have I spend on trying to get importlib bootstrapped (which you can follow in my sandbox/bcannon repo in the bootstrap_importlib branch) instead of blogging. But once the bootstrapping is done (or I come up with some rather clever thing) I will do a technical post.

How to import a module from just a file path

2011-07-16T15:11:00.001-07:00

Dr. Brown asked on Twitter whether there was a single expression (e.g., no semi-colons or abuses of and or or) that could import a module from a file path, either through a stdlib function call or just constructed from scratch; mod = import_from_path('some_file.txt'). Because it involved import, I got cc'ed on the tweet while various people tried to come up with a solution. In the end people realized that it's not possible in Python 2 (but it is in Python 3). But how hard could it be, right? Right?!?

Well, the devil is always in the details. While there are some functions in the stdlib that come close (e.g., execfile() and runpy.run_path()), none of them return the module, but the globals dict. Other people tried to come up with something using 'exec', but it doesn't return anything (in Python 2) so you have to assign what you planned to use as the module ahead of time (that's two lines total if you weren't counting along).

In Python 3, though, this is all totally doable thanks to 'exec' becoming a function instead of a keyword:

mod = exec(open('path', 'r').read(), __import__('sys').modules.setdefault('name', __import__('imp').new_module(name)).__dict__)

This is obviously a complete hack and not something you should use in the real world. But it is a single expression (meeting Titus' restriction) and it even puts the module into sys.modules so that future imports will not have to re-import the module. This hack, though, does assume that you are only importing modules and never packages.

So what would it take to do a proper function that took (at least) a file path and ended up importing that file? First thing is whether you wanted any arbitrary path to be used, or only something for properly named Python modules (e.g., ending in .py, packages being named __init__.py). If you go with the former you need to specify the module name and whether it is a package (and if it is what the value of __path__ should be), else you can just gleam all of that info from the path itself.

After that the next step is to create the module object and populate it. The bare bones thing to do is create the object and specify the __name__ and __file__ attributes, along with __path__ if it applies. This has to be done as a module might do something with those values as a side-effect of importation (and remember folks, never launch a thread as a side-effect of an import! Deadlock awaits those who don't heed that warning).

Third, you create the code object. Now you could technically just read the file and pass it to 'exec', but that won't set up the file path, etc. in the code object for easier traceback issues that may arise when you do the actual executing of the module code.

Fourth, you actually execute the module code and finish initializing your module.

Now all of this ignores storing the module in sys.modules or even checking if it even exists in there. It also doesn't add the module to sys.modules so that future imports can use the module instead of having to do the same thing again. But if you are honestly wanting to import a module by specifying a path, you probably don't care about any of this anyway. And because you ignore sys.modules, you get to ignore the import lock.

So, taking all of this together, here is a basically untested function that does what Titus basically wanted (works in Python 3, but changing it for Python 2 is literally just using the 'exec' statement instead of function):

import imp

# Step 1
def import_from_path(path, name='', package_path=None):
    """Import a module from a specified file path.

    If the module is a package, set package_path to a list of directories that
    is to become __path__.

    """
    # Step 2
    mod = imp.new_module(name)
    mod.__file__ = path
    if package_path is not None:
        mod.__path__ = package_path
    # Step 3
    with open(path, 'r') as file:
        code = compile(file.read(), path, 'exec', dont_inherit=True)
    # Step 4
    exec(code, mod.__dict__)
    return mod

My personal plans for Python 3.3

2011-06-28T21:34:00.000-07:00

Now that my life has hit another purgatory moment (waiting to here when my wife's permanent residency interview will be and still in the ramp-up period at work), I have had some time to think about what my personal goals will be for Python 3.3. This list is in no particular order and I make no promises to actually do any of it as I only have roughly 249 days until alpha1. =)

First, I want to bootstrap in importlib as the implementation of __import__. I think I can simply freeze importlib._bootstrap, do some imports of built-in modules, and then have it almost work. The reason it is "almost" is that importlib relies on _io, which itself relies on importing the os module. That's a problem when you are trying to essentially import what will act as import. I need to have a look still at the _io code to see if the importing of os can be postponed until after importlib is up and running.

Second, I need to finish PEP 399. Before the whole work/immigration whirlwind started I submitted this PEP to python-dev and got a bunch of feedback on it. The usual push-and-shove led to what seemed to be a general consensus of what the PEP should be striving for, so I think I can wrap it up and get it accepted once I have time to incorporate the feedback (mainly watering down the wording so it is less technical).

Third, I should try to finally get PEP 362 accepted (tracking bug). The code has been up on the Cheeseshop as pep362 for over 4 years now without any complaints. Since I know there are users out there (Michael Foord keeps asking me to finally get the code into the stdlib) I figure the API is generally acceptable and there are no secret bugs creeping in there. Basically I just need to rip out the silly use of inspect that is left since I dropped Python 2 support in the development version, make sure I have proper code coverage, and write docs.

Fourth, I need to get an Amazon or Rackspace instance up that generates a coverage report for the stdlib on some reasonable schedule. This first requires me to finally review and commit Brandon Rhodes' script which gets coverage reporting much earlier in the interpreter startup than it does by default. Once that is done I need to get a VM going somewhere that can freshly check out Python, coverage.py, run the test suite under coverage, and then make the coverage report available for viewing. This will make the test coverage not only more visible, but easier for people to help increase it as a starter project when contributing to Python.

So that is everything I am hoping to do before March 3, 2012. I plan on making the coverage and importlib bootstrapping my priorities, although I will finish PEP 399 first since I hate leaving open PEPs lying about. And I am definitely going to try to not take on any more Python projects until I get through this list as this is already plenty for me to do. =)

Framerates for movies and the debacle that they are

2011-04-22T15:15:00.000-07:00

One of my great weak points is technology, especially those involving movies. While I wouldn't go as far as calling myself a cinephile (e.g., I don't know every random fact about my favorite movies), I do love watching films which means I like trying to make my viewing experience at home rather nice (which honestly has not been that great thanks to my long-lasting student budget).

One of the latest things being debated about the quality of cinema is the frame rate of films. Currently movies project at 24 fps. ATSC (digital broadcasting version of NTSC) is typically 30 fps at 1080p although it can also do 24 fps and the funky rates film and old NTSC video show at (PAL is 25 fps but I'm North American, so I am going to stick with what I know in this blog post). Blu-ray can do 24 fps at 1080p. So everything these days can handle the native projection rate of 24 fps of 35 mm film.

But really, 24 fps?!? Any hardcore gamer is going to tell you that frame rate is unacceptable. You want at least 30, if not 60 fps. As it turns out, so do people watching movies. This is why most decent TVs these days at least output 120Hz video which helps to artificially raise the frame rate and lead to a smoother picture, no matter if the original material is 24 or 30 fps (or even 60, but that is typically reserved for video game output). And even newer TVs can do 240Hz, which leads to an even smoother experience (supposedly; have not experienced it myself).

But what about in the theater? Everything is still filmed and made for 24 fps and that is what projectors (digital and analog) use. Well, two of the more cutting edge directors currently working want to change this. James Cameron has talked about filming Avatar 2 & 3 at 48 or 60 fps. Peter Jackson has gone beyond saying and is actually doing by filming the Hobbit at 48 fps. According to Jackson the higher frame rate not only looks better, but it makes 3D something that is actually bearable to watch (i.e., the nasty flicker goes away). And it turns out most digital projectors can have their firmware upgraded to project films that use these higher frame rates. And with 48 fps being a multiple of 24, these directors could easily film at 48 fps and make 35 mm prints where every other frame is dropped so old projectors can still show at 24 fps.

But what about watching at home? While it's great that seeing those select films I want to see in the theaters will be projected at a higher frame rate (e.g., the Hobbit), what about when I want to watch at home? This is where that 48 fps gets sticky. Both blu-ray and ATSC do not support 48 fps at any resolution. Both do, however, support 60 fps at 720p. And I am willing to bet that Netflix streaming won't handle either 48 or 60 fps until we all have fiber connections. Another issue for the 48 fps is that it is not a common divisor of 120 (Hz), making it potentially nasty to artificially raise the frame rate; luckily 48 is a divisor for 240 (Hz).

So the question becomes, how does Hollywood decide to handle this theater/home frame rate discrepancy that is coming our way? Well, with this being Hollywood, I am willing to bet they do it in the way that screws over the home user the most, forcing you to go to the theater for the best experience. That would support the idea of going with 48 fps in theater projection. That way they still get a higher frame rate that helps out 3D projection, they can easily cut 24 fps versions for 35 mm prints, and they can stick with blu-ray and such at 24 fps as well. All of this while getting to keep a nice hook to get people who care about video quality to come to the theaters to pay for your expensive movie ticket. They can then come out with a new home video standard that supports 48 fps and force us to all buy our movie collections yet again (you know George would be happy to sell you Star Wars for the bazillionth time on a new format, although unless he lets Han shoot first I ain't going for it). If they go 60 fps they can at least do 720p60, but then again do we really want the drop in resolution?

importlib: doing it, and doing it, and doing it well

2011-03-23T18:43:00.000-07:00

Earlier today I pushed a change to CPython which allows importlib to pass Python's test suite as the implementation of __import__ (sans failures because nothing expects __loader__ to be universally set on modules). This means that there is no known compatibility issues standing between me and making importlib CPython's implementation of __import__.

So what was the final hurdle? It basically boils down to code objects and their immutability. When you import from bytecode you are essentially loading a code object which represents the module (basically using marshal.loads()). The issue is that code objects embed what file they were created from (the co_filename attribute). But what happens if you relocate a .pyc file? Should co_filename point to the place the file was originally created at or at its current location? Turns out Python thinks it should be the latter.

The issue with updating the co_filename attribute on a code object is that it can only be done by C code; attributes on a code object are immutable from Python. My original plan was to modify marshal.loads() to take a filename argument which represented where the marshal data came from. That way marshal could fix co_filename while keeping the attribute immutable. But this didn't work.

It turns out that the attribute used by __import__ for fixing co_filename is not as thorough as the recursive solution I came up with for marshal.loads(). There is a loop that looks for objects to fix, but as soon as one of them is accurate the search is stopped by __import__. Rather than replicate this somewhat odd solution for marshal, I simply exposed a private API in imp for me to use in order to mutate a code object's co_filename attribute. Not an elegant solution, but since I am taking a PyPy view to import (i.e., it's my job to make importlib as compatible as possible and not tweak __import__ to fit my needs) I didn't have much of a choice.

Regardless, it feels very satisfying to have an implementation of import that passes the test suite now. This means I can work towards bootstrapping importlib as the implementation of import. I am planning to go with some bytecode-freezing solution for CPython where there is a build rule which recreates the bytecode any time importlib._bootstrap is mutated. That will remove the worry of having to rely upon some external file for import to work (otherwise I could come up with some import shim that did enough to import importlib directly). And with performance of importlib already being acceptable I feel fairly confident that this bootstrapping will happen in Python 3.3 (I have started work in the bootstrap_importlib branch of my personal repo).

PyCon 2011 wrap-up

2011-03-20T17:56:00.000-07:00

tl;dr version: it was awesome! Read on to find out what I did at the sprints and two themes that I found at the conference.

Since I have not had a chance to blog until now, people like Nick Coghlan and Brian Curtin have beaten me to writing good summaries of the VM summit, language summit, conference, and sprints so I won't worry about doing an overall summary of everything.

At the sprints I almost exclusively focused on my Python 3 on PyPI website. After squashing a bunch of bugs I added two more popularity metrics. The first metric is download rate. This is to help balance out against total download count which favours older projects thanks to simply having been available longer. I also added the poll results as found on the front page of www.python.org (look in the upper-right corner of the page). The download rate has about 50% of those projects converted which is great, while the poll results is at 16% which is also very good considering people are explicitly asking for those projects to be converted because they were not at the time of voting. And because people seem to have some odd desire to see the crap code I wrote for this website, you can find it here.

As for the two themes I see gaining more exposure in the upcoming year, one was obviously Python 3 support from projects. There is now a book out by Lennart Regebro [Amazon] (which I wrote the foreword for), a good number of projects have already switched, Django will be switching this summer, and at the sprints Jim Fulton started porting zc.buildout to Python 3 (which happens to be the largest Python 3 holdout according to my website at the moment). In other words the momentum is there for Python 3 and I continue to not worry about it.

The other theme which gained more traction is separating Python the language from CPython the interpreter. PyPy's performance is now good enough to make it a serious consideration for anyone to use. And once they release a version supporting Python 2.7 (IronPython beat them to Python 2.7 compatibility) then they will be in a position to become the de-facto interpreter people use when they don't have specific requirements (e.g., embedding Python in a Java program and thus requiring Jython). What this means (at least to me) is that python-dev is going to need to start to shift towards a stance where the Python language takes precedence over CPython the interpreter. While CPython will continue to be the reference implementation (don't ask if PyPy will become it; I don't think they want that responsibility honestly), certain considerations are going to be needed in order to make sure that things we do in the Hg repository do not end up hurting the other VMs.

To help with this goal of making sure the Python language does not play favourites, I am planning on doing three things. One is a PEP I am drafting up dictating that C extension code in the stdlib must be kept fully compatible with any pure Python equivalent code. It will also say that any extension module with no matching pure Python version must get an exemption to be included in the stdlib. This is to prevent all the VMs from having to re-implement every extension module on their own, essentially leading to four versions of the same module.

The second thing is getting importlib to become the de-facto implementation of import in Python. Every single VM hates implementing import. It's subtle and complicated, making it very tough to implement in a way that is conformant. By moving it to pure Python, importlib will allow all of the VMs to simply use importlib and thus not have to re-implement import.

Lastly, a very long-term goal is to break out Python's stdlib into its own repository. This will make it even more obvious that CPython is the reference implementation of Python and that it is not special in any other way. It will also allow the other VMs to use the same code base to prevent duplicated work (the key contributors on the other VMs are already receiving commit privileges on CPython so they can start fixing bugs now). This is a very long-term project of mine and won't be started until my other two goals mentioned above are finished. In other words my goal is to start at PyCon 2012 with this.

Secret site goes live: Python 3 Support on PyPI

2011-03-04T14:23:00.000-08:00

When the Python 3 Wall of Shame launched a couple weeks ago, I (along with others) noticed discrepancies (discussed in the comments of the announcement blog post) between the real world and what the list said (e.g., doctutils was listed as not supporting Python 3). A similar issue came from On Python 3 Yet? Both websites seemed to be to accidentally spreading FUD about Python 3 support on PyPI thanks to the projects on PyPI not listing Python 3 support properly. While in no way intentional, it does cause issues when people use these sites as "proof" Python 3 still lacks support from the community.

So how did this happen? Well, projects on PyPI are listed as supporting Python 3 when they set the proper Python 3 trove classifier (instructions on how to fix this are in the Python 3 porting HOWTO). What I suspect has happened is that projects which do support Python 3 but don't have the proper classifier either (a) are not aware they should set it or (b) don't care/too lazy. Either way it leads to a mis-representation of what projects do and do not support Python 3.

To deal with this, I created a website to measure Python 3 support on PyPI. What differentiates this website from the others is that I am personally curating the list of projects that support Python 3. So while PyPI says docutils does not support Python 3, my website does. Same goes for projects which have forks that support Python 3 (e.g., setuptools "supports" Python 3 through distribute). I have even begun to flag projects which have working support in their version control system as "maybe" supporting Python 3. I completely understand why the creators of the other websites don't do this; it takes work. But since I have more invested in Python 3 than most people I am willing to put the work in.

If you go to the home page you will notice that I list a project as either supporting, not supporting, or maybe supporting Python 3. The latter case is there because a ton of projects do not specify at all if they even support Python 2! My hope is that projects begin to properly set their trove classifiers. This not only benefits the community by letting people know if they do or do not support Python 3, but because it can be used to know how far back they support Python. Take Django 1.2.5 as an example. It does not have trove classifiers set to let me know that it supports Python 2.4 through Python 2.7. Heck, I don't even know if they have tested against Python 2.7 yet. But if they set their trove classifiers this would be known both visibly and programmatically.

I am asking the community for help combating this problem of projects not listing their Python 3 support properly. If you know of a project which supports Python 3 somehow but that fact is not known on this new website, please either list it over on the Convore discussion on this topic, leave a comment here, or leave a comment on Google Buzz along with proof of the Python 3 support (e.g., a link to an official page stating the support). If you tack on a project's PyPI name on to http://py3ksupport.appspot.com/pypi/ (e.g., http://py3ksupport.appspot.com/pypi/docutils) you can see if the project's support has been picked up yet or not. As I said, forks do qualify for support as well as do projects which have functioning support in version control but have just not done a release yet. This is all rather important for the top projects as listed on the home page since that is what people will really pay attention to in order to notice Python 3 uptake.

To give people some scope of how much of a difference this curation makes, consider the top 50 projects by downloads for any release. 14 of the top 50 (i.e., 28%) support Python 3 somehow. But if you went by only trove classifiers you would think only 9 project (18%) supported Python 3. Even hard coding modules absorbed into Python's stdlib would leave off 2 of the projects. If you look at latest release downloads you have even bigger support and discrepancies: 16 out of 50 projects support Python 3. But of those, 8 I had to manually flag (4 of which are not from the stdlib). Percentage-wise this manual effort makes a difference and gives a much better indication of how much support there already is for Python 3.

So this is what I have been working on for the past two weeks. I learned a lot about App Engine and creating a scalable website. The whole thing updates every 20 minutes from PyPI in terms of project metdata changes. I also update download totals every day. Long term I hope to add more metrics to gauge what the top projects are (e.g., listed dependencies, Google Code searches, etc.). I also want to develop a Chrome extension which will let a user know when a project supports Python 3 when viewing PyPI even if it doesn't say so (maybe even getting to the point that people can notify me through a button click that the project actually does support Python 3). I also want an API so that people on other websites that use different metrics can have access to the same data I use to mark a project as supported or not. But as of right now, I'm ready to take the weekend off. =)

PSF core grant, the last days: the website works (so far)!

2011-03-03T23:18:00.000-08:00

The last couple of days I have been working hard to hack together (and I mean hack; it is not the most elegant code or solution I have ever written) the remaining bits of the website. I am currently waiting for the task queues to finish doing their thing so that I have a complete data set before I go "public" by asking the community for some help with something. But so far things are working out and hopefully I will wake up in the morning to no errors and the data set complete!

The biggest challenge I have run into over the last couple of days is App Engine's 10K limit on task queue data. that ain't much data. So I had to do some reworking of my data syncing workflow to use the datastore as a temporary storage solution. It worked out, but I'm sure I am doing something silly. At least I made sure everything happens in a transaction so that if anything fails the task queue will simply try again until it works (hopefully it's a transient failure and doesn't require me to fix some code =) .

Introducing mnfy

2011-03-01T18:13:00.000-08:00

Remember back in November when I did a blog post about an AST-based minifier for Python code? Remember how I said I gave up on it out of frustration over how the AST represented things in a way that didn't mirror the syntax? Yeah, that "giving up" part didn't sit well with me, so I created mnfy to help minify/obfuscate Python 3 code.

It's basically a little project that I created which helps in minifying code. I got over my issues with the AST by simply thinking it through and figuring out exactly which AST nodes needed special treatment (e.g., how to detect an 'elif' clause). Once I did that I was able to work on some of the transforms which is where the fun is.

The code currently doesn't do anything fancy that human beings who like to muck about with minifying/obfuscating Python don't already do:

Eliminates unneeded whitespace
Minimal parenthesis usage (mostly)
Uses hexadecimal numbers when it will save on characters
Combines imports into a single line that are defined in a row
Drop unused constants (including docstrings)
Functions into lambda definitions

As I said, nothing nuts, but at least takes some of the tedium out. The transformations of the syntax are also separated into safe and unsafe transforms. I consider safe transforms things that are semantically equivalent to what was changed in 99% of the cases (i.e., if you muck with execution frames you are on your own). Unsafe transforms will work in 90% of the cases, but are not exactly semantically equivalent (e.g., lambdas are not entirely equivalent to a defined function). Thus users can choose their exposure to transforms which may or may not break their code. You can also skip transforms all together and simply go with the elimination of whitespace and unneeded formatting characters (e.g., parentheses). In the latter case I am able to minify the entire standard library and still get back the exact same AST.

Anyway, this was just a fun little project that I don't plan to take that seriously (i.e., don't expect massive churn on the code any time soon =). If either minification/obfuscation or working with Python's AST interests you, there is a talk on using Python's ast module and another on obfuscating Python code at PyCon you can attend.

PSF core grant, day 39: darn scalability

2011-03-01T09:35:00.000-08:00

So the server component for the secret website is nearly complete, when I started to test the site through App Engine's dev appserver, some issues arose which were not triggered by unit tests. So I spent Sunday and yesterday (counting as one day thanks to working on bursts on-and-off) working on fixing those issues.

Otherwise the only other news is that the Python-Dev In a Box code is up. Please realize that it will not work for you! It is currently geared towards using the Hg test repo which is not ready for general use (e.g., it's missing some commits that landed in svn). I also am going on an assumption a patch I submitted to coverage.py about a __main__.py file for the repo directory gets accepted, and so I have a file I manually copy over in that instance until that happens. In other words, if you decide to play with the code realize that it probably won't work and so bug reports about it not running are probably premature. =)

PSF core grant, days 37 & 38: making Python-Dev In a Box a reality

2011-02-26T22:32:00.000-08:00

I got enough positive feedback on the Python-Dev In a Box idea from my last blog post that I decided to implement it. Over yesterday and today I have been writing some scripts to help bootstrap people into contributing. I will make the code and repository public as soon as I can get the remote repository at hg.python.org created.

So the goal of this project was to make it easy for people to jump in and start contributing at a sprint or on their way home from a conference. That meant providing the code, tools, and information needed to contribute. It also meant doing as much upfront for people as possible so as to not waste time for what everyone would end up doing along with not requiring a network connection. And all of this, at max, needs to fit in 700 MB on a CD.

First, there is a script called make_a_box.py which creates a Python-Dev In a Box instance. It will prompt you for what you want to include in the box:

CPython (from an Hg repository at the moment, but I might patch it to be from svn if Hg is not ready by PyCon)
Devguide
PEPs
coverage.py
Mercurial and TortoiseHg
Visual C++ Express (the Web installer)

When reasonable it will build what is provided. In the case of CPython and the various bits of documentation, the docs get built. For coverage.py it runs it so you have a basic coverage report for people to work off of.

For users, I included two extra scripts. One simply builds CPython (on UNIX). While building is very straightfoward (and documented in the devguide), I needed the functionality for generating the coverage report. Since I needed the code anyway I figured I would make it a stand-alone script. About the only real perk to it is that it uses multiprocessing.cpu_count() to run the Makefile with as many cores as you have (although CPython compiles in a fairly linear fashion so extra cores doesn't do that much).

I also provided a script that runs the test suite in the fastest, most rigorous way possible. Once again, while documented in the devguide, people can get a little lazy when it comes to running the entire test suite, so hopefully having a script that uses the "best" options will lead to more rigorous testing overall.

My plan is to create a Box for the PyCon sprints so that people attending the sprint can simply copy from a CD or flash drive everything they need to get going. This should speed up people getting started, leading to more productivity.

PSF core grant, day 36: maintaining a website is a pain, Hg, and Python-Dev In a Box

2011-02-24T19:20:00.000-08:00

I'm this close to having the data maintenance aspect of the "secret" website done, but came up short when I forgot that entities in App Engine cannot be stored in a transaction unless they are in the same entity group. Since I have tens of thousands of entities, putting them all in a single entity group seems stupid, so I will simply have to break out another worker to be called from a task queue in order to process each entity one by one (at least I can still batch the external network calls). Bah. So close!

While nothing to do directly with me, on the behalf of himself and Georg Brandl, Antoine Pitrou announced a test Mercurial repository for (C)Python. I'm obviously excited about this since I spearheaded this whole ~~debacle~~ movement to switch from svn to hg back in late 2008, making the decision of hg with Guido at PyCon 2009. It looks like there is a slight chance we might actually get this done in the near future (since Antoine and Georg are pushing this the last little distance I can't speak to whether they will get it done by PyCon 2011, but it would be neat if they did).

A rather nice side-effect of switching to Mercurial is the smaller amount of space that a Mercurial repo of Python takes up compared to Subversion. This led to a slight discussion #python-dev about creating a Python-Dev In a Box. The idea is that to help spur contributors one could put the Mercurial repo, the devguide (pre-built, but also along with the hg repo), Mercurial itself (including TortoiseHg), pre-build the docs (so that people have a copy of Sphinx and such which can also be used to build , and anything else I could think of (PEPs checkout? Visual Studio Express installer? coverage.py along with coverage results?) on a CD or flash drive so that people have everything they need to start hacking on Python. And by putting it on physical media it makes it easy to bootstrap everyone in the room quickly without slamming the network. It also provides everyone even the ability to get going while offline (e.g., "Can't stay for the sprint? Take a CD and start on the plane home (if you are not on Windows as that requires downloading Visual Studio Express)!"). Hell, think if this became a bag stuffer at PyCon! I wonder if that would increase contributions?

PSF core grant, days 33, 34, & 35: Python 3.2.0, fighting personal feature creep

2011-02-23T17:52:00.000-08:00

First off, for those of you who don't know, Python 3.2.0 was released on February 20th (which serendipitously happens to correspond to the 20 year anniversary of Python 0.9, the first release of Python). To celebrate Python being a score old, why don't you help port a project to Python 3.2.0? =)

With Python 3.2.0 out the door, the development branch of Python was open to commits again. I was able to commit all but one of my patch queue I have been sitting on for over a month. This means that the in-development branch of Python can now run the test suite under coverage.py without any special patches. I also fixed a bunch of static analysis warnings found by Clang. I am ignoring the time burned on trying to fix test_zlib failing under OS X as people much wiser than I at mmap solved that one. I also fixed the new crypt module changes to be PEP 8 compliant and simplified the API a little.

As for the secret project, I am working on not letting feature creep get the better of me. I scaled it back slightly so that it won't start out life with as much stuff since it didn't need it everything. But then I also thought of a cool way to get more people to participate with the website through a Chrome extension. Out with one feature, in with another.

But I am reaching the ability to launch in private beta in about a week. I don't see why I won't have this ready to go for public consumption (in a rather rough state) by PyCon.

PSF core grant, day 30, 31, and 32: lots of testing, moving on to data maintenance

2011-02-20T20:13:00.000-08:00

In a previous blog post I mentioned my refactoring/testing strategy for the website I am developing. Friday was all about implementing that strategy. Even without touching the network, my tests were good enough to lead to only two very shallow bugs that needed fixing. At that point I am basically caught up in my testing regimen.

That means new features! But yesterday and today have been a back-and-forth battle between trying to do a good job and worrying too much about rare edge cases and features that are not needed. Since the data only needs to portray details in a rough level and only for a subset of the available data, I realized I need to relax and not sweat every single little detail that could come up. It's just as easy to run an occasional sanity check to rectify any data drift that may come up.

And a tip to those using nose-GAE, if you get an odd error about not being able to delete a datastore file, it probably means you have a syntax error somewhere.

And a tip for those using Google Chrome: be aware that if you are manually triggering things through a GET request, Chrome's prefetching of pages can cause you to accidentally request twice. This was a problem for me as I was doubling the stuff I was putting on task queues and it took me a while to realize why.

Register for PyCon NOW if you actually plan to attend

2011-02-17T22:18:00.000-08:00

A trusted chair of PyCon US 2011, Van Lindberg, did a post explaining some of the behind-the-scenes issues that must be worked out when organizing the conference. It's an interesting post that's worth reading.

But one of the key take-aways from that post is that the hotel space is going quickly. We have already taken 105% of our allotted space. It's already projected that we will sell out PyCon this year. Make sure you register if you have not already else you may very well find yourself unable to attend no matter how badly you want to come.

PSF core grant, days 28 & 29: learning the hard way that you need tests

2011-02-17T20:04:00.000-08:00

My little skunkworks website got cleared to be worked on, so I have shifted my focus to it. I think I am going to keep it a secret until PyCon and announce it during a lightning talk for fun. But just because the subject of my work is a secret doesn't mean the process of creating the website needs to be kept from the public (even if it means I have to admit to what I am doing early; being secretive is not exactly critical).

I must admit that most of my experience in creating websites is relegated to either simple stuff or more JavaScript-heavy work. This is turning out to be the first website where there is a serious back-end that has offline workers, constantly updating data, etc. In other words it is a learning experience for me.

The first lesson is that I can't be sloppy with this website. Back on Sunday I started to slap together the website on Google App Engine on my machine, simply coding away until I thought I reached a point where I could actually examine some data. I was using task queues for the first time and trying to coordinate all of this communication with various queues running at various rates with different workers, etc. In other words I over-engineered.

But even worse is that I was not doing any proper testing. I figured I might just do some hand-driven tests on a subset of data to make sure things were working out. But my slap-dash coding quickly became a hindrance as I ran into several import errors thanks to me not paying attention and making sure to add the proper import statement. With the turn-around of launching a page which populated a task queue being long, I knew that I just couldn't keep going like this.

And so I knew I had to recode for testing. But how the heck do you test a website, let alone an App Engine website? I mean I know how to write proper unit tests, functional tests, etc., but I didn't know the best way to handle that for a website that has to run under some heavy infrastructure that has to be available.

This called for some research to even get tests executing. I stumbled across nose-gae which seemed to give me what I needed to get the tests executed if I was willing to use nose (which I was). I even tried to be a good little developer and work under virtualenv, but I ran up against odd path issues involving App Engine that I simply did not want to try to diagnose after an attempt to use gaeunit left me wanting (not enough traceback info to debug well). Luckily nose and nose-gae were the only things I felt the need to actually install so I didn't totally screw up my Python 2.5 install.

With tests runnable, I then had to decide how the heck I was going to do this. And that's when I decided I needed to refactor the hell out of my code. Being an App Engine app that is (at least currently) using no web framework, every URL resolves to a request handler class. That means there is at least a get() or post() method on each class. So I made the decision that all request handling (e.g., getting what is in some POST argument but not even decoding the JSON) and all response handling (e.g., writing back out any HTML) would be handled in the get/post method, but nothing more; essentially they are gutted to the point of simply stitching together method calls, as one typically should when there is I/O involved. I then broke code down into helper methods as necessary in order to think about the functionality in the way I needed to. I also put all transaction code in its own method since I am purposefully keeping that work to a computational minimum. This allows me to mock out things like task queue usage in order to verify what I want to happen is going on without worrying about App Engine actually using some task queue and having to clean that up manually later (handling actual datastore work is no big deal so I am happy to let that actually occur with proper test cleanup).

All of this leads to me writing abstracted, modular code where I can test non-network, non-App Engine code quickly and easily and then purposefully cleaning up or mock out as needed when I do need to work with App Engine's assets. At this point I have not written tests for the actual website page requests as the get()/post() methods as they are dirt-simple and they are just being called by App Engine task queue events (but eventually I will once I decide how I want to handle that specific case; webtest?).

What is the lesson in all of this? Don't be sloppy. Assume you will have to test your code eventually, forcing you to write code in a fashion that makes the code easy to test. And then actually bother to test your code. Dealing with the network or APIs that have no way to undo or cancel an event programmatically (e.g., something in a task queue) should not stop you from writing tests as there are ways to isolate these things such that they are not a burden.

PSF core grant, day 27: little details, a secret, and a progress report

2011-02-14T21:56:00.000-08:00

Started out the day creating a patch to link from the issue tracker to the devguide. There have been a couple people who have asked me to have the issue tracker fields link to the devguide to give a better explanation. Plus updating the docs is easier than the tracker.

But what really took up most of my time today (and yesterday, but I had errands throughout today so I am counting this as a day's work) is a little skunkworks project I started working on related to Python 3 and combating some (accidental) FUD that seems to keep coming up that really needs to get squashed.

Anyway, I also figured I would summarize what the heck I have done as part of my grant as I have finished day 27 of 41 (and I need to summarize for the PSF board this week). I completed the devguide, which I have wanted to do for literally years. There is also a branch of it already prepared for when the switch to Hg occurs. This also led to various patches that make sure the stdlib can have its test coverage measured, fixed a bunch of static analysis warnings from Clang, and have import failures not implicitly represent a skipped test.

I wrote a HOWTO on how to port a Python 2 project to Python 3. Now the community finally has a single document to read to at least get started in porting instead of having to pull up various blog posts on the subject.

All of this and I am three days behind "schedule" of ending my grant come March 1 (unless my math is wrong and this really isn't the 30th weekday since January 4th, inclusive). Overall I'm happy with what I have done and I am hoping the remaining 14 days are just as productive.