Coder Who Says Py

A decade of commits

2013-04-18T07:59:00.001-07:00

Today marks a decade since I made my first commit to CPython's repository on Sat, 19 Apr 2003 04:00:56 +0000 (python-checkins, hg.python.org). According to Ohloh, I currently sit as the 16th most prolific committers based on commit count which I can hardly believe. Boy have times changed over the past decade!

Back in April 2003, we were still on CVS on SourceForge (I somewhat foolishly took on projects to change both of those). Guido gave me my commit privileges himself (now I hand them out which is a bit scary =). It was less than a month after the first PyCon (or at least the first Python conference officially called PyCon, and I've managed to attend every single since and now have my wife asking if she can come) and me being elected to the Python Software Foundation (which I joined the board of directors for a time). This was before Python 2.3 we released (and now we are working on Python 3.4). Back then, Python was becoming popular and had an upward trend, heading towards its current position as the top dynamic language out there that isn't embedded in a browser (I would say it didn't really become really obvious this was going to happen until about 2005, so I got my wagon hitched at just the right time =).

But this post is not about reminiscing. It's for thanking the people and community who have made contributing to Python so enjoyable that I have actually wanted to do it for a whole decade (and will continue to do so for the foreseeable future).

I want to first thank python-dev. I have always said I truly learned how to program from my fellow core developers. Getting to work on CPython's interpreter core and the stdlib showed me how to manage complexity in APIs, keep my code clean and readable, when to optimize and when to go with the easier to read solution, etc. Pretty much everything that you would want to know when programming in the wild I didn't learn from a class or a book but from my fellow open source programmers. You just can't buy that experience. This is the reason I have always done what I could to make the lives of people who wanted to contribute as easy as possible (sometimes at the expense of other core devs depending on how you fall down on the svn -> hg transition).

I also want to thank the Python community. When I first started contributing I was doing it to gain experience in programming in the real world by contributing to a top-notch codebase with world-class programmers. But as time went on the things I gained in terms of experience dwindled. But what I lost in terms of fulfillment from what I learned was more than made up for in terms of the interactions I had with the community. Meeting people who have benefited from my code and said "thanks" for volunteering my time truly does inspire me to keep contributing, especially when I don't want to backport a bug fix. =)

But through the community I have also been able to gain great friends from across the globe. While I may only get to see my "open source friends" about once a year for a week at PyCon (which is the key reason I look forward to the conference as soon as the last one finished), they are truly friends. They are people I would let crash in my spare bedroom, give them a key, and say "welcome" without hesitation (if any of them were ever so inclined to visit Toronto, let alone Guelph). Those friendships are truly important to me and what will keep me coming back for years on in the future no matter how much or little I am able to contribute in my spare time.

So thanks to everyone reading this. By being a part of this great community of nice, caring individuals I continue to come back to contribute and participate however I can.

Why I'm signing up for Gittip

2013-04-12T14:29:00.002-07:00

While at PyCon I heard about plans to integrate Gittip into rubygems.org and so I decided to have another look at Gittip. For those that don't know, it's a website where you can give and/or receive money to/from others on a weekly basis. You can give as little as $0.25/week ($13/year) up to $24/week ($1,248/year). Being on a weekly schedule allows you to say "I appreciate the time and effort you put into open source; keep it up!", compared to bounties which are goal-specific and don't recognize people who make contributions that have no direct financial benefit or make contributions year-round.

As I was poking around the site I noticed that my friend Jesse Noller was the top recipient. I read his page on Gittip which listed his vast accomplishments that he has made in his spare time for no pay beyond gratitude from others and any feeling of accomplishment his hard work gives him. But the other thing his Gittip page mentions is what receiving tips means for him.

It basically boils down to a way for people to thank his family for letting him do his open source work. That sentiment really struck a cord with me. Like most open source contributors, I do it because I derive some enjoyment from it. It's a feeling of accomplishment, it's the camaraderie with my various friends that I have in the Python community, etc. In other words it's all very intangible but I do get something from doing my open source work.

But my family doesn't get any of the benefit that I get. Since I am not paid to do my open source work I need to take personal time to do it. That means I have to take time away from my wife to do this rather solitary work of contributing to open source. While my wife understands why I do what I do for Python, her benefit of getting to be proud of me is indirect and very diluted compared to what I get from it (although she is starting to increase her participation by attending PyCon).

But having people express gratitude through Gittip gives more direct benefit to one's family. When I asked on Twitter and Google+ for people to tip Jesse to thank him for all that he does through the year (and especially for PyCon in the past two years), he got a nice bump in his tips, and so he was able to take his daughter and family out bowling that night.

Tips then are a way for the community to thank someone's family for letting them share their loved one with open source. For instance, tips for me would be a way of thanking my wife for letting me spend the hours I do contributing to Python in my various ways by letting me treat my wife to a night out so neither of us has to cook. It also doesn't hurt that it acts like a small form of blackmail; "yes, Andrea, I do need to get this patch in and you should let me put the time in since the Python community treated you to a nice dinner last night" =) .

All of these reasons are why I'm joining Gittip. It has actually now reached the point of legitimacy to have Heroku as a company start leaving tips and Read the Docs is trying to pay for various expenses through Gittip; it's no longer just a bunch of individuals. So please consider signing up to both receive and send tips if you have the financial means to thank those in open source for their diligent and hard work.

PyCon 2013 report

2013-03-21T19:24:00.000-07:00

PyCon 2013 is now over and it was awesome (as usual)! As seems to happen every year, there were a few themes at the conference.

Packaging

For those of you who don't know, people are giving it another go to try and straighten out packaging in the Python world. The difference compared to the previous attempt is that Nick Coghlan, who is leading this endeavour, is working directly with pre-existing tools to gain consensus on things instead of trying to get the stdlib to handle it all. This means, for instance, he is working with the installer projects (e.g. pip) to agree on what should (and should not) happen in the evolution of packaging. This seems to have done a good job in energizing key people into supporting Nick's overall view (more on that later).

This means the stdlib is not going to try and solve all problems. The current thinking seems to be that the stdlib should house modules for which PEPs exist and then tools are to be built on top of that. This allows for all tools to act on metadata and such in a uniform way, letting them innovate on higher-level details (and keep the stdlib out of the installer game). Think PEPs 425, 426, and 427 details being handled by distlib.

He is also working from the top down on the stack. This means installer now, build-related stuff later. This has the nice benefit that the thing that most people directly interact with the most should get fixed first, rather than worrying about behind-the-scenes details later.

What does all of this mean? Eventually people will be able to get an installer, be able to securely install from the Cheeseshop (or any other package index of their choosing), and have it all bootstrap up on their system easily. A proposal (PEP 439) even went out this week to basically include a pip bootstrap script in Python which will install the real pip if it has not already happened and then continue on with the installation, making it all seamless. You can follow the discussion of this specific proposal on distutils-sig.

If all of this interests you I suggest you watch the packaging panel when the video goes up.

Python 3.3

I gave my Python 3.3 > Python 2.7 talk again (video here; PyCon Argentina video here although I think I like the US one more) where basically I pointed out all the wonderful features of Python 3.3 and that performance-wise you don't have to care which version you use (unless you have memory issues in which case you will want to use Python 3.3). I honestly was expecting some pushback since I have become a little jaded over the past 4+ years of Python 3's existence. But you know what? No pushback at all (but maybe it's because Armin wasn't there this year =). It was a really nice change of pace to not have to defend something I believe in and have worked hard to foster.

I heard numerous people tell me that they had finally been able to start using Python 3 and that they really enjoyed it. Jacob Kaplan-Moss of Django fame gave a talk on porting Django apps to Python 3 (no video yet) and told me that he not only liked Python 3, but that the no-argument version of super() made him "irrationally excited". David Beazley said that since he wrote the 3rd edition of the Python Cookbook for Python 3.3 he finds Python 2.7 a bit painful to use. It continues to be the case that almost everyone who gives Python 3 a fair shake ends up really liking it.

Diversity & Outreach

Watch Jesse Noller's opening statements. Then watch Eben Upton's keynote. Then realize that 20% of attendees were women. Then realize there was also a Raspberry Pi programming class for kids. Then really make sure you watch Jesse's opening statements if you ignored that initial link. Makes me want to be a better person and try to help people even more.

Everything else

I gave my "How Import Works" talk (US video here, Argentina video here and this time I prefer the latter thanks to having more time and thus feeling more relaxed).

The language summit happened. You can find numerous other summaries of what happend out there (Nick Coghlan, Kushal Das), so I won't rehash it here.

I wasn't able to stick around for the sprints this year (first time since the founding of the conference) past half of the first day. But hopefully next year I will be able to make it work out.

As I said, overall it was a great conference. Thanks to Jesse and everyone else who volunteered to help make it a great week.

Resolving a TOOWTDI interface problem for attributes

2013-02-17T10:45:00.000-08:00

TL;DR: choose one way to signify the lack of information in an API either as making the attribute optional or setting a default value (e.g. None), but not both.

When you read the docs for importlib.find_loader() you find that an exception is raised when __loader__ is set to None. But if you read the docs for importlib.abc.Loader.load_module() you will notice that __loader__ "should be set" (italics mine). So one part of the docs says having a value of None is fine while another says the attribute doesn't even have to exist. So the former is a LBYL version of the API while the latter is a EAFP version. While that's technically fine, I do like the concept of TOOWTDI in Python, so I would prefer choosing one of the approaches as the definitive way to signal that a module's loader is not known.

Does long-term (think in timescales of years) backwards-compatibility suggest a preference of one over the other? As it stands now, one must do:


if getattr(module, '__loader__', None) is not None:

# Use loader

That handles both the LBYL and EAFP approaches of either not setting the attribute or setting it to None. If this were to translate to LBYL it would become:


if module.__loader__ is not None:

# Use loader

Not a huge difference, just easier to read. The EAFP approach would be:

try:
loader = module.__loader__
except AttributeError:
pass
else:
# Use loader

Longer, but still totally readable and psychologically makes more sense since the attribute is set more often than not (importlib actually sets the attribute along with __package__ after importing if they are not already set).

But since most code that cares whether __loader__ is set already uses the getattr() approach, the None value approach is the least disruptive to changing to the eventual idiom.

But the thing that tipped the scales for me is I don't want the attribute to be optional but be required in the long run (think Python 4 long run; side-effect of how long Python versions last), so I plan to change the default attributes on the module type to always have __loader__ and __package__ and set them to None by default in Python 3.4. That means the optional approach won't mean anything going forward, so that makes the LBYL approach the one I plan to go with even if I personally prefer the EAFP approach for optional API attributes; I don't want this part of the API being viewed as optional by loader authors.

If you care about any of this specific API cleanup, you can follow issue #17115 as I clean up importlib's mixed approach to __loader__.

Remember that the "BC" in ABC means "Base Class"

2013-02-01T13:00:00.000-08:00

[UPDATE: had a talk with +Thomas Wouters on IM and has caused me to rethink things. New thoughts up top, original post after the jump break]

I had mis-heard a comment Thomas Wouters made in a meeting about not raising NotImplementedError and using ABCs, which led to me thinking about the problem of having ABCs in your MRO which were not at the bottom and defined methods which would override methods you wanted to access farther down the inheritance chain. I had thought that calling super() in your ABCs in some manner was the solution. But after discussing things with Thomas I believe I was in the wrong and I had badly misheard what he had said. =)

Because importlib has a bunch of overlapping ABCs which inherit from each other I thought that the situation might come up where you inherited from two different classes which had overlap in methods but for which you would want them to build off of each other. But as Thomas pointed out to me, ABCs are meant to be at the bottom of an MRO; the BC stands for "Base Class" for a reason. If you are trying to interleave methods between two different classes implementing the same interface then either the granularity of the ABC is wrong or you shouldn't be inheriting from the ABC to begin with. I tried to come up with counter-examples but they were so convoluted and leading to bad API design in order to justify that I gave up and admitted they were stupid.

What does all of this mean for you when you are writing an ABC? You should just treat your ABCs as the bottom of your MRO. That means you should have all of your methods, even the abstract ones, do something sensible in case they are somewhat blindly reached through a super() call. If you have a default return value, return that. If that does not exist then you should raise the exception which signifies failure as defined by the API. But raising NotImplementedError is not the right thing to do when it can be avoided with a sensible default reaction (which I have not been doing in importlib.abc). This also has a nice side benefit of making sure you clearly define what the default reaction is for a call to the method.

[ORIGINAL POST]

I was in a meeting yesterday with Google's Python team and the point of using super() in a base class and not raising NotImplementedError came up. Thomas Wouters said you should always use it and that he would discuss it with the other engineer offline as the rest of us "already knew what to do". And that's when I realized I had forgotten do use super() in importlib.abc and was doing exactly what I shouldn't be. Lesson learned: code you work on over the span of five years may not follow best practices. =)

If you don't have much experience with super(), read the post by +Raymond Hettinger which explains best practices. Unfortunately it doesn't cover ABCs which have their own set of best practices in terms of super(). In a nutshell, because multiple inheritance doesn't guarantee that you will be the bottom of the MRO even when it comes to ABCs, you should expect to have every method which doesn't actually implement anything fall through using super() when appropriate. E.g. if a method's return value doesn't have a sensible default (e.g. it's not all about the side-effects), then you should blindly call super(), else call super() only when it won't fail.

To illustrate, let's work with importlib.abc.MetaPathFinder (and I should mention all code samples in this post are assumed for Python 3):

class MetaPathFinder(Finder):
@abc.abstractmethod
def find_module(self, fullname, path):
"""Abstract method which, when implemented, should find a module.
The fullname is a str and the path is a str or None.
Returns a Loader object.
"""
raise NotImplementedError
def invalidate_caches(self):
"""An optional method for clearing the finder's cache, if any.
This method is used by importlib.invalidate_caches().
"""
return NotImplemented

What is wrong with this code? To start, what happens if I call MetaPathFinder.find_module() as part of a chain of super() calls? I get an exception, which is not really acceptable if this ABC is not at the bottom of the MRO for a class since a proper return value is expected for the method (one could argue that returning None is an acceptable default result in this specific instance, but a finder which never finds anything makes no sense in general so I'm not even entertaining that idea). What should really happen is return super().find_module(fullname, path). That way if someone calls this method in hopes of reaching a find_module() method farther down the MRO then they will be able to. And if no such method is defined? They still get an exception (albeit a different one) and so the (rough) semantics stay the same. Granted, most times this MRO will be the second-to-last class in an MRO (object is always last), but this is still an improvement worth making just in case.

What about MetaPathFinder.invalidate_caches()? That has no discernible return value (the method is all about its side-effects when implemented), so we don't have to worry about a call being a no-op as that is an acceptable result. That means invalidate_caches() should turn into:

super() = super_
if hasattr(super_, 'invalidate_caches'):
super().invalidate_caches()
# Or as a one-liner
return getattr(super(), 'invalidate_caches', lambda: None)()

This then allows the no-op to occur if there is no one else to fall-through to, else the fall-through is allowed without error (I'm also relying on Python's semantics of returning None by default plus the semantics of the method not having an important return value).

One method that can be a little sticky if not handled properly is the __init__() method. If you read Raymond's blog post you will notice he recommends that you strip off arguments as they are consumed so that when you hit near the bottom of your MRO there are no arguments, letting it hit object.__init__() which takes no arguments. But if you don't do that and don't pay attention to where you are in the MRO you can easily pass something to object.__init__() and get a TypeError. As a challenge I decided to see if I could solve the problem somehow with code instead of convention and I came up with:

def super___init__(super_, *args, **kwargs):
mro = super_.__self_class__.mro()
# Check this is not the relevant end of the MRO.
if mro[-1] != super_.__thisclass__:
super_.__init__(*args, **kwargs)
else:
assert mro[-1] is object # Always at the bottom.

This allows you to make a call like super___init__(super(), *args, **kwargs) from within your own __init__() and not have to worry about whether you have properly stripped everything out of args and kwargs. Now I don't necessarily recommend doing this instead of continually stripping arguments, but this could help in situations where you would have an adapter class instead to catch the bottom of the MRO from calling into object.__init__() (I suspect Raymond didn't suggest this over adapter classes as it won't work with classic classes in Python 2).

How much of Python can be written in Python?

2012-12-09T10:44:00.001-08:00

Now I don't mean in the PyPy sense where you can bootstrap yourself with another Python installation. No, I'm talking about all you have is a checkout of the CPython repository and a C compiler. How far could you go in writing stuff for Python in Python and not C (from my perspective, for maintainability, for others perhaps ease of extensibility). In Python 3.3 we now have import written in Python (technically the main import loop that is used is implemented in C to save 5% at startup, but that is entirely optional as equivalent pure Python code still exists) and it's actually faster than the C version from Python 3.2 thanks to directory content caching. So it is not entirely ridiculous to think about how far one could push the idea of replacing C code in CPython with Python code.

What restrictions do we have for this thought experiment? One is that CPython needs to continue to be performant. That means either that the feature is not executed constantly or can be made to work as close to C code as possible. The other requirement is that it can't really have dependencies on the stdlib beyond built-in modules. Since this concept works based on freezing Python bytecode into C-level char arrays you don't want to have to pull in half the stdlib just to make something work. But that's pretty much it.

The first possibility is the parser. If you either generated the parser like the one CPython uses (that has not really changed much since Guido wrote it way back when) or wrote a recursive descent one by hand, it could probably be written in Python. The real problem is how performance might be hit. Now if you are working off of bytecode files then this really is only a one-time cost per bytecode file creation. But if you are working primarily with modules that you specify on the command line then they get parsed every time you invoke the interpreter and that could be costly if you can't get performance to be good enough.

Going down the compiler chain, you could also go from CST (concrete syntax tree) to AST (abstract syntax tree) in pure Python. You can already get to the CST from the parser module, so the work to expose the CST at the Python level is done. And with the ast module already exposed it then becomes a matter of creating the AST nodes from the CST. But once again, it's a question of performance since this is invoked every time source code is compiled.

Next would be transforming the AST to bytecode. The AST is already exposed to Python code, so once again the initial work is done for access. But also once again there is the question of performance as this is also on the critical path if you continually compiling Python source code because you are executing scripts instead of importing code which was previously stored as a bytecode file.

You can't do anything for the interpreter eval loop as that becomes a bootstrap issue. If you really wanted to push this you could do a basic eval loop to bootstrap a more complex one, but that seems like more work than it's worth.

I suspect most of Python's builtins could be re-implemented in pure Python without any trouble. Re-implementing something like any(), map(), etc. is not exactly difficult. In this instance, though, performance definitely becomes a key issue due to the extensive use of builtin functions. And in the case of exceptions you have to worry about the C API surrounding them on top of any possible performance issue from exception raising (although I'm willing to bet this can easily be alleviated by just caching at the interpreter level the builtin exception classes so that at the C level it's still just PyObject pointers instead of having to extract them dynamically every time from the builtin module).

And as always every single module in the stdlib does not have to be implemented in C code if it doesn't wrap other C code. In that instance it is simply taking the time to either copy over and get working the pure Python versions of modules that other VMs have written or writing one from scratch. But thanks to PEP 399 this is only an issue for pre-existing modules (which is also why no one has bothered to backfill all of those modules as the other VMs have already done the work for themselves so no one really needs this to happen; I opened issue 16651 to find out exactly what modules don't have a pure Python version).

In other words, there are various possibilities for technically writing more of CPython in pure Python exists, but performance considerations will quite possibly not make it worth pursuing (but I would be quite happy if proved wrong =).

My (very shallow) thoughts on Dart

2012-05-13T21:11:00.002-07:00

Being the language nerd that I am, I actually find it fun to learn new programming languages. Now typically this is nothing more than me reading all of the official documentation and writing some toy examples that give me a very shallow, quick-and-dirty feel for a language. Since I have been involved in language design for nearly a decade (started participating on python-dev in June 2002) and have done toy examples now in 18 languages (17 actually still run; I have never bothered to get Forth to work again after a gforth change broke my code), this is actually usually enough for me to grasp the inspirations for a language and thus understand its essence.

At work I have been doing some JavaScript work for an internal Chrome extension and dashboard and so that led me to want to look into what Dart had to offer over JavaScript. I know the language is only at version 0.09 (and still changing weekly), but the fundamentals are there so I wanted to see what the general feel of the language is (and will continue to be).

I also know Dart is somewhat controversial for some people. Personally, I fall on the "competition is good" side of the argument, not the "OMG fragmentation" side. I want ECMAScript Harmony to still happen and give me a cleaner, tighter, more functional JavaScript, but that doesn't mean Dart doesn't have a place in the world as a cleaner OO language for the web. Besides, me thinking otherwise would make me a massive hypocrite as I began working on Python before it was cool (I feel like I need a hipster meme for that statement, but I digress) and I have worked hard to convert people to Python from other languages. Hell, I have tried to foster competition between the Python VMs to get them to push each other to perform better and be ever more interoperable. IOW I don't totally buy this fragmentation argument.

Going into learning Dart I knew who was involved with the language which is what will inherently define how a language feels. I knew Lars Bak of V8 helped design the language, which meant it would have some design restrictions put on it to make it have a damn fast VM. Josh Bloch has been helping to design Dart's library which meant some JDK feel to it. I also know Jim Hugunin is involved which should also help with the VM speed. So fast with an API designed like the JDK.

What did I find? A language with a damn fast VM and a standard library that felt like the JDK. =) Take OO as a Python programmer would expect (e.g. pure OO where everything is an object, not dogmatic OO like Java where everything has to be in an class definition), make types entirely optional for testing and tooling purposes but enough support to use interfaces and generics, and then toss in abilities based on what JavaScript allows and then you have a good idea of what Dart offers.

So, Dart has optional typing. In case you have not heard, Dart does not use type information at runtime for performance and only throws any form of fit if a type doesn't match what is specified unless you run in checked mode. If you do that then you get warnings about possible type issues. But Dart's type system is unsound so don't expect typing to catch every error that a more strict type system might even when you run in checked mode. Dart views types as helpful documentation and a way to help tools assist with things, period. I actually find it rather refreshing to have a language that treats types as just documentation since that is really what they are for the programmer (VMs can use it for performance, but it isn't required for good performance and type safety only saves you from a minor set of bugs which every Python programmer probably realizes eventually =).

But that's even if you bother with types! You can write all of your code without types and everything will run without issue. Even generics are optional, so you can declare a function accepts a List or List; Dart doesn't care either way and it alleviates covariance/contravariance headaches by not caring if you don't care either. It's actually rather nice to have non-library code be written quickly using dynamic typing and only add in the type information for library code where you care about what interface is expected. IOW I think Dart strike a nice balance with how it does typing and I actually feel fine using types when I know what I expect to accept in my own code that I don't expect anyone else to rely upon.

Dart is OO, not prototypical like JavaScript. It's single-inheritance, which I'm fine with. It does have interfaces as one would expect in a statically typed language, but it softens their expense by allowing one to define a default implementation of an interface. What this means is that the Map interface will also give you a HashMap instance if you call new Map(). I suspect they snagged the idea from Scala where you have the Map class which hides HashMap from the user if you simply don't care about what Map implementation you use.

It does have a modicum of privacy by using a leading underscore for signaling something is private, much like Python. But the privacy is enforced at the library-level or is public, period. Every field automatically has a getter and setter defined for them, so there is no way to force a private field (which I think is a good thing since I find private privacy bloody annoying). I also like that getters and setters are directly supported by the language with automatic generation show you don't ever have to see a setSomething()/getSomething() function call just to read/write a field, but you can do something like Python's properties very easily.

The standard libraries are fine and just feel like the JDK. Things are very much LBYL rather than EAFP. I am willing to bet (although I have not tested this) that exceptions are a little expensive in Dart (since exceptions are hard to optimize) and so they would rather go the LBYL way. But they still went a little overboard in my opinion on some things (e.g. the list interface has a last() method instead of supporting negative indexes). But there is nothing there that is making me run away screaming.

One place I do think Dart could use some improvement is simplifying their constructor rules. Upfront Dart has some nice syntactic sugar for a construction where you directly specify how a constructor's arguments map to instance fields, avoiding having to declare the constructor parameters and then also write an assignment. OK, I like that.

Dart also has initializer lists which let you initialize final fields. OK, that's cool and a nice idea taken from C++.

Constructors are not inherited. OK, that's fine since you probably want to be explicit about how you tweak stuff. But there is an exception about the default, no-argument constructor calling the superclass' no-argument constructor. So while not technically inherited, it might as well be in that single instance. And all defined constructors will automatically call the default constructor, which if it isn't defined you must explicitly call a constructor somehow (probably in the initializer list of your constructor). Um, OK...

And you have named constructors. This gets you around from the lack of type-based method overloading for constructors. OK, I can go with that.

You also have constant constructors since fields can only be initialized to compile-constant values. Fine, that's for performance and determinism in instance creation, so I can grasp the desire for that.

And then you have factory constructors. OK, this is where I go "WTF people". This is so that you can have a constructor that actually doesn't create a new instance but instead can return something else other than a new instance (think of Python's __new__() or any of Java's static factory methods). But this lets you use the new keyword on a factory constructor instead of using a static method. And that to me seems unneeded.

So lets recap what constructor options we have. We have regular constructors, default and defined, which supports initialize lists. You have named constructors. There are constant constructors. And you also have factory constructors. If you don't count the default constructor as special that means Dart has four types of constructors. WTF!?! I realize that Java's FactoryFactoryOfFactories crap has probably spooked the crap out of the Dart designers, all the while having Java influences making them think they need the new keyword for anything that would return an instance of a class, but this seems a bit much. Dart's function definitions are rich enough to allow for optional arguments, etc. which would suggest that the typical constructor can do the job of named constructors with static methods picking up the slack where absolutely necessary where factory constructors are used. Maybe I'm missing something here, but I think they tried to design for everything that is bad about Java's constructor mess without stopping to think what their function definitions already buy them, all while making sure the new keyword was used.

Luckily that is the only bit of Dart that I found poorly designed. Everything else is reasonable and something any JavaScript programmer will be somewhat familiar with or quickly grasp.

Now as I said, I only did toy examples in Dart beyond reading the docs from beginning to end. If I had more time this weekend I may have done one more coding example that was more involved, but I ran out of time. But based on what I have read and what I learned, I am happy with Dart and would be content in using it for programming for the Internet. I would also be totally happy being asked to use it in a situation where others wanted to use types (e.g. I would be fine ditching Java for Dart if people really felt the need to hold on to their types).

Thoughts on using function signatures as a DSL for CLI parsers

2012-05-12T13:02:00.002-07:00

I have no idea why, but this morning I thought about a decorator for delineating what function should be treated as the main function (e.g. using a decorator instead of the traditional if __name__ == '__main__' idiom). Now I solved it in my head on the spot, and then immediately realized someone had to have solved this already. Turns out various people have done things as nuts as examine stack levels to detect the __main__ name, but the most straight-forward solution I found doesn't do anything nearly as nuts or CPython-specific and is basically what I came up with. There was a red herring, though, in everyone's solution where they claim the decorator has to be on the last function in your module. While technically true when using the decorator as a decorator only, you can also just as easily not decorate the function and instead, at the end of your module, do something like main(func) since that is the same as decorating func with main.

A really simple expansion of this idea of helping out with defining what function is the main function, is to pass in sys.argv and to return a value to signify exit status: sys.exit(func(sys.argv[1:])). So now you have made the decorator more useful than replacing the old __name__ idiom.

But while that is nice and helps deal with the very common case, I wanted more. Why can't you introspect on the arguments the function takes and use that to automatically generate a command-line parser? I did a search and the best I could find is entrypoint, but it doesn't go far enough for me. What I want is to use the full expressiveness of function parameters in Python to express as much about what should/could be given on the command-line along with passing in as little as possible to the decorator in order to replicate the common case of command-line parsing; think just as easy as getopt but more powerful by using as much of argparse as you can without coming up with complicated rules about how things should work (since once you pass a certain complexity threshold you should just build the argument parser using argparse's API directly and stop trying to optimize for it like I'm suggesting).

So what do we have at our disposal to build such a decorator? We have positional arguments so we know how many arguments are required without some specific qualifier. We have variable positional arguments (e.g. *args) to take an optional number of extra arguments at the end of the command-line. We have keyword arguments which are optional flags that one can specify. You could even have variable keyword arguments for major flexibility, but that just seems like a total lack of structure the CLIs just don't typically provide. With all of that you can reproduce getopt without any issue for long-form names. For short names, I would say you need to pass in a mapping of short names to long names into the decorator. Same goes for long names to help string (you can use the function's docstring for the main help for the app itself).

But where things get really interesting is when you take into consideration function annotations. That opens up the possibility of going beyond getopt and potentially supporting argparse's action, nargs, and type options. Take the type option as an example. You could say limit:int=10 to have a command-line option called --limit which only accepted an integer and defaulted to 10. This obviously could also work with float or any other type where you can just pass in a string to the constructor to get back an instance of the type. So you have a general case which can be useful, but you can you potentially special-case some things to get enhanced functionality where it doesn't make sense to simply take in a string?

Lists pose an interesting option as argparse provides both nargs for specifying the number of arguments to a single option, or the append action for accepting multiple instances of the same option and accumulating them. In my mind both can be expressed in a way that I think makes sense but some might view as too magical. If you specify names:list=[], then that supports the append action, e.g. --names Brett --names Andrea leads to names being set to ['Brett', 'Andrea']. But if you were to do names:['+']=[], then that would get the same result from --names Brett Andrea. In other words, the list type specifies the append action while a list instance specifies using the nargs option with the single item in the list acting as the value to set to nargs.

For booleans, I would want the use of the bool type to mean use either the store_true or store_false action based on what the default argument was. So turn_on:bool=True would use the store_false action since the argument is meant to be a boolean and it's default value is True, meaning that if the option was specified it represents the reverse.

Finally, the tricky bit is for files since that is a common command-line argument and you might as well open the file and close it for the function. The solution argparse uses is a specific FileType class where you can pass specific arguments to use when opening the file. The problem is that it doesn't support everything open() does, e.g. encoding. So what I would want to do instead is provide a partial function that took everything but the file path and then when it came time to call the main function, passed in the file path to the partial function, passed the returned file to contextlib.closing(), and then passed it on to the main function. You could even generalize a lot of this and simply say that whatever is specified as the function annotation, if it isn't a special-case like lists, then you call the annotation with what came from the command-line and if it provides a context manager it is used before calling the main function.

So those are my thoughts on using function parameters as a DSL for getopt++/argparse-- functionality on a Saturday morning. Honestly the most complicated bit would be constructing the arguments to pass to the main function in the right order, otherwise it's just introspecting on a function's parameters and making the proper call to argparse. But then again the real question is whether anyone thinks this at all sounds reasonable enough to code it up.

Playing with the Ninja build system

2012-04-28T20:21:00.001-07:00

Whenever I learn a new programming language I end up writing some toy examples to try to get a feel for what the language is about. This leads to the need to build code using many different compilers with their own flags, quirks, etc. Up until today I had used SCons for my build setup. But honestly, it always seemed like overkill to me. Because I only had about 5 programs to build per language with at most two files used to produce the program, a full-blown build system was never really needed. Add to the fact that I am building for languages that no build system would have built-in support for, it led me to always have a wandering eye for another build system I could use.

This past week someone on Google+ shared a post comparing configure+make, cmake+make, and cmake+ninja. I had never heard of Ninja, so I decided to have a look. It turns out someone had written a build tool whose only explicit job was to take a DAG, figure out what needed to be built, and then execute the commands for the build. No crazy metadata checks like Make, or fanciful features, just bare-bones building. Ninja was actually designed to be a target for other higher-level build systems like cmake which can do the pre-computation of what the DAG should be, leaving it to Ninja to drive the needed compilation.

What attracted me to it was that it was fast and the syntax was simple. I have code examples for 16 languages, of which 10 have build rules (one happens to be Python 2.7 as I pre-compile the .pyo files). Turned out to be a pretty straight-forward process to take my custom SCons commands and just translate them to the subsequent shell commands that Ninja would execute for me. They are a tad verbose in order to make sure that the ninja -t clean command would clean up all intermediary files (I'm looking at you OCaml, Haskell, Java, and Scala). But as I said, I typically never have more than 5 programs to build per language, so it wasn't that much of a burden. And if I really cared I could have written a Python script to auto-generate the Ninja files for me, but I decided the effort of writing the code would be just as much as writing the build files by hand.

I realize I could have used Make, but I honestly am not enamoured with that tool; requiring tabs just rubs me the wrong way. Plus it's rather slow in the common case of only changing a file or two compared to a complete build from scratch.

Overall, for my weird case Ninja worked out. For something more complex, though, I will consider looking at cmake+ninja as a build solution.

The re-launch of py3ksupport!

2012-02-14T15:29:00.001-08:00

The reason the past few blog posts I have written have been App Engine-themed is because I have re-launched py3ksupport! I did a complete rewrite of the code to make it more efficient (since I'm paying $9/month for the site) and at the same time moved over to HRD so as to guarantee the site is always up.

Before I discuss some unique features of py3ksupport, I want to point out that right now that 56 - 60% of the top 50 projects based on downloads of their latest PyPI release support Python 3. The reason for the range is that some projects had in-development support last time I looked and since that can change underneath me I wanted to cover the possibility the data was stale. But the key point is that 8 of the top 10 projects support Python 3 and one of them has support under development along with over half of the top 50 projects.

So I'm sure the site explains itself (and the FAQ fills in gaps), but I figured I should explain some of the more unique features of the site. One is the metadata rating given to each project. Basically I wanted to shame project owners into updating their project metadata. LOTS of projects don't bother to specify the Python support metadata for their projects which makes my life difficult and is unfortunate for users. For instance, of projects bothered to specify the exact versions of Python they support then users could easily tell from the Cheeseshop (nee PyPI) whether they could use the project based on what version of Python they are tied to. I might have to do some public shaming at PyCon if the situation doesn't improve. =)

The other key point is that I personally keep the front page up to date. If a new project shows up on the front page that does not have the proper metadata specified then I personally search online to find out the status of the Python 3 support. Since I get emailed each day when this happens the situation tends to get fixed that day and will be noticed within an hour of me fixing it (the length of time I have things cached).

I'm hoping to eventually move to measuring an project's popularity based on the download rate for the lifetime of the application (instead of just the latest release) to give a more accurate reflection of how popular a project is.

How I bootstrapped importlib

2012-02-06T16:44:00.000-08:00

If you have been reading this blog over the past five years I am sure you have read a post or five about my desire to bootstrap importlib into Python as the implementation of __import__. Well, as of today I'm willing to say that the difficult technological hurdles have been scaled! At this point the only thing holding me back from taking my code from https://hg.python.org/sandbox/bcannon#bootstrap_importlib and making importlib drive import statements are some small compatibility issues, integrating into the build process better, a code review, and python-dev sign-off. In other words all of the interesting problems have been solved, so I'm finally ready to write a blog post discussing how I pulled off what I have.

So how exactly do you import __import__? To begin, as with any bootstrap challenge, you need to figure out what is available to you so you know what your design parameters are. In my case I knew I couldn't import anything that required filesystem access since half of import is handling the search for a module (the other half is the actual importing); if I wanted to import a file I would need to essentially write half of import in C to work properly. This restriction also has unexpected side-effects, e.g. you can't rely on open() because that is part of the io module which is a Python module.

That meant I could only rely on built-in modules. If you run sys.builtin_module_names you will discover what is available directly within the CPython binary. The question then becomes if that is enough? It turns out that yes, those built-in modules are enough to perform an import. OK, so you know you have the bare minimum modules required to do an import, but how the heck do you get the built-in modules into the global scope of the module that imports module since you can't use an import statements?

This is when Python's dynamism comes in handy. Since the import statement doesn't do much more than pull in the module object and assign it to a variable at the global scope of the module, I just needed to get the module object for importlib and assign to its __dict__ the built-in modules I needed. Turns out that sys and imp are enough to allow importlib to handle the import of the rest of the built-in modules needed for import to work, so that kept this bit of code short.

But this brings up the next quandry: how do I create a module object of importlib? If I end up searching for importlib on sys.modules then I would have ended up implementing a decent chunk of import itself. So how could I get the module object? This is when frozen modules comes into play.

A frozen module is just a C array containing the marshaled code for a module (which is what a .pyc file is sans magic number, timestamp, and now file size of the source). Since marshal is a built-in module then frozen modules can be loaded without issue. That means you can load a frozen module without using import (much like importing built-in modules).

And that is all of the parts needed to import importlib w/o import. =) To summarize, you get importlib set as __import__ by doing the following:

Import the frozen module (i.e. read in a C array of a marshaled module object and unmarshal it)
Import sys and imp (built-in modules, so done in C code by calling key C functions which return module objects) and set it on the module object
Call Python code to import the rest of the built-in modules using sys and imp
Set Python-based __import__ on the builtins module

And voila! __import__ ends up implemented in pure Python code. Now I just need to clean up the code, fix the compatibility issues, rip out the old C code, and get python-dev to sign off. =) Hopefully I will get far enough I will have a lightning talk at PyCon with benchmark numbers to show this is actually all a good thing (including ripping out a ton of C code, especially if I can re-implement chunks of imp in pure Python =).

Asynchronous XML-RPC in Python

2012-01-26T09:58:00.002-08:00

Do you use XML-RPC (and specifically the xmlrpclib/xmlrpc.client from Python's stdlib)? Do you like multi-calls? Wish you could construct your XML-RPC multi-calls in a way so that you could make them asynchronous by constructing the call from scratch? Then you're in luck because I already did the hard work of figuring out the details for you! =)

In my case I needed to communicate with an XML-RPC server that supported multi-calls, but in an asynchronous fashion with a non-standard communication object (i.e. not sockets but App Engine's urlfetch). So I had to piece together how to make a XML-RPC client call that used multi-calls so that I could do it asynchronously myself.

The first thing to know is that a multi-call is a system.multicall function call to the server. It takes a sequence of arguments that specify what each individual function call should be. Each sequence item contains a dict with two keys: "methodName" and "params". The "methodName" key has a string value of the function name to call on the XML-RPC server. The "params" key holds the tuple of arguments (and it must be the tuple type and you must use a tuple even if it is just a single argument --think of what it would take for the call to work with *args) to pass to the function named in "methodName".

Knowing what function you are calling on the other end (system.multicall) and having a sequence of dicts specifying what you are calling in hand, you can make your call to the server. First you need to encode everything into XML: xmlrpc.client.dumps((seq_of_args,), 'system.multicall') (notice how I passed the sequence of dicts in a tuple -- and I mean a tuple, not a list). With the XML now in hand, you make the actual call to the server by making a POST with the XML as the body and set the Content-Type header to text/xml using whatever asynchronous mechanism you want to use to make the call.

What you get back from the server is XML that you can load with xmlrpc.client.loads(data, use_datetime=True)(chances are you will want the [0][0] value as that contains a sequence of results from each function call you requested). Do realize, though, that there is no indication of what function returned what, so you will need to correlate the index of the return value with what call you made (zip() makes this trivial).

And that's how you construct an XML-RPC multi-call from scratch in Python so you can make an asynchronous call. If you want to see an example of all this, see this code.

Grab bag of tips when working with App Engine unit testing

2012-01-24T15:44:00.000-08:00

[update: App Engine doc bug is being fixed and should be publicly visible in 1.6.2 or 1.6.3]

First, an announcement: register for PyCon! It's a great conference and tons of fun. Early bird ends on the 25th.

With that out of the way, this blog post is going to be testing an App Engine app. This can be, liking testing any complex system, is painful. And unfortunately the documentation on this subject is somewhat lacking for App Engine. But hopefully this post about random things to be aware of when writing unit tests for App Engine can help prevent someone else from having to learn the hard way like I did.

First thing you should do when testing an App Engine app is set up a testbed. This will allow you to stub out various parts of App Engine so your tests run in isolation without having stale state carrying over. You basically create a testbed which holds state. You then activate the system by setting some core environment variables with defaults. After that you do your own tweaks to env vars. From there you activate various stubs. And then finally you deactivate the whole thing to clean up...

But you need to realize there is an apparent error in the docs (which will be fixed no later than 1.6.3, possibly 1.6.2). The stated order of calling activate() and setup_env() are wrong; you should call activate() before calling setup_env() in order to set the default env vars before you begin to set your own.

Another thing you should know about is that setup_env() has an overwrite argument. This tells the method to overwrite any pre-existing env var; otherwise the call will skip changing any pre-existing env vars. This can call issues if your app presets some env vars that should be overridden during testing. I would honestly consider using that argument every time to make sure things work the way you expect.

And the last tip specific to setup_env()is make sure to pass all of your env var changes in a single call. Otherwise you can end up with some settings be changed as they have a default value.

In relation to handling users, to specify who the logged in user is you should set the USER_EMAIL env var. That will cause App Engine to return a user with the specified email address when calling users.get_current_user().

Random thing about email testing is that mail.send_mail_to_admins() does not write to the mail stub to verify any email was sent. This is because while testing there is no way to specify who admins are, so there is no way to know who to say the emails are sent to. So in order to test the function you will need to mock out the call yourself (personally I'm a fan of mock if you don't already have a preferred mocking library).

And finally, you can mock our the request and response objects for a handler. You can call a handler's initialize() method with the object to use -- for the request and response, respectively -- which allows you to mock and test how the handler does things.

Bonus for those of you using the Python 2.7 runtime: I would consider using unittest's discovery features to drive your unit tests. You can start from my driver code and add what you need. But it's nice to just be able to plop down a new test module and have it automatically picked up.

Working with App Engine backends

2012-01-17T15:53:00.000-08:00

This post is all about backends in App Engine, but first I want to make two personal points.

One is that I'm sorry I have not been updating this blog (or really contributing to Python and its great community) more this past year. It was a rather insane 2011 for me personally: I finished my Ph.D., got married, moved to SF (w/o my wife thanks to immigration), started working at Google full-time, moved to Toronto (yes, I moved internationally within the span of 6 months), transferred to the Google Waterloo office and started on a new team. In other words I have been stressed out and busy and busy continuously for what feels like ages. But now that I am back with my wife and I am done with the crazy international moving I don't expect 2012 to be anywhere near as nuts (nor any other year in the near future for that matter if I can help it).

Two, I am no longer on the App Engine team, so all that I am about to say (and will also say in the future as I have a couple of posts to write on App Engine) is from me as just another person using the service. I have been working on a redesign of a website of mine in my spare time and these next couple of posts are based on that work (if you following me on Google+ you know what it is, but I'm not ready for a full-blown Internet debut so I'm not going to link here yet). I don't have any insider knowledge here nor speak for the team.

With all of that out of the way, let's talk about backends! First you need to understand what an instance is to App Engine. When you hit a website hosted on App Engine, an instance is what is serving that request. Now an App Engine app can have anywhere from 0 to thousands of instances up and running at any one time to handle traffic. So think of an instance as a copy of your app that is running on App Engine serving traffic.

Normally you have some time restrictions on how long any of your instances can take to service a request. If it is an HTTP request from the general Internet you have 60 seconds. If it is from a cron job or push queue (i.e. from within your own app) you have 10 minutes. Now both time restrictions are actually rather long and you can accomplish a lot within those restrictions. And in the case of push queues, you can break your work up into multiple tasks so that you really can take as much time as you need as long as you can slice your work into 10 minute time chunks.

But there are just those times when you want something that will take more than 10 minutes, such as a process that is constantly polling some other service to get data. In those instances you want a backend since they are instances that run for as long as you want (assuming App Engine doesn't kill it for misbehaving or some other sysadmin reason). You can have a resident backend which never stops running or a dynamic backend that will stop running when it is done executing, but either type can take about 24 hours to run if it wants.

The way you use instances is to send requests to the backend's subdomain. You name your backends in your backend.yaml file which becomes the subdomain off of your appspot.com subdomain for addressing your backend. So, assuming you have a backend name "forever" for an app named "spam", the domain for the instance will be forever.spam.appspot.com. You can then create URL handlers in your instance and hit those URLs to cause your instance to do it's long-running thing.

Now for those of you who have been using App Engine for a while you might notice that the subdomain for a backend looks just like the specific version ID URL for your app, and that's on purpose. Thanks to it having the same format you can specify a backend name anywhere you can specify a version ID for your app and have it use your backend. E.g. to have a cron job that hits your backend you specify your backend's name as the "version" of your app to hit in your cron.yaml.

And before anyone worries, your backend defaults to being private to your app. If you really want your backend reachable from outside of your app (i.e. the public) you can specify that in your backend.yaml.

While backends sound fantastic and address an old complaint against App Engine and its lack of background processes, I have two warnings/tips about using them.

First, the tip: you can't use the OS X AppEngineLauncher to use backends. When you launch a dev_appserver you need to specify the --backends flag. You can do that manually in the AppEngineLauncher, but you need the flag for appcfg.py as well which you can't specify. So if you want to use backends (and thus want to specify the --backends flag) you will need to use the command-line version of the tools. Obviously this is a minor thing, but it took me quite a while to realize that was why backends were not working for me.

And my warning is that don't expect to use backends on the free quota. While you get 9 free hours of backend usage, you can blow through that surprisingly quickly. But the real issue is all of the other quotas that you have on App Engine. For instance, I was hitting another server to get data to process. What ended up happening is that I was maxing our my incoming bandwidth quota for urlfetch. So then I started to write code to throttle my requests. But even then I occasionally hit the quota. To properly handle that I would need to grab the quota exception and then retry or heavily throttle myself so that I never hit the issue. At that point I was writing code to throttle my requests and to retry when I managed to request to quickly; that's just silly when I have push queues to do that for me automatically. So I personally ended up not using backends in my case (although ironically after I refactored the code I started hitting another free quota limit which led to me turning on billing, so now I'm using $0.05/day in quota, so the app is just costing me $9/month).

But when you are willing to pay for backends they are great for what they provide, but you just have to realize they are just another tool and are not always going to be the best solution for everything (especially if you don't want to pay to use App Engine).

PyCon 2012 CFP ends in a month!

2011-09-12T11:32:00.002-07:00

The title says it all: the CFP for PyCon 2012 ends in a month, so you should seriously start thinking about getting those proposals in! As I say every year, you do not need to be a Python celebrity or have presented previously in order to get a talk accepted. Having been on the PC from the beginning of PyCon I can tell you that we very happily give slots to people we have never heard of. As long as your proposal shows you are organized and your topic is interesting your talk will be given a serious chance for acceptance.

And sorry about the lack of posts. My Python free time is limited at the moment as most free time is spent talking with my wife over a G+ Hangout thanks to US immigration perpetually dragging their feet, keeping her in Toronto and me in SF and causing general misery spread across the continent. That means what little time I do have I spend on trying to get importlib bootstrapped (which you can follow in my sandbox/bcannon repo in the bootstrap_importlib branch) instead of blogging. But once the bootstrapping is done (or I come up with some rather clever thing) I will do a technical post.

How to import a module from just a file path

2011-07-16T15:11:00.001-07:00

Dr. Brown asked on Twitter whether there was a single expression (e.g., no semi-colons or abuses of and or or) that could import a module from a file path, either through a stdlib function call or just constructed from scratch; mod = import_from_path('some_file.txt'). Because it involved import, I got cc'ed on the tweet while various people tried to come up with a solution. In the end people realized that it's not possible in Python 2 (but it is in Python 3). But how hard could it be, right? Right?!?

Well, the devil is always in the details. While there are some functions in the stdlib that come close (e.g., execfile() and runpy.run_path()), none of them return the module, but the globals dict. Other people tried to come up with something using 'exec', but it doesn't return anything (in Python 2) so you have to assign what you planned to use as the module ahead of time (that's two lines total if you weren't counting along).

In Python 3, though, this is all totally doable thanks to 'exec' becoming a function instead of a keyword:

mod = exec(open('path', 'r').read(), __import__('sys').modules.setdefault('name', __import__('imp').new_module(name)).__dict__)

This is obviously a complete hack and not something you should use in the real world. But it is a single expression (meeting Titus' restriction) and it even puts the module into sys.modules so that future imports will not have to re-import the module. This hack, though, does assume that you are only importing modules and never packages.

So what would it take to do a proper function that took (at least) a file path and ended up importing that file? First thing is whether you wanted any arbitrary path to be used, or only something for properly named Python modules (e.g., ending in .py, packages being named __init__.py). If you go with the former you need to specify the module name and whether it is a package (and if it is what the value of __path__ should be), else you can just gleam all of that info from the path itself.

After that the next step is to create the module object and populate it. The bare bones thing to do is create the object and specify the __name__ and __file__ attributes, along with __path__ if it applies. This has to be done as a module might do something with those values as a side-effect of importation (and remember folks, never launch a thread as a side-effect of an import! Deadlock awaits those who don't heed that warning).

Third, you create the code object. Now you could technically just read the file and pass it to 'exec', but that won't set up the file path, etc. in the code object for easier traceback issues that may arise when you do the actual executing of the module code.

Fourth, you actually execute the module code and finish initializing your module.

Now all of this ignores storing the module in sys.modules or even checking if it even exists in there. It also doesn't add the module to sys.modules so that future imports can use the module instead of having to do the same thing again. But if you are honestly wanting to import a module by specifying a path, you probably don't care about any of this anyway. And because you ignore sys.modules, you get to ignore the import lock.

So, taking all of this together, here is a basically untested function that does what Titus basically wanted (works in Python 3, but changing it for Python 2 is literally just using the 'exec' statement instead of function):

import imp

# Step 1
def import_from_path(path, name='', package_path=None):
    """Import a module from a specified file path.

    If the module is a package, set package_path to a list of directories that
    is to become __path__.

    """
    # Step 2
    mod = imp.new_module(name)
    mod.__file__ = path
    if package_path is not None:
        mod.__path__ = package_path
    # Step 3
    with open(path, 'r') as file:
        code = compile(file.read(), path, 'exec', dont_inherit=True)
    # Step 4
    exec(code, mod.__dict__)
    return mod

My personal plans for Python 3.3

2011-06-28T21:34:00.000-07:00

Now that my life has hit another purgatory moment (waiting to here when my wife's permanent residency interview will be and still in the ramp-up period at work), I have had some time to think about what my personal goals will be for Python 3.3. This list is in no particular order and I make no promises to actually do any of it as I only have roughly 249 days until alpha1. =)

First, I want to bootstrap in importlib as the implementation of __import__. I think I can simply freeze importlib._bootstrap, do some imports of built-in modules, and then have it almost work. The reason it is "almost" is that importlib relies on _io, which itself relies on importing the os module. That's a problem when you are trying to essentially import what will act as import. I need to have a look still at the _io code to see if the importing of os can be postponed until after importlib is up and running.

Second, I need to finish PEP 399. Before the whole work/immigration whirlwind started I submitted this PEP to python-dev and got a bunch of feedback on it. The usual push-and-shove led to what seemed to be a general consensus of what the PEP should be striving for, so I think I can wrap it up and get it accepted once I have time to incorporate the feedback (mainly watering down the wording so it is less technical).

Third, I should try to finally get PEP 362 accepted (tracking bug). The code has been up on the Cheeseshop as pep362 for over 4 years now without any complaints. Since I know there are users out there (Michael Foord keeps asking me to finally get the code into the stdlib) I figure the API is generally acceptable and there are no secret bugs creeping in there. Basically I just need to rip out the silly use of inspect that is left since I dropped Python 2 support in the development version, make sure I have proper code coverage, and write docs.

Fourth, I need to get an Amazon or Rackspace instance up that generates a coverage report for the stdlib on some reasonable schedule. This first requires me to finally review and commit Brandon Rhodes' script which gets coverage reporting much earlier in the interpreter startup than it does by default. Once that is done I need to get a VM going somewhere that can freshly check out Python, coverage.py, run the test suite under coverage, and then make the coverage report available for viewing. This will make the test coverage not only more visible, but easier for people to help increase it as a starter project when contributing to Python.

So that is everything I am hoping to do before March 3, 2012. I plan on making the coverage and importlib bootstrapping my priorities, although I will finish PEP 399 first since I hate leaving open PEPs lying about. And I am definitely going to try to not take on any more Python projects until I get through this list as this is already plenty for me to do. =)

Framerates for movies and the debacle that they are

2011-04-22T15:15:00.000-07:00

One of my great weak points is technology, especially those involving movies. While I wouldn't go as far as calling myself a cinephile (e.g., I don't know every random fact about my favorite movies), I do love watching films which means I like trying to make my viewing experience at home rather nice (which honestly has not been that great thanks to my long-lasting student budget).

One of the latest things being debated about the quality of cinema is the frame rate of films. Currently movies project at 24 fps. ATSC (digital broadcasting version of NTSC) is typically 30 fps at 1080p although it can also do 24 fps and the funky rates film and old NTSC video show at (PAL is 25 fps but I'm North American, so I am going to stick with what I know in this blog post). Blu-ray can do 24 fps at 1080p. So everything these days can handle the native projection rate of 24 fps of 35 mm film.

But really, 24 fps?!? Any hardcore gamer is going to tell you that frame rate is unacceptable. You want at least 30, if not 60 fps. As it turns out, so do people watching movies. This is why most decent TVs these days at least output 120Hz video which helps to artificially raise the frame rate and lead to a smoother picture, no matter if the original material is 24 or 30 fps (or even 60, but that is typically reserved for video game output). And even newer TVs can do 240Hz, which leads to an even smoother experience (supposedly; have not experienced it myself).

But what about in the theater? Everything is still filmed and made for 24 fps and that is what projectors (digital and analog) use. Well, two of the more cutting edge directors currently working want to change this. James Cameron has talked about filming Avatar 2 & 3 at 48 or 60 fps. Peter Jackson has gone beyond saying and is actually doing by filming the Hobbit at 48 fps. According to Jackson the higher frame rate not only looks better, but it makes 3D something that is actually bearable to watch (i.e., the nasty flicker goes away). And it turns out most digital projectors can have their firmware upgraded to project films that use these higher frame rates. And with 48 fps being a multiple of 24, these directors could easily film at 48 fps and make 35 mm prints where every other frame is dropped so old projectors can still show at 24 fps.

But what about watching at home? While it's great that seeing those select films I want to see in the theaters will be projected at a higher frame rate (e.g., the Hobbit), what about when I want to watch at home? This is where that 48 fps gets sticky. Both blu-ray and ATSC do not support 48 fps at any resolution. Both do, however, support 60 fps at 720p. And I am willing to bet that Netflix streaming won't handle either 48 or 60 fps until we all have fiber connections. Another issue for the 48 fps is that it is not a common divisor of 120 (Hz), making it potentially nasty to artificially raise the frame rate; luckily 48 is a divisor for 240 (Hz).

So the question becomes, how does Hollywood decide to handle this theater/home frame rate discrepancy that is coming our way? Well, with this being Hollywood, I am willing to bet they do it in the way that screws over the home user the most, forcing you to go to the theater for the best experience. That would support the idea of going with 48 fps in theater projection. That way they still get a higher frame rate that helps out 3D projection, they can easily cut 24 fps versions for 35 mm prints, and they can stick with blu-ray and such at 24 fps as well. All of this while getting to keep a nice hook to get people who care about video quality to come to the theaters to pay for your expensive movie ticket. They can then come out with a new home video standard that supports 48 fps and force us to all buy our movie collections yet again (you know George would be happy to sell you Star Wars for the bazillionth time on a new format, although unless he lets Han shoot first I ain't going for it). If they go 60 fps they can at least do 720p60, but then again do we really want the drop in resolution?

importlib: doing it, and doing it, and doing it well

2011-03-23T18:43:00.000-07:00

Earlier today I pushed a change to CPython which allows importlib to pass Python's test suite as the implementation of __import__ (sans failures because nothing expects __loader__ to be universally set on modules). This means that there is no known compatibility issues standing between me and making importlib CPython's implementation of __import__.

So what was the final hurdle? It basically boils down to code objects and their immutability. When you import from bytecode you are essentially loading a code object which represents the module (basically using marshal.loads()). The issue is that code objects embed what file they were created from (the co_filename attribute). But what happens if you relocate a .pyc file? Should co_filename point to the place the file was originally created at or at its current location? Turns out Python thinks it should be the latter.

The issue with updating the co_filename attribute on a code object is that it can only be done by C code; attributes on a code object are immutable from Python. My original plan was to modify marshal.loads() to take a filename argument which represented where the marshal data came from. That way marshal could fix co_filename while keeping the attribute immutable. But this didn't work.

It turns out that the attribute used by __import__ for fixing co_filename is not as thorough as the recursive solution I came up with for marshal.loads(). There is a loop that looks for objects to fix, but as soon as one of them is accurate the search is stopped by __import__. Rather than replicate this somewhat odd solution for marshal, I simply exposed a private API in imp for me to use in order to mutate a code object's co_filename attribute. Not an elegant solution, but since I am taking a PyPy view to import (i.e., it's my job to make importlib as compatible as possible and not tweak __import__ to fit my needs) I didn't have much of a choice.

Regardless, it feels very satisfying to have an implementation of import that passes the test suite now. This means I can work towards bootstrapping importlib as the implementation of import. I am planning to go with some bytecode-freezing solution for CPython where there is a build rule which recreates the bytecode any time importlib._bootstrap is mutated. That will remove the worry of having to rely upon some external file for import to work (otherwise I could come up with some import shim that did enough to import importlib directly). And with performance of importlib already being acceptable I feel fairly confident that this bootstrapping will happen in Python 3.3 (I have started work in the bootstrap_importlib branch of my personal repo).

PyCon 2011 wrap-up

2011-03-20T17:56:00.000-07:00

tl;dr version: it was awesome! Read on to find out what I did at the sprints and two themes that I found at the conference.

Since I have not had a chance to blog until now, people like Nick Coghlan and Brian Curtin have beaten me to writing good summaries of the VM summit, language summit, conference, and sprints so I won't worry about doing an overall summary of everything.

At the sprints I almost exclusively focused on my Python 3 on PyPI website. After squashing a bunch of bugs I added two more popularity metrics. The first metric is download rate. This is to help balance out against total download count which favours older projects thanks to simply having been available longer. I also added the poll results as found on the front page of www.python.org (look in the upper-right corner of the page). The download rate has about 50% of those projects converted which is great, while the poll results is at 16% which is also very good considering people are explicitly asking for those projects to be converted because they were not at the time of voting. And because people seem to have some odd desire to see the crap code I wrote for this website, you can find it here.

As for the two themes I see gaining more exposure in the upcoming year, one was obviously Python 3 support from projects. There is now a book out by Lennart Regebro [Amazon] (which I wrote the foreword for), a good number of projects have already switched, Django will be switching this summer, and at the sprints Jim Fulton started porting zc.buildout to Python 3 (which happens to be the largest Python 3 holdout according to my website at the moment). In other words the momentum is there for Python 3 and I continue to not worry about it.

The other theme which gained more traction is separating Python the language from CPython the interpreter. PyPy's performance is now good enough to make it a serious consideration for anyone to use. And once they release a version supporting Python 2.7 (IronPython beat them to Python 2.7 compatibility) then they will be in a position to become the de-facto interpreter people use when they don't have specific requirements (e.g., embedding Python in a Java program and thus requiring Jython). What this means (at least to me) is that python-dev is going to need to start to shift towards a stance where the Python language takes precedence over CPython the interpreter. While CPython will continue to be the reference implementation (don't ask if PyPy will become it; I don't think they want that responsibility honestly), certain considerations are going to be needed in order to make sure that things we do in the Hg repository do not end up hurting the other VMs.

To help with this goal of making sure the Python language does not play favourites, I am planning on doing three things. One is a PEP I am drafting up dictating that C extension code in the stdlib must be kept fully compatible with any pure Python equivalent code. It will also say that any extension module with no matching pure Python version must get an exemption to be included in the stdlib. This is to prevent all the VMs from having to re-implement every extension module on their own, essentially leading to four versions of the same module.

The second thing is getting importlib to become the de-facto implementation of import in Python. Every single VM hates implementing import. It's subtle and complicated, making it very tough to implement in a way that is conformant. By moving it to pure Python, importlib will allow all of the VMs to simply use importlib and thus not have to re-implement import.

Lastly, a very long-term goal is to break out Python's stdlib into its own repository. This will make it even more obvious that CPython is the reference implementation of Python and that it is not special in any other way. It will also allow the other VMs to use the same code base to prevent duplicated work (the key contributors on the other VMs are already receiving commit privileges on CPython so they can start fixing bugs now). This is a very long-term project of mine and won't be started until my other two goals mentioned above are finished. In other words my goal is to start at PyCon 2012 with this.

Secret site goes live: Python 3 Support on PyPI

2011-03-04T14:23:00.000-08:00

When the Python 3 Wall of Shame launched a couple weeks ago, I (along with others) noticed discrepancies (discussed in the comments of the announcement blog post) between the real world and what the list said (e.g., doctutils was listed as not supporting Python 3). A similar issue came from On Python 3 Yet? Both websites seemed to be to accidentally spreading FUD about Python 3 support on PyPI thanks to the projects on PyPI not listing Python 3 support properly. While in no way intentional, it does cause issues when people use these sites as "proof" Python 3 still lacks support from the community.

So how did this happen? Well, projects on PyPI are listed as supporting Python 3 when they set the proper Python 3 trove classifier (instructions on how to fix this are in the Python 3 porting HOWTO). What I suspect has happened is that projects which do support Python 3 but don't have the proper classifier either (a) are not aware they should set it or (b) don't care/too lazy. Either way it leads to a mis-representation of what projects do and do not support Python 3.

To deal with this, I created a website to measure Python 3 support on PyPI. What differentiates this website from the others is that I am personally curating the list of projects that support Python 3. So while PyPI says docutils does not support Python 3, my website does. Same goes for projects which have forks that support Python 3 (e.g., setuptools "supports" Python 3 through distribute). I have even begun to flag projects which have working support in their version control system as "maybe" supporting Python 3. I completely understand why the creators of the other websites don't do this; it takes work. But since I have more invested in Python 3 than most people I am willing to put the work in.

If you go to the home page you will notice that I list a project as either supporting, not supporting, or maybe supporting Python 3. The latter case is there because a ton of projects do not specify at all if they even support Python 2! My hope is that projects begin to properly set their trove classifiers. This not only benefits the community by letting people know if they do or do not support Python 3, but because it can be used to know how far back they support Python. Take Django 1.2.5 as an example. It does not have trove classifiers set to let me know that it supports Python 2.4 through Python 2.7. Heck, I don't even know if they have tested against Python 2.7 yet. But if they set their trove classifiers this would be known both visibly and programmatically.

I am asking the community for help combating this problem of projects not listing their Python 3 support properly. If you know of a project which supports Python 3 somehow but that fact is not known on this new website, please either list it over on the Convore discussion on this topic, leave a comment here, or leave a comment on Google Buzz along with proof of the Python 3 support (e.g., a link to an official page stating the support). If you tack on a project's PyPI name on to http://py3ksupport.appspot.com/pypi/ (e.g., http://py3ksupport.appspot.com/pypi/docutils) you can see if the project's support has been picked up yet or not. As I said, forks do qualify for support as well as do projects which have functioning support in version control but have just not done a release yet. This is all rather important for the top projects as listed on the home page since that is what people will really pay attention to in order to notice Python 3 uptake.

To give people some scope of how much of a difference this curation makes, consider the top 50 projects by downloads for any release. 14 of the top 50 (i.e., 28%) support Python 3 somehow. But if you went by only trove classifiers you would think only 9 project (18%) supported Python 3. Even hard coding modules absorbed into Python's stdlib would leave off 2 of the projects. If you look at latest release downloads you have even bigger support and discrepancies: 16 out of 50 projects support Python 3. But of those, 8 I had to manually flag (4 of which are not from the stdlib). Percentage-wise this manual effort makes a difference and gives a much better indication of how much support there already is for Python 3.

So this is what I have been working on for the past two weeks. I learned a lot about App Engine and creating a scalable website. The whole thing updates every 20 minutes from PyPI in terms of project metdata changes. I also update download totals every day. Long term I hope to add more metrics to gauge what the top projects are (e.g., listed dependencies, Google Code searches, etc.). I also want to develop a Chrome extension which will let a user know when a project supports Python 3 when viewing PyPI even if it doesn't say so (maybe even getting to the point that people can notify me through a button click that the project actually does support Python 3). I also want an API so that people on other websites that use different metrics can have access to the same data I use to mark a project as supported or not. But as of right now, I'm ready to take the weekend off. =)

PSF core grant, the last days: the website works (so far)!

2011-03-03T23:18:00.000-08:00

The last couple of days I have been working hard to hack together (and I mean hack; it is not the most elegant code or solution I have ever written) the remaining bits of the website. I am currently waiting for the task queues to finish doing their thing so that I have a complete data set before I go "public" by asking the community for some help with something. But so far things are working out and hopefully I will wake up in the morning to no errors and the data set complete!

The biggest challenge I have run into over the last couple of days is App Engine's 10K limit on task queue data. that ain't much data. So I had to do some reworking of my data syncing workflow to use the datastore as a temporary storage solution. It worked out, but I'm sure I am doing something silly. At least I made sure everything happens in a transaction so that if anything fails the task queue will simply try again until it works (hopefully it's a transient failure and doesn't require me to fix some code =) .

Introducing mnfy

2011-03-01T18:13:00.000-08:00

Remember back in November when I did a blog post about an AST-based minifier for Python code? Remember how I said I gave up on it out of frustration over how the AST represented things in a way that didn't mirror the syntax? Yeah, that "giving up" part didn't sit well with me, so I created mnfy to help minify/obfuscate Python 3 code.

It's basically a little project that I created which helps in minifying code. I got over my issues with the AST by simply thinking it through and figuring out exactly which AST nodes needed special treatment (e.g., how to detect an 'elif' clause). Once I did that I was able to work on some of the transforms which is where the fun is.

The code currently doesn't do anything fancy that human beings who like to muck about with minifying/obfuscating Python don't already do:

Eliminates unneeded whitespace
Minimal parenthesis usage (mostly)
Uses hexadecimal numbers when it will save on characters
Combines imports into a single line that are defined in a row
Drop unused constants (including docstrings)
Functions into lambda definitions

As I said, nothing nuts, but at least takes some of the tedium out. The transformations of the syntax are also separated into safe and unsafe transforms. I consider safe transforms things that are semantically equivalent to what was changed in 99% of the cases (i.e., if you muck with execution frames you are on your own). Unsafe transforms will work in 90% of the cases, but are not exactly semantically equivalent (e.g., lambdas are not entirely equivalent to a defined function). Thus users can choose their exposure to transforms which may or may not break their code. You can also skip transforms all together and simply go with the elimination of whitespace and unneeded formatting characters (e.g., parentheses). In the latter case I am able to minify the entire standard library and still get back the exact same AST.

Anyway, this was just a fun little project that I don't plan to take that seriously (i.e., don't expect massive churn on the code any time soon =). If either minification/obfuscation or working with Python's AST interests you, there is a talk on using Python's ast module and another on obfuscating Python code at PyCon you can attend.

PSF core grant, day 39: darn scalability

2011-03-01T09:35:00.000-08:00

So the server component for the secret website is nearly complete, when I started to test the site through App Engine's dev appserver, some issues arose which were not triggered by unit tests. So I spent Sunday and yesterday (counting as one day thanks to working on bursts on-and-off) working on fixing those issues.

Otherwise the only other news is that the Python-Dev In a Box code is up. Please realize that it will not work for you! It is currently geared towards using the Hg test repo which is not ready for general use (e.g., it's missing some commits that landed in svn). I also am going on an assumption a patch I submitted to coverage.py about a __main__.py file for the repo directory gets accepted, and so I have a file I manually copy over in that instance until that happens. In other words, if you decide to play with the code realize that it probably won't work and so bug reports about it not running are probably premature. =)

PSF core grant, days 37 & 38: making Python-Dev In a Box a reality

2011-02-26T22:32:00.000-08:00

I got enough positive feedback on the Python-Dev In a Box idea from my last blog post that I decided to implement it. Over yesterday and today I have been writing some scripts to help bootstrap people into contributing. I will make the code and repository public as soon as I can get the remote repository at hg.python.org created.

So the goal of this project was to make it easy for people to jump in and start contributing at a sprint or on their way home from a conference. That meant providing the code, tools, and information needed to contribute. It also meant doing as much upfront for people as possible so as to not waste time for what everyone would end up doing along with not requiring a network connection. And all of this, at max, needs to fit in 700 MB on a CD.

First, there is a script called make_a_box.py which creates a Python-Dev In a Box instance. It will prompt you for what you want to include in the box:

CPython (from an Hg repository at the moment, but I might patch it to be from svn if Hg is not ready by PyCon)
Devguide
PEPs
coverage.py
Mercurial and TortoiseHg
Visual C++ Express (the Web installer)

When reasonable it will build what is provided. In the case of CPython and the various bits of documentation, the docs get built. For coverage.py it runs it so you have a basic coverage report for people to work off of.

For users, I included two extra scripts. One simply builds CPython (on UNIX). While building is very straightfoward (and documented in the devguide), I needed the functionality for generating the coverage report. Since I needed the code anyway I figured I would make it a stand-alone script. About the only real perk to it is that it uses multiprocessing.cpu_count() to run the Makefile with as many cores as you have (although CPython compiles in a fairly linear fashion so extra cores doesn't do that much).

I also provided a script that runs the test suite in the fastest, most rigorous way possible. Once again, while documented in the devguide, people can get a little lazy when it comes to running the entire test suite, so hopefully having a script that uses the "best" options will lead to more rigorous testing overall.

My plan is to create a Box for the PyCon sprints so that people attending the sprint can simply copy from a CD or flash drive everything they need to get going. This should speed up people getting started, leading to more productivity.