Journal

42

Sat, 30 Jul 2011 21:41:01 +0530

My answer to the Ultimate Question of Life, the Universe, and Everything. Answers, actually; looking at our existence and purpose from different perspectives.

One common answer to this dilemma is to assert that life is meaningless and without purpose. A useless answer without information or philosophical connotations. Ofcourse, it is the first step that someone needs to take to break away from the notion that life has a God-given meaning or purpose. However, we can easily provide much more meaningful answers.

From a biological perspective, it's survival of the species, right? While that applies to animals in general, it doesn't apply to humans in modern society. Survival is hardly a problem anymore- there's enough food and medicine for everyone today. We've evolved to stop killing each other, have learnt to respect each other and peacefully co-exist. We're under no immediate external threat either.

This is something that defines us as a species fundamentally. Showing kindness, love and compassion towards other human beings; respecting other members of their species. Many of modern society's problems can be boiled down to humans not respecting each other enough. In a way, we are the biggest threat to our own existence: our society still creates poorly educated, short-sighted and immature members and we have to learn to correct this.

History has many clues as well- what's common over so many years of existence? Where are we heading? Reducing the mundane day-to-day activities like food gathering to gain leisure time. Hunter-gathers turned into farmers and started forming societies when they realized the benefits: the mundane day-to-day activities are shared, creating more leisure time for everyone. Time for them to paint, play music, invent sport, invent language. This is how humans wanted to utilize their free time- to use their minds to make the world a more interesting place.

Continuing along that line of thought, we need to question the purpose of human society. In general, it structures individuals so they can enjoy and build upon each others' intellectual outputs. Essentially, it leverages the power of the collective- to learn from mistakes of other people, and build upon the successful ideas. Some things like medicine help in survival, but that's hardly ever the direct objective of exercising physical and mental capacities. Looking at it from the race's perspective, society tries to encourage and nurture new creations; an individual is but a servant working to improve conditions of existence for his race. An individual who ends up creating nothing with her free time is a failure from the society's perspective- it has failed to provide the right motivation and incentives to get her to produce something for the race.

Looking at it from the individual's perspective, life is like an amusement park. Let's take examples of different people in the park.

Bob takes all the rides and has a ball of a time his entire life. He has fun enjoying everyone else's creations, but does not create anything.
Leia walks into the haunted house pretty early on and it scares her so much that she refuse to try the other rides; she just stands aside and observes everyone else having fun until her life ends.
Aaron walks into the roller coaster pretty early on, and is so impressed with it that he keeps riding the different roller coasters all his life, and asserts that all the other rides are stupid without so much as finding out what they are. He invents yet another roller coaster in his lifetime. Had he tried out the water rides, he might have invented something much more awesome- a hybrid between a water ride and a roller coaster.

All three people are failures in some way. Bob never experienced the joy of creating something, Aaron only had one kind of experience, and Leia never experienced a single thing in her entire life. So, from this perspective, the purpose of human life is to educate oneself to enjoy most (if not all) of the rides, and create something remarkable to leave behind for future generations to enjoy and build upon.

Happiness

Sat, 30 Jul 2011 21:36:21 +0530

Aristotle enshrines happiness as a central purpose of human life and a goal in itself. Examining this closely leads us to a deeper question: are humans simply pleasure-seeking automaton or something more? We have to question the nature of happiness, how invariant it is, and what it means to us.

Instincts are the result of years of genetic evolution. They've programmed us to feel happy when we do something that's genetically desirable. Natural selection has ensured that the traits that made our ancestors survive has been passed onto us, and the animal in us is programmed with all these survival instincts. I like to think of humans as animals with high computational power at their disposal. That computational power too, has developed over generations of evolution: at the time, the bottleneck in species survival had less to do with physical characteristics, and more to do with the ability to think through a survival problem and design a solution.

There is nothing divine about our instincts. Many of those instincts are not desirable in modern society. For instance, the desire for high-calorie items has come out of years of food insecurity. Today, excessive and reckless eating leads to obesity. The raw aggression shown by males in a species when they're searching for suitable mates is something modern society looks down upon. Irrational prejudice is yet another example: when being chased by a predator, instead of objectively evaluating the strength and agility of the opponent, the prey simply runs as fast as it can. It's a shortcut that helps in day-to-day life by caching and short-circuiting frequent computations. Unfortunately, after a while, many people forget that it was a shortcut in the first place and fail to look beyond it.

Does that mean that we should override every possible instinct and emotion using our higher rationality? That sort of extremism leads to a different kind of problem: humans will turn into computers the inability to produce creative work. Apart from this, it creates a huge amount of internal conflict: the instinct says one thing, and rationality says the exact opposite thing. Instead of fighting instinct with rationality, we must train so as to align both of them.

Happiness is meaningless. It is neither divine nor invariant- people can program themselves to feel happy when they do something right using simple incentive mechanisms. It's as mechanical as training a dog. Modern society constantly tries to design incentives to get its members to behave in a certain way. For instance, by giving away prestigeous awards to people who do ground-breaking research, it essentially ensures that the people who contribute most to society are generally happy.

Yes, a well-trained human being doing the things that she likes is generally happy. My point is that this is incidental, not fundamental. For this reason, following happiness blindly without designing the correct incentive mechanisms and without the proper mental training can lead to a meaningless animal-like existence.

Why we need Ubuntu

Sun, 10 Oct 2010 23:09:42 +0530

I must admit that I was pretty anti-Ubuntu sometime ago. My ideology was that the hackers can get onto a real distro like Debian or Fedora, while end-users can blissfully use Windows. Why create something that's neither here nor there? Why bake a half-assed distro to trade off hackability in the name of tight integration and out-of-the-box functionality?

I vividly recall opening up radios and alarm clocks in my childhood and being fascinated by them. I learnt about electricity from AA cells, small electrical bulbs, and tiny motors in toy cars. I learnt about light while playfully burning paper under the sun using a magnifying glass. My intuitive understanding about magnetic poles can be traced back to my experiments with refrigerator magnets as a child. Right from then to when I discovered HTML in web pages; it all relied on the fact that I was free to play with all these devices. Even without writing any code, and just playing with my new Linux machine, I learnt a lot of things that were opaque on Windows. Other kids might have been exposed to the same devices, but might not learnt as much - nobody's consciously thinking about how hackable everyday devices are.

No, most normal people don't care about whether something is open source or closed source. They want do their email and and get off from work quickly. This isn't about them. This is about those curious kids who, given the right environment, can be the innovators of tomorrow. If all devices were sealed black boxes, their curiosity would die out. Get them a hobbyist electronics kit, right? Wrong. It's so over-whelming, the kid'll probably throw it out of the window and go back to its comfort zone- playing with teddy bears. Dumping Debian on a Windows user is like that- it's incredibly inaccessible: only those in the elite top bracket will switch, and we've potentially lost many great minds.

If someone really wants Windows, they should buy it; but they should not have to pay for it simply because there isn't a viable alternative. Windows is as bad as a sealed iPhone- there's nothing wrong with it, but curious people should have a choice to explore the richer world that we've created. They should have access to alarm clocks, radios, and software they can pry open. Ubuntu must become widespread- so much that every one of those curious kids has access to it. It should be marketed as a "viable alternative to Windows for the curious". I consider it a grave error in the educational system if a school adopts Windows instead.

Ubuntu has to make some compromises like including proprietary software as part of its default installation; I'm all for it if it makes users comfortable while giving them a peek into the richer world. After all, very few will pick Ubuntu over Windows just because it's open source. Personally, I wish it were more hackable, but there are other distros like Debian for that.

The true nature of Git

Fri, 27 Aug 2010 15:20:06 +0530

The central idea is simple- Git is not a versioning system at all; atleast not in the way CVS, SVN, Perforce, or Mercurial are. This is the reason no versioning system can even come close to Git in terms of features, speed, and compactness of storage.

So what is Git? It's a key-value object store (in .git/objects) plus the infrastructure to play with it. The keys are SHA1 hashes and values can be four kinds of objects: blobs, trees, commits, and tags.

Git is often described as a "dumb content tracker". It is dumb because traditional versioning systems use a delta storage system; they use hashing algorithms to find out the differences between the way your project currently looks and the way it looked in the previous commit, figure out how to represent the differences, and finally store them in compressed deltas. Git on the other hand, simply stores a complete snapshot of the repository everytime a commit is created*; consequently, committing is a very cheap operation in Git. It is a content tracker because it separates the content of the files from the properties of the files and directory structure.

.
|-- README
`-- lib
    |-- inc
    |   `-- tricks.rb
    `-- mylib.rb

2 directories, 3 files

Note: The same object is never stored twice. So, if you have a
file that's untouched between five commits, all five commit
objects will reference the same blob object. More correctly, a
blob isn't a delta- it's full text.

The above diagram is what one revision of the repository looks like in the object store. Now imagine many such commit objects connected in a DAG; consquently, when a the SHA1 hashes of the tips of the DAG (called heads) are known, every commit should be accessible. The end-user can't be expected to memorize cryptic SHA1 hashes: that's why refs in .git/refs exist.

Using this model has several ramifications. Let's say there are 100 commits in the repository- I add a commit and slap the ref moo onto it, while you create one and slap the ref foo onto it. Voila! moo and foo are branches. There's one last piece of the puzzle missing: when a command such as git commit is issued, how does Git know which commit in the DAG to link up the new commit object to? In other words, what is the "current" commit or ref I'm on? There's a special symbolic ref .git/HEAD to take care of this; switching to branch moo is as simple as a echo ref: refs/heads/moo > .git/HEAD.

The most obvious problem that comes to mind is size. If Git doesn't deltify objects, how does it afford network operations? Enter the Git packfile format. Using git pack-objects, git unpack-objects and git repack, it deltifies and packs loose objects (in .git/objects) into packfiles along with an index to locate the loose objects again (in .git/pack). To minimize network transfer, Git will ideally try to find the packfile that contains almost all the requested objects and repack it to include/ prune objects before sending it over the wire. End-users don't need to specify any objects- they only ever need to sync their local refs (in .git/refs/head) with the corresponding remote refs (in .git/refs/remotes). The painful details of the actual transfer are outlined in another ProGit chapter.

So far, we've covered how Git keeps revision history and transports it between two machines. The rest is infrastructure + "porcelain" commands for end-users. The index, log, merge, blame, bisect, rebase, and filter-branch are just among the more advertised features. Here's an outline of what's possible within the current infrastructure:

Rewrite one commit somewhere in the history and automatically re-create all the dependent commits, asking for user intervention only in the case of a conflict. This is what git rebase essentially does.
Replace one commit with another in-place without rewriting any other commits. See git replace.
Chop the revision history somewhere in the middle and make it look like all the commits before that point never existed. This operation is called a graft.
Given a file located in a certain directory in the current revision, it's possible to move the file to another directory and rewrite the history to make it look like it was always present there. See the index-filter in git filter-branch.
Given the revision histories of two completely unrelated projects, it's possible to rewrite the history of one of them to make it look like the other project was developed in a subdirectory. See subtree merge.
Given a line range in a file in any revision, Git can tell how it got there (taking into account how the lines might might have been moved around by other commits), and where it went in the current revision. This isn't possible yet, but git log -L is a step in this direction.

User-Agent: Mutt/1.5.20

Mon, 28 Jun 2010 23:03:59 +0530

After nearly five years of first using it, I've gone back to using good old mutt. I've tried everything and come to a single conclusion: mutt is the most no-nonsense featureless (and suckless) email client. Even with so many new clients, nothing beats it in 2010. With TokyoCabinet caching, IMAP is blazing fast and GMail just works out-of-the-box. With gdata API + caching, anybody can whip up a quick Python/ Ruby script to fetch the Google Contacts. It took me four years because I've only started feeling the pain now- I've been handling large volumes of email over the last few months, especially patches and patch reviews. Now that I'm an active contributor to git.git, email is the one thing that I cannot live without. To put it tersely, mutt is a compromise- it does certain things really well and is completely incapable of handling the rest: it's completely featureless. I'd certainly not recommend it to anyone who writes less than 20~30 large emails everyday. Here's what I realized: at this volume, I don't need fancy features like cute colored labels, filters, high speed full-text search, and easy-to-grasp UI as much as the other basic features. Yes, the closest competitor is the GMail web interface.

What the web interface gets right: keyboard shortcuts, filters, labels, threading and humanized date display are awesome. It's awesome when email newbies are simply top-posting and adding to an ongoing conversation casually. On the Linux kernel and Git mailing lists, people are actually quoting individual chunks, and respecting the In-Reply-To header. GMail simply collapses everything into a linear thread the way it "feels like" (depending on In-Reply-To, subject etc); it also randomly collapses quote blocks with no apparent consistency. Not to mention how it damanges whitespace, making it unusable for sending or even downloading patches from. Another major problem is the lack of configurability: any hacker will automatically find out his/ her point of inefficiency during work and invest time to write a program to (partially) automate the repetitive task.

Other desktop email clients are worse. GMail has made sure that nothing comes close by exposing labels as separate folders over IMAP: hence, if your email client had to pull labels from GMail, it must first discover all the IMAP directories and download mail headers from them; then it must run a filter to find mails matching the Message-Id in "All Mail" and label them appropriately. Huge PITA. Archiving actually means copying the email from INBOX to "All Mail", and deleting the copy in INBOX. Filters are just inaccessible. Also, no (other) email client seems to implement IMAP in a sane manner: opening up "All Mail" is like asking for trouble (I never delete a single email, and often search through my archives). They also seem to have forgotten to implement the killer feature in GMail: search. It seems to be the problem with a lot of end-user GUI software in general: the engineers focus so much on which button to put where that they completely forget to implement basic features like sending emails without damaging whitespace.

Sup came close with its full-text search, but it's not without its share of problems- users were relying on a hopeless piece of software called offlineimap to mirror their complete GMail accounts locally. Sup had some "cute" features, but it takes years to engineer and build something as solid as mutt. I don't trust infrastructure in the Ruby world.

So what does mutt do? It does IMAP and threading exceptionally well, I can use an editor of my choice, and it's supremely configurable. It's what an email client should ideally be. Yes, I'm annoyed that it's featureless, but that's a trade-off that I have to make. I configured it heavily to behave like a combination of GMail and Emacs, and I'm quite happy with it. It's not without its share of annoyances however:

You can't have conditionals in the muttrc. That means no humanized relative dates, no choosing an editor depending on the situation. The workaround is to use a script to generate the muttrc everytime you start mutt.
The full-text search is obviously slow if you don't have the message bodies in the cache.
The default keybindings are not suitable for GMail users who often need to flip between IMAP folders. A LOT of painful configuration is required: even with macros, things like send-and-archive-conversation can be pretty challenging to get right. Even things like the default date display is useless: it just says that all the 100 emails in my INBOX were received today; how useful is that? In short, it needs to be customized heavily before first use.
No out-of-the-box support for muting conversations. You'll need some sort of folder hook.
No reading and composing at the same time. When I hit compose, mutt fires off Emacs to edit, and just hangs there waiting for Emacs to finish. For sup, writing the feature took just a few hours, but it'll probably be much harder to get right in C. The workaround I've devised is to compose emails at my leisure and then just paste it into the buffer when mutt pops it up.
Last but not least, mutt can't fill in missing parts of the conversation from "All Mail" to produce GMail-like threading for messages in my INBOX. This is probably very hard to get right- I don't know of any email client that does it.

I'm sick of hearing myself say it- the world really needs an email client that wasn't created in 1995. Maybe I'll do something about it one weekend, but I'm too busy to get into this at the moment. Until then, mutt it is; web interface for full-text search.

Rejourn

Sun, 23 May 2010 19:17:41 +0530

In short, I got sick of Wordpress and stopped writing altogether. I evaluated Hyde and HiSlain briefly before I decided that I needed to write my own static HTML generator engine that behaves exactly the way I want it to. Hyde's bloated and has too many features I don't care about, and HiSlain's looks like it's written in C#. So I wrote rejourn in half a week. It has two main differentiators:

Post headers- every post begins with a small header section which contains properties like title, author, pubdate etc. So if I want to remove certain a post, I can just open it up, set draft to true and rebuild.
Build system uses GNU Make- it's not my favorite build system, but it does the job. Using this build system has several advantages: for example, the whole website can be rebuilt by invoking make from ViM/ Emacs without extra configuration. Ofcourse, when I write a new post, only that post and the index are rebuilt.

I also bought static HTML hosting from NFSN for pennies- thanks to their SSH support, my workflow's reduced to writing, saving, make'ing, and pushing via Magit. On the remote server, I have a bare repository with a detached work tree that runs "git checkout -f" as a post-receive hook.

Dumping most of my older posts, I imported a few good posts from my Wordpress blog, but I haven't figured out how to import the corresponding comments yet- I probably won't, and just leave it at that. Disqus is quite bloated, but there don't seem to be any alternatives :\

Unfortunately, the design's ripped off from Steve Losh's site (sorry Steve!), but I'll hopefully make some changes in future and find a way to credit him for the nice minimalistic design.

As far as future plans for rejourn are concerned, I still have to figure out a system for attaching files and displaying images inline. I'm also planning to clean up the codebase and remove references to local directories so everyone can use it- it's currently quite a hacky mess. More importantly, this should help me stop procrastinating writing.

What drives Open Source?

Tue, 05 Jan 2010 00:00:00 +0530

Everyone understands how Microsoft makes money, but very few understand why they're getting a wonderful operating system like Ubuntu free of cost. I've tried to explain it here.

The problem with opensource software is simple. The "vegetable vendor" model that most closed-source software uses fails with opensource. Take the most popular and simplest example: RedHat. It essentially creates two operating systems from the same codebase: RHEL and Fedora. They sell support for RHEL to corporates, while giving away Fedora to the community, and sponsoring its development. Make no mistake: RHEL is opensource as well, but it has no community like Fedora does. RedHat decides what to go into RHEL, and they build it themselves. Anyone else is free to build it for themselves ofcourse, but they don't support those builds [1].

Why bother with Fedora at all? Simple. The first thing a RedHat service engineer does when he gets a complaint from a customer: attempt to replicate the bug on Fedora. If the bug's fixed in Fedora, fixing it in RHEL is as simple as backporting the changes. If it isn't, they'll file a bug in Fedora. Depending on the situation, either they'll get the bug fixed free-of-cost by the community, or will have to fix it themselves. In other words, they're just getting code from the community free of cost by maintaining Fedora. Then why not give Fedora to the customers? Because bleeding-edge development takes place on it, and stuff often tends to break and get fixed at a later date [2]: RedHat cannot control what gets committed to Fedora, but they will make sure that only mature features get into RHEL. This service model not only applies to complete distributions, but to individual software packages as well- PostgreSQL, MySQL, Django are just a few examples.

So finally, who writes Fedora? The Fedora Board consists of nine members: five of whom are RedHat employees and four of whom are community members [3]. A lot of mainstream opensource software has a similar distribution of paid/ community developers.

Another famous model is the dual-licensing model followed by MySQL AB (now acquired by Sun). They give away MySQL community edition for free, but their enterprise edition is proprietary [4]. The advantage of having these two editions is similar to the one RedHat gains from having RHEL and Fedora. A similar approach is followed by OpenOffice.org: Their proprietary product that earns them revenue is called StarOffice. Advertising is yet another way: Google pays Mozilla for example.

So, the final question remains: apart from the corporates, who is this "community" that writes the code that end-users get free of cost? Most free software projects started out as "pet projects" with little monetary incentive. The Linux kernel was written by a student as a personal project, and several parts of the GNU operating system was written by Stallman in an attempt to create a free operating system- both were individuals who weren't concentrating on revenue at that time. Things have changed a LOT since then. While there are still many tiny pet projects that are developed by a few people for little/ no monetary incentive, majority of the contribution to mainstream software such as Linux is from people with monetary incentive. The corporates these people work for benefit either directly or indirectly from that opensource project. Corporates that provide support for these opensource projects, that build their products on top of these opensource products, and those that simply use these products in their everyday work. Simple example: Google finds a bug in Chrome, and finds out that the underlying reason is WebKit. After fixing WebKit, getting that fix merged into upstream will benefit them, because they're not the only people developing it. Same reason Google will contribute to SSH: They're using it, and they're not the only ones developing it.

Even non-monetary incentive doesn't mean no incentive. Contributing to opensource is a great way to secure one's future. All the work is out there in the public for everyone to see, and you don't have to explain the work you did behind the closed doors of a large corporation in flowery language. Code talks. Everything else is CV-noise.

The whole model is sustainable in my opinion. A large corporate uses several bits and pieces of opensource software to create a product. They create an opensource project to drive it, and appoint some developers to head the project. I've already discussed how they'll make money to drive it. Students and other prospective employees start contributing to the project for no monetary incentive, with the intention of rising higher in the project's hierarchy. They're essentially making themselves employable by the company- once the company recognizes some brilliant developers, it starts paying them and giving them specific tasks. An additional bonus kicks in when other companies also become interested in the project and start paying more developers to develop it. There's always some ratio of developers working for monetary benefit to those working their way up.

Modern Thoughts on Open Source

Fri, 01 Jan 2010 00:00:00 +0530

Few people understand open source. Even fewer use this knowledge to their advantage. On one hand, there are free software fundamentalists with their utopian views, and on the other hand we have closed corporates who don't possibly see how open source could work. Both miss the point. I've attempted to explain it using four questions to guide the article. Admittedly, a lot of what I've said here was explained to me by Kiran Jonnalagadda (aka jace, jackerhack).

Why do RedHat, IBM, and Novell contribute so much to the Linux kernel? Is it because they care about the free software philosophy and are being charitable? To quote the Linux Foundation August 2009 issue, "Over 70% of all kernel development is demonstrably done by developers who are being paid for their work" and "None of these companies are supporting Linux development as an act of charity; in each case, these companies find that improving the kernel helps them to be more competitive in their markets"
Why does Google pump money into a competing against Firefox with an open source alternative, Chrome? To quote an excerpt from their December 2009 blog post, "Today's open source goes far beyond the "patent pooling" of the early auto manufacturers, and has led to the development of the sophisticated software components- Linux, Apache, SSH, and others- upon which Google is built"
Why are several of Microsoft's products losing market share? Why does Microsoft feel threatened by Linux? To quote Ballmer, "Linux is a cancer that attaches itself in an intellectual property sense to everything it touches". Is it because the general public is convinced by the GNU philosophy to "share with thy neighbor", or is it simply because of competition?
And finally, a slightly off-topic question to reinforce my point. Why did Apple finally make the move to x86 even after their innumerable claims that PPC is better? Is it because PPC is inherently inferior to Intel's x86, and nothing could have been done to save it?

All this points to one thing: Open source is an infrastructure. Just like the roads in the city that automobile manufacturers are interested in, software companies want access to existing infrastructure to build their own products upon and make money, instead of having to re-invent the wheel everytime. It's a win-win situation for all companies. Paying Microsoft to get Windows to support their products can be an expensive affair. So making an operating system a commodity is in everyone's interests (well, except Microsoft's). The web browser is another example of a commodity today. Everyone's interested in a good, conformant, open platform on which to develop their web applications on.

There are commodities in other sectors too. Apple moved to x86 because it was the standard commodity to use. Too many companies pumped money into it resulting in its development, and mass manufacturing brought down costs. Apple understood this- one company pumping money into PPC cannot match the rest of the world pumping money into x86.

The day a product comes out, every other company tries to clone it and make the product a commodity. In the small time window between the product being niche and the product becoming a commodity, there is money to be made. Take Netscape's example: Once they felt the pressure of the competing web browsers, they realized that keeping their browser closed and exclusively pumping money into it was pointless. So they opened it out, and started the Mozilla project. In other words, open sourcing a piece of software typically means that that piece of software can no longer make money as a traditional product anymore. Same reason Facebook open-sourced many of its components: It has everything to gain, nothing to lose. It's already too big on the social networking front to worry about competition there.

Largely, there are two ways to make money off this infrastructure: sell the cars that use these roads most sensibly, or follow a "pure service" model. The problem with the pure service model is that only RedHat and a few other companies have been successful at it. The argument "I created it, so I can service it best" doesn't hold anymore- too many people have technical expertise in open source software. So unless your service is the cheapest and the best, it'll be killed off by other service providers.

Products don't have to start off being closed and then open up when they stop making money. Companies are also interested in starting off free software projects. Google Chrome for example. They wanted to make this web browser a commodity, a commodity created by them. They'll have a foot at the door when Chrome gains larger market share.

Demystifying Unladen Swallow

Mon, 12 Oct 2009 00:00:00 +0530

Unladen Swallow is "an optimization branch of CPython, intended to be fully compatible and significantly faster". Interesting. In order to understand Unladen Swallow, we first needed to understand the CPython compiler. It's written in C and fairly easy to understand, very unlike the self-hosting complex beasts Steel Bank Common Lisp and Glasgow Haskell Compiler. The design of the compiler is explained fully in PEP 339, and I'm just including a small warm-up discussion.

Start off at the top level: the evaluator (Python/cevel.c). It executes Python bytecode. From the Python program you write to the Python bytecode that is produced, there are several stages with optimizations at every stage. It goes like this:

Using the SPARK parser generator, generate the Parse Tree from the Python code (Parser/pgen.c)
Convert the Parse Tree into an Abstract Syntax Tree (Python/ast.c)
Transform the AST into a Control Flow Graph. Several compiler optimizations like removing unreachable code are implemented here (Python/compile.c)
Post-order DFS the CFG and flatten it to emit the final Python bytecode. The bytecode is then subjected to various peephole optimizations (Python/compile.c)
Instead of having to go through this this long procedure everytime we want to run the same program, the Python bytecode produced is stored in a .pyc file.

Enough warming up. The Unladen Swallow developers want to optimize CPython very smoothly, preferably without breaking any existing code. Their Project Plan is laid out on the Wiki page; I'll mainly discuss 2009Q2.

If they work at any stage higher than the Python bytecode, they'll have to re-implement the fantastic peephole optimizations written by the community over the years. The plan is simple: instead of executing the high-level bytecode in the eval loop (which can be painfully slow), compile atleast some part of it into a lower representation that can be executed faster. Why not compile all of it to x86 assembly? Firstly, we'd lose the interactive shell, a very major part of Python. Secondly, we'd lose out on Python's high portability, that arises because it's not tied down to any single architecture's assembly language (I even run Python on my phone using the PyS60 package). Thirdly, it's terribly difficult to do, makes Python uglier and unmaintainable, and it's utterly pointless- we're trading off too much for execution speed.

Just-in-time (JIT) compilation is the solution. Compile the code only when needed. Then again, don't compile everything; compile when the new execution time is less than the old execution time by an extent enough to justify the compile time. Otherwise, the startup delay will become unbearable and the interactive interpreter won't really be interactive anymore. The perfect low-level platform independent representation to JIT to is LLVM (Low-level virtual machine). So, the Unladen Swallow developers first wrote a compiler from Python bytecode to LLVM IR from scratch (Python/llvm_compile.cc). The LLVM IR (Intermediate Representation) looks a lot like assembly, except that it's architecture-independent, and can be subject to a nice set of optimizations. Then they modified the eval loop to execute LLVM IR, while keeping the existing Python bytecode evaluator (file renamed: Python/eval.cc).

Q3 and Q4 aren't updated on the Wiki page, but it should be easy enough to find out the work in progress from the IRC channel, mailing list, and repository commits.

Introductory CS course

Wed, 30 Sep 2009 00:00:00 +0530

This course attempts to teach programming methodology from an engineering viewpoint, as well give an insight into theoretical computer science.

Guidelines:

Teach programming, and not the programming language. Discuss abstract ideas in class, and ask students to write implementations in a language of their choice. Avoid building language-specific patterns (for example, the double-for loop for generating prime numbers in C got stuck in my head). Discuss various different implementations AFTER the students have tried it themselves.
Discussions should span several programming paradigms. Initially, cast problem statements to strongly indicate a paradigm. For example "Go over numbers from 1 to 10 and print those that are even" versus "Define an even number and print even numbers <= 10". Later, let students figure it out themselves.
Develop a hacker mindset- give students incomplete solutions written by other students and ask them to complete it.
For routine assignments, prepare a huge problem set of ascending difficulty to randomly pick from, along the lines of Project Euler. Problems should cover algorithms, pure math, computational geometry, and dynamic programming.
Conduct collaborative programming sessions where students learn to implement from each other. No single instructor will be able to provide such a vast variety of implementations.
Prepare a reading list for software engineering, including articles on editing (introduction to specialized editors such as Vim and Emacs), debugging, versioning, testing, and profiling. It's important to make students write large programs collaboratively in steps, so that they will be able to appreciate the importance of these tools.
Prepare another reading list for theoretical computer science. Also give students sneak peeks into the latest work in every field. The purpose of this exercise is to get students excited, so that they can work independently to eventually discover their research interests.
Finally, it all boils down to grading. If you're going to ask students to evaluate things like (i++) + (++i) in the exam, it's clear that you're expecting students to know programming language intricacies, and not programming methodology.

Here's a link to the PDF of my first lecture.