Software development is traditionally a field dominated by men. The presence of women in programming is commanding more attention at conferences and events, particularly in recent years. There are many initiatives within the Python community to promote involvement, such as the group Women Who Code, and the long-term benefits of efforts like these cannot be overstated. At Caktus, three of the ten current members of our Django development staff are female, and we are actively working to further equalize the gender balance through future hires.

One part of closing the gender gap in programming communities involves creating a safe and inclusive environment for all parties, both in the workplace and at community events. This begins with strong employment policies on harassment — which we have implemented here at Caktus — and also extends to implementing policies at conferences or other events relevant to the development community. This blog post comes out of several recent events in the broader tech community that Caktus is a part of. This includes two blog posts that drew a lot of attention, and two corporate events that demonstrated clearly for us how pervasive sexism still is in our industry.

In March, Katie Cunningham, a fellow Django developer and friend made through DjangoCon, wrote an excellent and widely read blog post titled "Lighten Up." The post details a story that is all too common in our industry, that of ongoing, but usually subtle sexist jokes and comments that serve only, whether intentionally or not, to gradually demoralize and discourage those to whom they are addressed. She discusses how the typical response, should someone get upset by such a "joke" or comment, is simply to "lighten up" and "stop being so sensitive." I can see how that would get really old. Fast. In fact, just in reading up for writing this post, I was blown away by the number of comments in response to posts such as Katie's that fit exactly that description.

Just a couple weeks ago, the computer manufacturer Dell hosted a summit in Copenhagen. Dell hired a "comedian" to MC the event, who proceeded to entertain the attendees with a variety of sexist jokes, acclaiming the "success" of the IT community as indicated by its male-dominated culture, and included a suggestion to the mostly male attendees that they go home and tell their partners to "shut up bitch." Dell has finally put out an apology about what happened, but the consensus seems to be that it's weak at best and, given its tardiness, lost any sincerity it may once have had.

In closing, I’d like to draw attention to one other aspect of the software development community, particularly at conferences, that often tends to aggravate any pre-existing sexism that might exist in the community. Last month, Ryan Funduk wrote an extensive and much-debated blog post, titled "Our Culture of Exclusion," about the pervasiveness of drinking at many programming conferences. He also discusses how, all too often, these environments are ripe with sexist or racist jokes (or worse) that quickly erode any efforts to further diversify such communities. One example that he highlights is the daily deal API provider Sqoot, who alienated and offended many when they suggested that women are better fit to serve beer than to program. Sqoot made several apologies about this unfortunate advertisement, but many of them still fell short and only worsened the offense, as this response from Gayle McDowell aptly sums up:

Your language, as far as I understand it, is making the assumption that all coders are male, that all are straight, that women are sexual objects offered as a reward (HIGHLY inappropriate in a professional context), and that the women are there to serve them.

She goes on to suggest an appropriate response if and when events like this do happen:

People screw up sometimes. It happens. A TON of people (both men and women) have done equally offensive things. // But when you do it, you need to own up to it and think about what you've done. Not lie about it, as you're doing now.

Clearly, we need to have policies in place for handling instances of sexist or racist jokes and sexual harassment. When they do happen, McDowell says, one needs to take responsibility and own up to the mistakes one has made in an honest way.

The bar has been set, and I think it’s worth noting that several lessons can be taken from these events and applied to our own communities. Caktus is a long-time sponsor of several conferences relevant to the work that we do. Along with this blog post, Caktus is asking conference organizers and other sponsors to join us in the following effort: Moving forward, Caktus will require that a zero-tolerance sexual harassment policy is established and enforced by the organizers of any conference that we sponsor or attend. We want to ensure that our community events are safe, welcoming, and supportive for all of our colleagues — both male and female. While drinking can certainly make things worse, it is not the root of the problem. To overcome these issues, we need to talk about them more, continue to raise awareness, and treat all people in our workplaces and communities with the professionalism and respect that they deserve.

I’m pretty excited! Now if only the Python Imaging Library would convert too…

http://wxpython.org/Phoenix/ItsAlive/

PyFrameObject *
PyFrame_New(PyThreadState *tstate, PyCodeObject *code, PyObject *globals,
PyObject *locals)
...
if (code->co_zombieframe != NULL) {
f = code->co_zombieframe;
code->co_zombieframe = NULL;
_Py_NewReference((PyObject *)f);
assert(f->f_code == code);
}

Intrigued by the name, I examined the header where it is defined, code.h:

...
void *co_zombieframe; /* for optimization only (see frameobject.c) */
...
} PyCodeObject;

It turns out that for every PyCodeObject object that has been executed, a PyFrameObject of a suitable size is cached and kept with the code object. Now, caching is fine and good, but this cache is unbounded. Every code object has the potential to hang on to a frame, which may then never be released.
Further, there is a separate freelist cache for PyFrameObjects already, in case a frame is not found on the code object:

if (free_list == NULL) {
f = PyObject_GC_NewVar(PyFrameObject, &PyFrame_Type,
extras);
if (f == NULL) {
Py_DECREF(builtins);
return NULL;
}
}
else {
assert(numfree > 0);
--numfree;
f = free_list;
free_list = free_list->f_back;
...

Always concious about memory these days, I tried disabling this in version 3.3 and running the pybench test. I was not able to see any conclusive difference in execution speed.

Update:

Disabling the zombieframe on the PS3 shaved off some 50k on startup.  Not the jackpot, but still, small things add up.

——————————————————————————-
PYBENCH 2.1
——————————————————————————-
* using CPython 3.3.0a3+ (default, May 23 2012, 20:02:34) [MSC v.1600 64 bit (AMD64)]
* disabled garbage collection
* system check interval set to maximum: 2147483647
* using timer: time.perf_counter
* timer: resolution=2.9680909446810176e-07, implementation=QueryPerformanceCounter()

——————————————————————————-
Benchmark: nozombie
——————————————————————————-

Rounds: 10
Warp: 10
Timer: time.perf_counter

Machine Details:
Platform ID: Windows-7-6.1.7601-SP1
Processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel

Python:
Implementation: CPython
Executable: D:pydevhgcpython2pcbuildamd64python.exe
Version: 3.3.0a3+
Compiler: MSC v.1600 64 bit (AMD64)
Bits: 64bit
Build: May 23 2012 20:02:34 (#default)
Unicode: UCS4

——————————————————————————-
Comparing with: zombie
——————————————————————————-

Rounds: 10
Warp: 10
Timer: time.perf_counter

Machine Details:
Platform ID: Windows-7-6.1.7601-SP1
Processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel

Python:
Implementation: CPython
Executable: D:pydevhgcpython2pcbuildamd64python.exe
Version: 3.3.0a3+
Compiler: MSC v.1600 64 bit (AMD64)
Bits: 64bit
Build: May 23 2012 20:00:42 (#default)
Unicode: UCS4

Test minimum run-time average run-time
this other diff this other diff
——————————————————————————-
BuiltinFunctionCalls: 51ms 52ms -3.3% 52ms 53ms -2.0%
BuiltinMethodLookup: 33ms 33ms +0.0% 34ms 34ms +0.8%
CompareFloats: 50ms 50ms +0.1% 50ms 50ms +0.4%
CompareFloatsIntegers: 99ms 98ms +0.8% 99ms 99ms +0.6%
CompareIntegers: 77ms 77ms -0.5% 77ms 77ms -0.3%
CompareInternedStrings: 60ms 60ms +0.0% 61ms 61ms -0.1%
CompareLongs: 46ms 45ms +1.5% 46ms 45ms +1.2%
CompareStrings: 61ms 59ms +3.6% 61ms 59ms +3.6%
ComplexPythonFunctionCalls: 60ms 58ms +3.3% 60ms 58ms +3.2%
ConcatStrings: 48ms 47ms +2.4% 48ms 47ms +2.1%
CreateInstances: 58ms 57ms +1.3% 59ms 58ms +1.3%
CreateNewInstances: 43ms 43ms +1.1% 44ms 44ms +1.1%
CreateStringsWithConcat: 79ms 79ms -0.3% 79ms 79ms -0.1%
DictCreation: 71ms 71ms +0.4% 72ms 72ms +1.0%
DictWithFloatKeys: 72ms 70ms +2.1% 72ms 71ms +1.8%
DictWithIntegerKeys: 46ms 46ms +0.7% 46ms 46ms +0.4%
DictWithStringKeys: 41ms 41ms +0.0% 41ms 41ms -0.1%
ForLoops: 35ms 37ms -4.0% 35ms 37ms -4.0%
IfThenElse: 64ms 64ms -0.1% 64ms 64ms -0.4%
ListSlicing: 49ms 50ms -1.0% 53ms 53ms -0.8%
NestedForLoops: 54ms 51ms +6.7% 55ms 51ms +6.7%
NestedListComprehensions: 54ms 54ms -0.7% 54ms 55ms -2.2%
NormalClassAttribute: 94ms 94ms +0.1% 94ms 94ms +0.1%
NormalInstanceAttribute: 54ms 54ms +0.3% 54ms 54ms +0.2%
PythonFunctionCalls: 58ms 57ms +0.8% 58ms 58ms +0.6%
PythonMethodCalls: 65ms 61ms +6.3% 66ms 62ms +5.9%
Recursion: 84ms 85ms -1.0% 85ms 85ms -0.9%
SecondImport: 74ms 76ms -2.5% 74ms 77ms -3.5%
SecondPackageImport: 75ms 78ms -3.8% 76ms 79ms -3.9%
SecondSubmoduleImport: 163ms 169ms -3.4% 164ms 170ms -3.3%
SimpleComplexArithmetic: 43ms 43ms +1.0% 43ms 43ms +1.0%
SimpleDictManipulation: 80ms 78ms +2.2% 81ms 79ms +2.4%
SimpleFloatArithmetic: 42ms 42ms +0.1% 42ms 42ms -0.0%
SimpleIntFloatArithmetic: 52ms 53ms -1.2% 52ms 53ms -1.1%
SimpleIntegerArithmetic: 52ms 52ms -0.7% 52ms 53ms -0.8%
SimpleListComprehensions: 45ms 45ms -0.2% 45ms 45ms +0.3%
SimpleListManipulation: 44ms 46ms -4.0% 44ms 46ms -3.9%
SimpleLongArithmetic: 32ms 32ms -0.9% 32ms 32ms -0.1%
SmallLists: 58ms 57ms +1.2% 58ms 67ms -12.8%
SmallTuples: 64ms 65ms -0.5% 65ms 65ms -0.2%
SpecialClassAttribute: 148ms 149ms -0.8% 149ms 150ms -1.0%
SpecialInstanceAttribute: 54ms 54ms +0.2% 54ms 54ms +0.0%
StringMappings: 120ms 117ms +2.5% 120ms 117ms +2.5%
StringPredicates: 62ms 62ms +0.9% 62ms 62ms +1.0%
StringSlicing: 69ms 68ms +1.6% 69ms 68ms +2.1%
TryExcept: 37ms 37ms +0.0% 37ms 37ms +0.5%
TryFinally: 40ms 37ms +6.7% 40ms 37ms +6.5%
TryRaiseExcept: 19ms 20ms -1.0% 20ms 20ms -0.4%
TupleSlicing: 65ms 65ms +0.5% 66ms 65ms +1.2%
WithFinally: 57ms 56ms +1.9% 57ms 56ms +2.1%
WithRaiseExcept: 53ms 53ms +0.3% 54ms 54ms -0.8%
——————————————————————————-
Totals: 3154ms 3145ms +0.3% 3176ms 3177ms -0.0%

(this=nozombie, other=zombie)

I’m going to remove this weird, unbounded cache from the python interpreter we use on the PS3.

Do you ever get the urge to kill? How many of us cringe whenever we see these words? Lately I’ve been spending a lot of time developing pythonpackages.com, (now running on heroku!) during which time I see a lot of these kinds of packages being released:

I kid about the killing part, but seriously: this is a problem. Fortunately for us, our PyPI overloads see fit to occasionally remove these packages, and for this I am grateful. I love seeing this:

I mean it makes me dance-around-the-room happy! Ahem. But are they really all gone? Close enough. A quick crate.io search now shows only 2 packages instead of 4 pages of results:

Hallelujah! But is this the best we can do? I know that some well-meaning person wrote a book containing the example that is leading some poor, misguided souls to spam PyPI (if only the author listed the test site instead: http://testpypi.python.org/pypi). And I have to assume that this was just some terrible mistake. But do we all have to live with this mistake?

I’m asking because I honestly don’t know the answer. I remember when I started pythonpackages.com, the Deliverance documentation was being updated something like every 5 minutes (kidding again, but it was frequent enough to be annoying). After grousing about it in public, it stopped happening!

I wonder if some good natured grousing about our friends (read: enemies) the simple printers of nested lists will do the same?

Things to take away from the experience:

• Try to stay on topic! I actually found a second issue with the paragraph I was fixing in the devguide and that probably should have gone in a separate bug report.
• Number your patches! I don’t know why I didn’t think of that, but Eli told me I should do that in the future to make it less confusing for the committer. That was a face palm moment.

I’ve been reading some of the supposedly “easy bugs” and trying to figure out where else I can help. I already spotted another typo in the docs that are included with Python itself which I’ll probably try to fix. Of course, I want to actually contribute to the code, not just the documentation, but I am probably more likely to be able to find documentation bugs I can help with. Hopefully with more experience I’ll be able to contribute more effectively. Happy hacking my fellow Pythoneers!

Just because I have to use a callback-oriented style on the client doesn't mean I want to use a callback-oriented style on the server. Now, before anyone gets all upset and tells me that I don't know the difference between async and a kitchen sink, let me explain :)

The client is necessarily an event-oriented place. If I don't know which button the user is going to press, it makes a lot of sense to use a different callback for each button. The server is different. If I'm waiting for the result of a database query before I can continue processing a request, it sure is convenient to just block and wait.

My key point is that it's important to separate what style you want to code with and what performance and scalability characteristics you want. You shouldn't necessarily pick a callback-oriented style just because you want the performance and scalability characteristics of asynchronous networking APIs.

My favorite two examples are gevent and Erlang, but Go is similar. When you code using gevent or Erlang, your code looks like synchronous, blocking code. However, below the covers, they use asynchronous networking APIs. Now, before anyone tells me that it's impossible, buggy, or that it'll never work, let me point out that these tricks have been in production for decades at Ericsson, Yahoo Groups, and IronPort Cisco.

Furthermore, I should point out that asynchronous networking APIs aren't a perfect fit for every problem. For instance, if your goal is to send 10 gigabytes of information to another server, it turns out that synchronous networking APIs will actually outperform asynchronous networking APIs. The reason asynchronous networking APIs are so popular is because they can handle a larger number of clients than synchronous networking APIs can and because they use less memory than a large number of threads, which each have to have their own stack. gevent and Erlang can handle a large number of clients, don't use up much memory, and don't require a real OS-level stack per client.

So what's my problem with the callback-oriented style? I find it a lot harder to read. I've coded projects in Twisted, Node.js, etc., and I prefer the gevent approach. You get roughly the same performance and scalability characteristics, but with much easier to read code. Of course, what's readable to me may not be readable to other people. I've met people who are perfectly happy using Twisted Web 1 and don't think that callback-oriented code poses any real challenge.

If you're interested in hearing more about my thoughts on async and concurrency, check out my other blog posts, which include a link to my Dr. Dobb's Journal article on Python concurrency.

You can see the RFP here: http://pythonorg-redesign.readthedocs.org/en/latest/

It’s taken me several years of false starts, other attempts (including skunkworks attempts), political and social discussions, and the hard work of many to make this come to fruition. Now, we can only sit back and hope that we see some amazing proposals from the community and others.

I sincerely hope this will be successful, and that we will see a modern, well designed Python.org that showcases not only the language, but the vibrant, open, welcoming and active community we are all part of. 

The book I am reviewing has many examples which can easily be applied to real-world web applications, which I praise. As I have been reading through the book, I have also been playing around and exploring Web2Py in general. It has some very interesting design choices and features. It is definitely good for someone who can never remember what to import, as there is rarely any need to import anything in web2py, besides when you need to import a 3rd party Python module. At first, this design choice caught me off guard, as in Python, I am used to explicitly importing my modules for use. Web2Py includes it's technical reference with the application itself, in the form of Epydoc. I am not terribly fond of this documentation format, but it is how I learned Pyjamas, as it uses the same documentation system.

Web2Py has some interesting magic, which frankly does make web development a breeze. Generating forms and cruds is literally a one-liner, further customization can be made by using more lines of code. Web2Py also comes with a nice default template, and the application wizard, has many additional templates for use as well. This feature is definitely well welcomed in my book. Sure, it implies that tons of websites made with Web2Py will have the same look and feel, but is that such a bad thing? It just means that visitors will know where most resources can be found, and how to use the included form widgets. Think of Wordpress, most Wordpress sites I visit keep the included wordpress theme. Blogspot is another example of a set of websites which share a common theme. The included themes allow one to get to developing the website as quickly as possible without having to worry about how it will look and feel, but rather focus on the functionality. I am more of a functionality person, not a look-and-feel type of person.

All in all, I am very excited to be trying out Web2Py for a second time, and do hope that when I complete this book, I can both provide a review of the book itself, as well as the framework in general. Neither of these reviews will be compared to anything, and be strictly of the book and framework itself. I am thinking of making a future website using Web2Py to see where it goes. I will most likely keep my blog running on Django, as I have used many Django-specific features which would be otherwise troublesome to migrate over. These articles contain Django template code, for example.

This version requires pip installing the following into your virtualenv.

• pip install django-crispy-forms so we can do Python driven layouts.
• pip install django-floppyforms so we get HTML5 elements for free.
• pip install django-recaptcha to do the RECAPTCHA work.

Don't forget to add the app to your INSTALLED_APPS in settings.py:

INSTALLED_APPS += (
    'crispy_forms',
    'floppyforms',
    'captcha',
)

Generate your KEYs from the RECAPTCHA site and add them in settings.py:

RECAPTCHA_PUBLIC_KEY = '6LcVu9ESAAAAANVWwbM5-PLuLES94GQ2bIYmSNTG'
RECAPTCHA_PRIVATE_KEY = '6LcVu9ESAAAAAGxz7aEIACWRa3CVnXN3mFd-cajP'

In myapp.forms.py:

from captcha.fields import ReCaptchaField  # Only import different from yesterday
from crispy_forms.helper import FormHelper
from crispy_forms.layout import Submit
import floppyforms as forms

class ContactForm(forms.Form):

    name = forms.CharField(required=True)
    email = forms.EmailField(required=True)
    subject = forms.CharField(required=True)
    message = forms.CharField(widget=forms.Textarea)
    captcha = ReCaptchaField()  # Only field different from yesterday

    def __init__(self, *args, **kwargs):
        self.helper = FormHelper()
        self.helper.add_input(Submit('submit', 'Submit'))
        super(ContactForm, self).__init__(*args, **kwargs)

In myapp.views.py:

# Unchanged from yesterday. :-)
from django.conf import settings
from django.core.mail import send_mail
from django.views.generic import FormView

from myapp.forms import ContactForm

class ContactFormView(FormView):

    form_class = ContactForm
    template_name = "myapp/email_form.html"
    success_url = '/email-sent/'

    def form_valid(self, form):
        message = "{name} / {email} said: ".format(
            name=form.cleaned_data.get('name'),
            email=form.cleaned_data.get('email'))
        message += "\n\n{0}".format(form.cleaned_data.get('message'))
        send_mail(
            subject=form.cleaned_data.get('subject'),
            message=message,
            from_email='contact-form@myapp.com',
            recipient_list=[settings.LIST_OF_EMAIL_RECIPIENTS],
        )
        return super(ContactFormView, self).form_valid(form)

In templates/myapp/email_form.html:

{# Also unchanged from yesterday. :-)  #}
{% extends 'base.html' %}
{% load crispy_forms_tags %}

{% block title %}Send an email{% endblock %}

{% block content %}
    <div class="row">
        <div class="span6">
            <h1>Send an email</h1>
            {% crispy form form.helper %}
        </div>
    </div>
{% endblock %}

{% block extrajs %}
<script src="{{ STATIC_URL }}js/jquery-1.7.1.min.js"></script>
<script type="text/javascript">
$(function() {
    $('#id_name').focus()
});
</script>
{% endblock %}

Getting Started

First off, you need to get an account with the Bug Tracker for Python. If you hope to become a core developer, then you’ll need to make sure your username follows their guidelines, which are really simple:


firstname.lastname


Once you’ve got that, you can start looking for something to patch. There’s a link that says “Easy issues” that is a good starting place. You can also do a search for a component that you’re competent in using and see if there are any bugs in there that you think you can fix. Once you find something, you’ll need to make sure you update your local repo and then read the devguide’s patch page.

Creating the Patch

Assuming you have the necessary repository checked out on your local machine, all you need to do is go edit the appropriate file. In my case, I had to check out the devguide (which you can read about here) and edit the setup.rst file. If you’re editing Python code, then you’ll have to conform to PEP8. Once I finished editing the file, I saved my changes and then had to use Mercurial to create the patch. Here’s the command I used per the Python patch instructions.


hg diff > setup.patch


And here is the contents of that patch file:


diff -r b1c1d15271c0 setup.rst
--- a/setup.rst Tue May 22 00:33:42 2012 +0200
+++ b/setup.rst Tue May 22 13:55:09 2012 -0500
@@ -173,7 +173,7 @@
To build from the Visual Studio GUI, open pcbuild.sln to load the project
files and choose the Build Solution option from the Build menu, often
associated with the F7 key. Make sure you have chosen the "Debug" option from
-the build configuration drop-down first.
+the configuration toolbar drop-down first.

Once built you might want to set Python as a startup project. Pressing F5 in
Visual Studio, or choosing Start Debugging from the Debug menu, will launch

Now that we have a patch we need to submit it!

Submitting a Patch

Put your shields up, we’re going in! Submitting a patch is a little daunting. What will people think of you? I suspect if you plan to work on something major, then you better start growing some thick skin. In my case, I’m going to submit a really simple typo fix, so I’m hoping that sort of thing isn’t flame-worthy. Then again, this is my first patch, so I may submit it in a completely erroneous way. Since my patch will be for something (presumably) new, I did a quick search to make sure it hadn’t already been reported. Seeing nothing, I clicked the “Create New” link with some trepidation and choose the “devguide” as my component. I also chose the latest version of Python. I don’t see anything in the devguide that says it applies to just one set of Python versions, so I’m just going to leave it at that. I didn’t really see a “type” that fit a devguide edit, so I left that blank for my betters to fix. Finally, I attached my patch file to the bug ticket. You can see my bug ticket here if you like.

When contributing a patch to Python, you should fill out a contributor agreement form which allows the Python Software Foundation to license your code for use with Python while you get to keep the copyright. Yes, you too can become famous just for writing Python code! Assuming people read the source or those acknowledgement pages.

Wrapping Up

I don’t know what will happen to my rather lame contribution. Maybe it’ll get accepted, maybe not. But I think I’ll spend some time trying to figure out some other bugs and just see if there’s anything I can do to help out the Python community. Feel free to join me on this adventure!

OSX’s iTerm 2, and maybe some other terminal applications, support ANSI control sequence extensions which allow shell to set the color of the terminal tab.

Below is a Python script which

• Randomizes a color based on the server host name. The same hostname always results to the same color.
• The color is randomized in HSL color space, so that only the hue component varies and saturation and lightness are locked. This prevents the creation of ugly color combinations like black text on black tab background.

Note: The effect can be also applied on terminal windows  – for those who don’t use tabs.

The effective result is that

• You learn to identify terminal tabs by the color
• You can much more faster to switch between tabs, because you can visually pick up the terminal without needing to be able to read the text on it or remember its location in the list

Note: If your puny terminal does not support setting the color of window decorations, you can always set the terminal background color. This is useful e.g. if you want to red background for danger zone ™ when you are logged in as root on the production server 23:00 Friday night.

Note: Naturally you also need to have the script installed on the servers you are ssh’ing into

precmd() hook

You can run the script once and the tab color is set. However, if you SSH from the computer to another  and then exit back, the color of the latest server would remain in this case.

This can be avoided by

1. Calculating the OSC control code sequence needed to set the terminal tab color when the shell starts
2. Have a precmd() hook (zsh terminology, not sure what other shells use) to reset the tab color every time the shell prompt is displayd

We, me with my friend, are maintaining (yet another) zsh toolkit called ztanesh (github). There  you can find precmd() example codes in 1) 98-server-color and 2) 80-statusbar.

rainbow-parade.py

The script code lives on Github. Currently it supports iTerm 2 only and we plan to expand support to Konsole. Patches for other terminals are welcome.

(This probably could be done in pure shell code too, but Python is just so much more fun…)

#!/usr/bin/env python
"""

       Set terminal tab / decoration color by the server name.

       Get a random colour which matches the server name and use it for the tab colour:
       the benefit is that each server gets a distinct color which you do not need
       to configure beforehand.

"""

import socket
import random
import colorsys
import sys

# http://stackoverflow.com/questions/1523427/python-what-is-the-common-header-format
__copyright__ = "Copyright 2012 Mikko Ohtamaa - http://opensourcehacker.com"
__author__ = "Mikko Ohtamaa <mikko@opensourcehacker.com>"
__licence__ = "WTFPL"
__credits__ = ["Antti Haapala"]

USAGE = """
Colorize terminal tab based on the current host name.

Usage: rainbow-parade.py [0-1.0] [0-1.0] # Lightness and saturation values

An iTerm 2 example (recolorize dark grey background and black text):

    rainbow-parade.py 0.7 0.4
"""

def get_random_by_string(s):
    """
    Get always the same 0...1 random number based on an arbitrary string
    """

    # Initialize random gen by server name hash
    random.seed(s)
    return random.random()

def decorate_terminal(color):
    """
    Set terminal tab / decoration color.

    Please note that iTerm 2 / Konsole have different control codes over this.
    Note sure what other terminals support this behavior.

    :param color: tuple of (r, g, b)
    """

    r, g, b = color

    # iTerm 2
    # http://www.iterm2.com/#/section/documentation/escape_codes"
    sys.stdout.write("\033]6;1;bg;red;brightness;%d\a" % int(r * 255))
    sys.stdout.write("\033]6;1;bg;green;brightness;%d\a" % int(g * 255))
    sys.stdout.write("\033]6;1;bg;blue;brightness;%d\a" % int(b * 255))
    sys.stdout.flush()

    # Konsole
    # TODO
    # http://meta.ath0.com/2006/05/24/unix-shell-games-with-kde/

def rainbow_unicorn(lightness, saturation):
    """
    Colorize terminal tab by your server name.

    Create a color in HSL space where lightness and saturation is locked, tune only hue by the server.

    http://games.adultswim.com/robot-unicorn-attack-twitchy-online-game.html
    """

    name = socket.gethostname()

    hue = get_random_by_string(name)

    color = colorsys.hls_to_rgb(hue, lightness, saturation)

    decorate_terminal(color)

def main():
    """
    From Toholampi with love http://www.toholampi.fi/tiedostot/119_yleisesite_englanti_naytto.pdf
    """
    if(len(sys.argv) < 3):
        sys.exit(USAGE)

    lightness = float(sys.argv[1])
    saturation = float(sys.argv[2])

    rainbow_unicorn(lightness, saturation)

if __name__ == "__main__":
    main()

 Subscribe to this blog in a reader Follow me on Twitter

1. You can get genuinely high quality code hosting for free. Sure Sourceforge was already around back then, but Git and Mercurial stomp all over CVS from a collaboration point of view. These also come with decent issue trackers and various other collaboration tools. If you don't want to trust a service provider with your code, than tools like GitLab let you set up similar environments internally.
2. Web based issue trackers are everywhere, with the ubiquitous "issue URL" allowing effective cross-linking between tracker issues, documentation, code comments, source control browsers, code review systems, etc.
3. Dedicated code review tools like Gerrit and Reitveld are published as open source (and, in the case of the latter, even available as a free service on Google App Engine).
4. Services like ReadTheDocs exist, allowing you to easily build and publish high quality documentation. All with nice URLs so you can link it from emails, tracker issues, source code, etc.
5. Organisations like Shining Panda CI and Travis CI provide hosted continuous integration services that put the internal capabilities of many large companies to shame.
6. Language communities provide cross-platform distribution services to reach a global audience.
7. Depending on the language you use, you may even have tools like SonarSource available
8. Once you go into production in the web application world, service components like Sentry, Piwik, and Graphite are again available for no charge.

• they're hosted on BitBucket as Mercurial projects (I happen to prefer Mercurial, although I can definitely see why people like Git, too). That gives me integrated issue tracking and online source code browsing, too. (OK, so I could have had essentially that back in the early SourceForge days, but the UI aspects have improved in many respects in the intervening years)
• I can publish my projects on the Python Package Index with a simple "setup.py sdist upload". They're then available for anyone in the world to install with a straightforward command like "pip install walkdir"
• thanks to Shining Panda CI, I know the downloads from PyPI work, and I also know that the projects work on all the versions and implementations of Python I want to support
• thanks to ReadTheDocs and Sphinx, you can read nicely formatted documentation like this rather than trying to decipher plain text files or wiki pages.

Enough of my jibber-jabber. Here is my Haskell solution first (trying to change things up here):

module Main where
 
main :: IO()
main = do
    print . take 10 . show $ sum big_number
    where big_number = [ 37107287533902102798797998220837590246510135740250
                       , 46376937677490009712648124896970078050417018260538
                       , 74324986199524741059474233309513058123726617309629
                       , 91942213363574161572522430563301811072406154908250
                       , 23067588207539346171171980310421047513778063246676
                       , 89261670696623633820136378418383684178734361726757
                       , 28112879812849979408065481931592621691275889832738
                       , 44274228917432520321923589422876796487670272189318
                       , 47451445736001306439091167216856844588711603153276
                       , 70386486105843025439939619828917593665686757934951
                       , 62176457141856560629502157223196586755079324193331
                       , 64906352462741904929101432445813822663347944758178
                       , 92575867718337217661963751590579239728245598838407
                       , 58203565325359399008402633568948830189458628227828
                       , 80181199384826282014278194139940567587151170094390
                       , 35398664372827112653829987240784473053190104293586
                       , 86515506006295864861532075273371959191420517255829
                       , 71693888707715466499115593487603532921714970056938
                       , 54370070576826684624621495650076471787294438377604
                       , 53282654108756828443191190634694037855217779295145
                       , 36123272525000296071075082563815656710885258350721
                       , 45876576172410976447339110607218265236877223636045
                       , 17423706905851860660448207621209813287860733969412
                       , 81142660418086830619328460811191061556940512689692
                       , 51934325451728388641918047049293215058642563049483
                       , 62467221648435076201727918039944693004732956340691
                       , 15732444386908125794514089057706229429197107928209
                       , 55037687525678773091862540744969844508330393682126
                       , 18336384825330154686196124348767681297534375946515
                       , 80386287592878490201521685554828717201219257766954
                       , 78182833757993103614740356856449095527097864797581
                       , 16726320100436897842553539920931837441497806860984
                       , 48403098129077791799088218795327364475675590848030
                       , 87086987551392711854517078544161852424320693150332
                       , 59959406895756536782107074926966537676326235447210
                       , 69793950679652694742597709739166693763042633987085
                       , 41052684708299085211399427365734116182760315001271
                       , 65378607361501080857009149939512557028198746004375
                       , 35829035317434717326932123578154982629742552737307
                       , 94953759765105305946966067683156574377167401875275
                       , 88902802571733229619176668713819931811048770190271
                       , 25267680276078003013678680992525463401061632866526
                       , 36270218540497705585629946580636237993140746255962
                       , 24074486908231174977792365466257246923322810917141
                       , 91430288197103288597806669760892938638285025333403
                       , 34413065578016127815921815005561868836468420090470
                       , 23053081172816430487623791969842487255036638784583
                       , 11487696932154902810424020138335124462181441773470
                       , 63783299490636259666498587618221225225512486764533
                       , 67720186971698544312419572409913959008952310058822
                       , 95548255300263520781532296796249481641953868218774
                       , 76085327132285723110424803456124867697064507995236
                       , 37774242535411291684276865538926205024910326572967
                       , 23701913275725675285653248258265463092207058596522
                       , 29798860272258331913126375147341994889534765745501
                       , 18495701454879288984856827726077713721403798879715
                       , 38298203783031473527721580348144513491373226651381
                       , 34829543829199918180278916522431027392251122869539
                       , 40957953066405232632538044100059654939159879593635
                       , 29746152185502371307642255121183693803580388584903
                       , 41698116222072977186158236678424689157993532961922
                       , 62467957194401269043877107275048102390895523597457
                       , 23189706772547915061505504953922979530901129967519
                       , 86188088225875314529584099251203829009407770775672
                       , 11306739708304724483816533873502340845647058077308
                       , 82959174767140363198008187129011875491310547126581
                       , 97623331044818386269515456334926366572897563400500
                       , 42846280183517070527831839425882145521227251250327
                       , 55121603546981200581762165212827652751691296897789
                       , 32238195734329339946437501907836945765883352399886
                       , 75506164965184775180738168837861091527357929701337
                       , 62177842752192623401942399639168044983993173312731
                       , 32924185707147349566916674687634660915035914677504
                       , 99518671430235219628894890102423325116913619626622
                       , 73267460800591547471830798392868535206946944540724
                       , 76841822524674417161514036427982273348055556214818
                       , 97142617910342598647204516893989422179826088076852
                       , 87783646182799346313767754307809363333018982642090
                       , 10848802521674670883215120185883543223812876952786
                       , 71329612474782464538636993009049310363619763878039
                       , 62184073572399794223406235393808339651327408011116
                       , 66627891981488087797941876876144230030984490851411
                       , 60661826293682836764744779239180335110989069790714
                       , 85786944089552990653640447425576083659976645795096
                       , 66024396409905389607120198219976047599490197230297
                       , 64913982680032973156037120041377903785566085089252
                       , 16730939319872750275468906903707539413042652315011
                       , 94809377245048795150954100921645863754710598436791
                       , 78639167021187492431995700641917969777599028300699
                       , 15368713711936614952811305876380278410754449733078
                       , 40789923115535562561142322423255033685442488917353
                       , 44889911501440648020369068063960672322193204149535
                       , 41503128880339536053299340368006977710650566631954
                       , 81234880673210146739058568557934581403627822703280
                       , 82616570773948327592232845941706525094512325230608
                       , 22918802058777319719839450180888072429661980811197
                       , 77158542502016545090413245809786882778948721859617
                       , 72107838435069186155435662884062257473692284509516
                       , 20849603980134001723930671666823555245252804609722
                       , 53503534226472524250874054075591789781264330331690]

followed by my Python solution:

#!/usr/bin/python
"""
code solution for project euler's problem #13 in python.
"""
from __future__ import print_function
 
def print_10(number):
    print(str(number)[0:10])
 
if __name__ == "__main__":
 
    big_number = [ 37107287533902102798797998220837590246510135740250,
                   46376937677490009712648124896970078050417018260538,
                   74324986199524741059474233309513058123726617309629,
                   91942213363574161572522430563301811072406154908250,
                   23067588207539346171171980310421047513778063246676,
                   89261670696623633820136378418383684178734361726757,
                   28112879812849979408065481931592621691275889832738,
                   44274228917432520321923589422876796487670272189318,
                   47451445736001306439091167216856844588711603153276,
                   70386486105843025439939619828917593665686757934951,
                   62176457141856560629502157223196586755079324193331,
                   64906352462741904929101432445813822663347944758178,
                   92575867718337217661963751590579239728245598838407,
                   58203565325359399008402633568948830189458628227828,
                   80181199384826282014278194139940567587151170094390,
                   35398664372827112653829987240784473053190104293586,
                   86515506006295864861532075273371959191420517255829,
                   71693888707715466499115593487603532921714970056938,
                   54370070576826684624621495650076471787294438377604,
                   53282654108756828443191190634694037855217779295145,
                   36123272525000296071075082563815656710885258350721,
                   45876576172410976447339110607218265236877223636045,
                   17423706905851860660448207621209813287860733969412,
                   81142660418086830619328460811191061556940512689692,
                   51934325451728388641918047049293215058642563049483,
                   62467221648435076201727918039944693004732956340691,
                   15732444386908125794514089057706229429197107928209,
                   55037687525678773091862540744969844508330393682126,
                   18336384825330154686196124348767681297534375946515,
                   80386287592878490201521685554828717201219257766954,
                   78182833757993103614740356856449095527097864797581,
                   16726320100436897842553539920931837441497806860984,
                   48403098129077791799088218795327364475675590848030,
                   87086987551392711854517078544161852424320693150332,
                   59959406895756536782107074926966537676326235447210,
                   69793950679652694742597709739166693763042633987085,
                   41052684708299085211399427365734116182760315001271,
                   65378607361501080857009149939512557028198746004375,
                   35829035317434717326932123578154982629742552737307,
                   94953759765105305946966067683156574377167401875275,
                   88902802571733229619176668713819931811048770190271,
                   25267680276078003013678680992525463401061632866526,
                   36270218540497705585629946580636237993140746255962,
                   24074486908231174977792365466257246923322810917141,
                   91430288197103288597806669760892938638285025333403,
                   34413065578016127815921815005561868836468420090470,
                   23053081172816430487623791969842487255036638784583,
                   11487696932154902810424020138335124462181441773470,
                   63783299490636259666498587618221225225512486764533,
                   67720186971698544312419572409913959008952310058822,
                   95548255300263520781532296796249481641953868218774,
                   76085327132285723110424803456124867697064507995236,
                   37774242535411291684276865538926205024910326572967,
                   23701913275725675285653248258265463092207058596522,
                   29798860272258331913126375147341994889534765745501,
                   18495701454879288984856827726077713721403798879715,
                   38298203783031473527721580348144513491373226651381,
                   34829543829199918180278916522431027392251122869539,
                   40957953066405232632538044100059654939159879593635,
                   29746152185502371307642255121183693803580388584903,
                   41698116222072977186158236678424689157993532961922,
                   62467957194401269043877107275048102390895523597457,
                   23189706772547915061505504953922979530901129967519,
                   86188088225875314529584099251203829009407770775672,
                   11306739708304724483816533873502340845647058077308,
                   82959174767140363198008187129011875491310547126581,
                   97623331044818386269515456334926366572897563400500,
                   42846280183517070527831839425882145521227251250327,
                   55121603546981200581762165212827652751691296897789,
                   32238195734329339946437501907836945765883352399886,
                   75506164965184775180738168837861091527357929701337,
                   62177842752192623401942399639168044983993173312731,
                   32924185707147349566916674687634660915035914677504,
                   99518671430235219628894890102423325116913619626622,
                   73267460800591547471830798392868535206946944540724,
                   76841822524674417161514036427982273348055556214818,
                   97142617910342598647204516893989422179826088076852,
                   87783646182799346313767754307809363333018982642090,
                   10848802521674670883215120185883543223812876952786,
                   71329612474782464538636993009049310363619763878039,
                   62184073572399794223406235393808339651327408011116,
                   66627891981488087797941876876144230030984490851411,
                   60661826293682836764744779239180335110989069790714,
                   85786944089552990653640447425576083659976645795096,
                   66024396409905389607120198219976047599490197230297,
                   64913982680032973156037120041377903785566085089252,
                   16730939319872750275468906903707539413042652315011,
                   94809377245048795150954100921645863754710598436791,
                   78639167021187492431995700641917969777599028300699,
                   15368713711936614952811305876380278410754449733078,
                   40789923115535562561142322423255033685442488917353,
                   44889911501440648020369068063960672322193204149535,
                   41503128880339536053299340368006977710650566631954,
                   81234880673210146739058568557934581403627822703280,
                   82616570773948327592232845941706525094512325230608,
                   22918802058777319719839450180888072429661980811197,
                   77158542502016545090413245809786882778948721859617,
                   72107838435069186155435662884062257473692284509516,
                   20849603980134001723930671666823555245252804609722,
                   53503534226472524250874054075591789781264330331690]
 
    print_10(sum(big_number))

and to continue adding in the spice, I have included a solution in Scala:

import BigInt._
 
object problem_13 {
    def main (args : Array[String]){
        val big_number = List("37107287533902102798797998220837590246510135740250",
                              "46376937677490009712648124896970078050417018260538",
                              "74324986199524741059474233309513058123726617309629",
                              "91942213363574161572522430563301811072406154908250",
                              "23067588207539346171171980310421047513778063246676",
                              "89261670696623633820136378418383684178734361726757",
                              "28112879812849979408065481931592621691275889832738",
                              "44274228917432520321923589422876796487670272189318",
                              "47451445736001306439091167216856844588711603153276",
                              "70386486105843025439939619828917593665686757934951",
                              "62176457141856560629502157223196586755079324193331",
                              "64906352462741904929101432445813822663347944758178",
                              "92575867718337217661963751590579239728245598838407",
                              "58203565325359399008402633568948830189458628227828",
                              "80181199384826282014278194139940567587151170094390",
                              "35398664372827112653829987240784473053190104293586",
                              "86515506006295864861532075273371959191420517255829",
                              "71693888707715466499115593487603532921714970056938",
                              "54370070576826684624621495650076471787294438377604",
                              "53282654108756828443191190634694037855217779295145",
                              "36123272525000296071075082563815656710885258350721",
                              "45876576172410976447339110607218265236877223636045",
                              "17423706905851860660448207621209813287860733969412",
                              "81142660418086830619328460811191061556940512689692",
                              "51934325451728388641918047049293215058642563049483",
                              "62467221648435076201727918039944693004732956340691",
                              "15732444386908125794514089057706229429197107928209",
                              "55037687525678773091862540744969844508330393682126",
                              "18336384825330154686196124348767681297534375946515",
                              "80386287592878490201521685554828717201219257766954",
                              "78182833757993103614740356856449095527097864797581",
                              "16726320100436897842553539920931837441497806860984",
                              "48403098129077791799088218795327364475675590848030",
                              "87086987551392711854517078544161852424320693150332",
                              "59959406895756536782107074926966537676326235447210",
                              "69793950679652694742597709739166693763042633987085",
                              "41052684708299085211399427365734116182760315001271",
                              "65378607361501080857009149939512557028198746004375",
                              "35829035317434717326932123578154982629742552737307",
                              "94953759765105305946966067683156574377167401875275",
                              "88902802571733229619176668713819931811048770190271",
                              "25267680276078003013678680992525463401061632866526",
                              "36270218540497705585629946580636237993140746255962",
                              "24074486908231174977792365466257246923322810917141",
                              "91430288197103288597806669760892938638285025333403",
                              "34413065578016127815921815005561868836468420090470",
                              "23053081172816430487623791969842487255036638784583",
                              "11487696932154902810424020138335124462181441773470",
                              "63783299490636259666498587618221225225512486764533",
                              "67720186971698544312419572409913959008952310058822",
                              "95548255300263520781532296796249481641953868218774",
                              "76085327132285723110424803456124867697064507995236",
                              "37774242535411291684276865538926205024910326572967",
                              "23701913275725675285653248258265463092207058596522",
                              "29798860272258331913126375147341994889534765745501",
                              "18495701454879288984856827726077713721403798879715",
                              "38298203783031473527721580348144513491373226651381",
                              "34829543829199918180278916522431027392251122869539",
                              "40957953066405232632538044100059654939159879593635",
                              "29746152185502371307642255121183693803580388584903",
                              "41698116222072977186158236678424689157993532961922",
                              "62467957194401269043877107275048102390895523597457",
                              "23189706772547915061505504953922979530901129967519",
                              "86188088225875314529584099251203829009407770775672",
                              "11306739708304724483816533873502340845647058077308",
                              "82959174767140363198008187129011875491310547126581",
                              "97623331044818386269515456334926366572897563400500",
                              "42846280183517070527831839425882145521227251250327",
                              "55121603546981200581762165212827652751691296897789",
                              "32238195734329339946437501907836945765883352399886",
                              "75506164965184775180738168837861091527357929701337",
                              "62177842752192623401942399639168044983993173312731",
                              "32924185707147349566916674687634660915035914677504",
                              "99518671430235219628894890102423325116913619626622",
                              "73267460800591547471830798392868535206946944540724",
                              "76841822524674417161514036427982273348055556214818",
                              "97142617910342598647204516893989422179826088076852",
                              "87783646182799346313767754307809363333018982642090",
                              "10848802521674670883215120185883543223812876952786",
                              "71329612474782464538636993009049310363619763878039",
                              "62184073572399794223406235393808339651327408011116",
                              "66627891981488087797941876876144230030984490851411",
                              "60661826293682836764744779239180335110989069790714",
                              "85786944089552990653640447425576083659976645795096",
                              "66024396409905389607120198219976047599490197230297",
                              "64913982680032973156037120041377903785566085089252",
                              "16730939319872750275468906903707539413042652315011",
                              "94809377245048795150954100921645863754710598436791",
                              "78639167021187492431995700641917969777599028300699",
                              "15368713711936614952811305876380278410754449733078",
                              "40789923115535562561142322423255033685442488917353",
                              "44889911501440648020369068063960672322193204149535",
                              "41503128880339536053299340368006977710650566631954",
                              "81234880673210146739058568557934581403627822703280",
                              "82616570773948327592232845941706525094512325230608",
                              "22918802058777319719839450180888072429661980811197",
                              "77158542502016545090413245809786882778948721859617",
                              "72107838435069186155435662884062257473692284509516",
                              "20849603980134001723930671666823555245252804609722",
                              "53503534226472524250874054075591789781264330331690") map {BigInt(_)}
        val sums = big_number sum
        val su = sums toString
        val su10 = su take 10
        println(su10)
    }
}

The solution, in all three languages, is pretty simple. The recipe essentially says, “Put all numbers into a list. Get the sum of that list, turn that number into a string, and get the first 10 characters of that string.”

Times:
Haskell (compiled) : real 0m0.004s
Haskell (runghc) : real 0m0.314s
Python : real 0m0.059s
Scala (compiled) : real 0m0.757s

For the most part it's pretty standard in these tests to see performance times such that Haskell (compiled) Python Haskell (runghc). Java and Perl usually fall somewhere between the Haskell (compiled) and Python, in that order. To see Scala be 2x slower than Haskell (runghc) was a shocker. The only thing that makes sense to me for the slowdown is having to use the BigInt library. That is probably the biggest thing I took away from these time tests - if I want to do REALLY large number crunching and performance DOES matter, JVM-based languages might not be the best option.

A few thoughts on Scala:
If I haven't stated it already in this blog, I should now give the disclaimer that I'm not a Java fan. I know it still has its loyal followers, but I'm not one of them. Moving on. This was my first time working with Scala, and I'd like to finally welcome Java to the 21st century. While doing some research on the Scala language itself I read that “the industry” was moving to replace Java with Scala. I welcome that change. Does that mean I “like” Scala? The honest answer is, to butcher the quote the appliances from the Flintstones, “Eh, it's a language.” Scala is definitely an improvement over Java – not really that hard to do in my opinion – but, the language still feels unpolished. One quick way to kill the interpreter in Scala is to type “Int” then hit the enter key. Instead of error-ing out, the interpreter does a great job of interpreting a crash test car hitting a cement wall (I had to restart the whole thing.) When I tried the same “technique” in the Python interpreter, I got as a response and for Haskell's interpreter I received “Not in scope: data constructor `Int'”. I also found Scala's function composition to be a little lacking when compared to Haskell. I wasn't able to cleanly change the BigInt data type to String, and then only print out ten characters without requiring three separate val's. Yes, I could have used one var instead, but that's beside the point. I will admit it could be my inexperience with the language showing, so if anyone knows a smoother way to do this in Scala please share it in the comments.

All that being said, I do like the way Scala is trying to handle the reducing of Java's dot notation, and I think it's starting to make strides in the right direction in other areas. I'm open to working with Scala more, and look forward to seeing how it evolves over the next few years.

This version requires the following packages pip installed into your virtualenv.

• django-crispy-forms so we can do Python driven layouts.
• django-floppyforms so we get HTML5 elements for free.

They also need to be added to your list of INSTALLED_APPS:

INSTALLED_APPS += (
    'crispy_forms',
    'floppyforms',
)

In myapp.forms.py:

from crispy_forms.helper import FormHelper
from crispy_forms.layout import Submit
import floppyforms as forms

class ContactForm(forms.Form):

    name = forms.CharField(required=True)
    email = forms.EmailField(required=True)
    subject = forms.CharField(required=True)
    message = forms.CharField(widget=forms.Textarea)

    def __init__(self, *args, **kwargs):
        self.helper = FormHelper()
        self.helper.add_input(Submit('submit', 'Submit'))
        super(ContactForm, self).__init__(*args, **kwargs)

In myapp.views.py:

from django.conf import settings
from django.core.mail import send_mail
from django.views.generic import FormView

from myapp.forms import ContactForm

class ContactFormView(FormView):

    form_class = ContactForm
    template_name = "myapp/email_form.html"
    success_url = '/email-sent/'

    def form_valid(self, form):
        message = "{name} / {email} said: ".format(
            name=form.cleaned_data.get('name'),
            email=form.cleaned_data.get('email'))
        message += "\n\n{0}".format(form.cleaned_data.get('message'))
        send_mail(
            subject=form.cleaned_data.get('subject'),
            message=message,
            from_email='contact-form@myapp.com',
            recipient_list=[settings.LIST_OF_EMAIL_RECIPIENTS],
        )
        return super(ContactFormView, self).form_valid(form)

In templates/myapp/email_form.html:

{% extends 'base.html' %}
{% load crispy_forms_tags %}

{% block title %}Send an email{% endblock %}

{% block content %}
    <div class="row">
        <div class="span6">
            <h1>Send an email</h1>
            {% crispy form form.helper %}
        </div>
    </div>
{% endblock %}

{% block extrajs %}
<script src="{{ STATIC_URL }}js/jquery-1.7.1.min.js"></script>
<script type="text/javascript">
$(function() {
    $('#id_name').focus()
});
</script>
{% endblock %}

Last tutorial we covered ‘parsing’.  We broke a standard config file up into separate lines and then we broke each line into pairs, each pair having a ‘key’ and a ‘value’.   If we’re to edit the file, we need a way to edit the values (and after that we’ll write the edited values back to the config file).  We’re going to see how to use Tkinter to do that in this tutorial.

Let’s start thinking about how a single key/value pair will look.  I am thinking of having the key on the left hand side with a space to input the value on the right hand side.  To do this we will use the Label widget that we’ve met before and the Text widget.  Label is used to display static text – ie text that will not be edited, while the Text widget allows the user to edit the text which is displayed in the widget, and for the program to read what is entered in the widget.  The Text widget is the GUI equivalent of raw_input() that we met so long ago.

from Tkinter import *

root = Tk()
labelWidget = Label(root,text="A key:")
textWidget = Text(root)

textWidget.insert('1.0',"A Value")
labelWidget.pack(side=LEFT)
textWidget.pack(side=RIGHT)

root.mainloop()

Here we first create a Tk() object, then create a label widget and a text widget in that object.   We pack the label first and we pack it on the LEFT side (LEFT is actually the name of a constant in Tk which Tk translates to ‘put this on the left’) and pack the text widget on the right side.  At location “1.0″ we add the text “A Value” to the Text widget.  Here the number “1.0″ means “row 1, at character position 0″, which is to say, at the very start of the text.

If I run this code I get something like this:

Which is sort of what I wanted – a label on the left, and an editable text box on the right (can you see the cursor in the screen shot?) – click the close window widget in the top right corner to close the window.

Exercise:  Type something into the text box.  See if you can do it to the label.

However, this isn’t really what I wanted.  I wanted a little text box, not the enormous one I’ve got here.  Since I didn’t specify a height and width for it, the Text widget used its default size (which is way too big).  The user also doesn’t have any way to tell the program to use (or cancel) the edit.  Let’s change the code to add an ok and cancel button, and to change the size of the text widget.

Here is a revised version which is nearer to what I was looking for:

from Tkinter import *

def okClicked():
    '''Get the edited values and write them to the file then quit'''
    #TODO: get values and write them to the file!
    exit()
def cancelClicked():
    '''Cancel edits and quit'''
    exit()

root = Tk()
labelWidget = Label(root,text="A key:")
textWidget = Text(root, width=60, height = 1)
okWidget = Button(root, text= "Ok", command = okClicked)
cancelWidget = Button(root, text="Cancel", command = cancelClicked)

textWidget.insert('1.0',"A Value")
labelWidget.pack(side=LEFT)
cancelWidget.pack(side=RIGHT)
okWidget.pack(side=RIGHT)

textWidget.pack(side=RIGHT)

root.mainloop()

This gives:

I have added a couple of functions to be run when the ok and cancel buttons are clicked.  The ok button’s function is still a little empty at the moment though…  I have specified the width to be 60 characters and the height to be one row.  Note these are not pixel measurements.  If you change the size of the font the text box will also change.

Notice also the way the geometry is working.  The widgets which are pack(side=RIGHT) are added at the right in the order they are packed.  If the buttons were packed last they would be between the label and the text window.

Exercise: change the program so that the widgets are packed in a different order. What happens if you try side=TOP or side=BOTTOM?

The value can be edited by the user typing directly into the text box.  The text in the text box can also be edited programmatically, which is to say its contents can be changed by the program without the user typing.  See the Tkinter documentation for details – or clamour on this site and I’ll add a tute.

One thing that I don’t like about this layout is the fact the buttons are on the same line as the key label and value text.  When you did the exercise above, you should have noticed that side=TOP and side=BOTTOM don’t really help, since you can’t position the ok and cancel buttons on the same line.  What we need to use is the Frame widget.  Frames can be thought of as empty spaces in which you can group widgets together.  By treating the widgets within the Frame as a group, additional layouts can be achieved.   Frames can be packed inside other frames. We will use two frames, one on top of the other.  In the first frame we pack the label and text widgets.  In the second frame we pack the two buttons.

Here is the code:

from Tkinter import *

def okClicked():
    '''Get the edited values and write them to the file then quit'''
    #TODO: get values and write them to the file!
    exit()
def cancelClicked():
    '''Cancel edits and quit'''
    exit()

root = Tk()

topFrame = Frame(root)
bottomFrame = Frame(root)
labelWidget = Label(topFrame,text="A key:")
textWidget = Text(topFrame, width=60, height = 1)
okWidget = Button(bottomFrame, text= "Ok", command = okClicked)
cancelWidget = Button(bottomFrame, text="Cancel", command = cancelClicked)

textWidget.insert('1.0',"A Value")
labelWidget.pack(side=LEFT)
cancelWidget.pack(side=RIGHT)
okWidget.pack(side=LEFT)
textWidget.pack(side=RIGHT)

topFrame.pack(side=TOP)
bottomFrame.pack(side=BOTTOM)

root.mainloop()

This is more like what I wanted (notice you can’t see the individual frames).  One might quibble with the location of the ok and cancel buttons.  Maybe they should be offset a little from the centre?  Maybe they should be off to one side.  In any event they are in the general layout that i was looking for: a key label and an editable value text above the ok and cancel buttons. Notice in the code that the widgets we used before have had their parent changed from root to either topFrame or bottomFrame?  However these Frame widgets have root as their parent.   So the original widgets we were using have effectively been pushed down one level in the hierarchy.

We still don’t know how to get what the user has typed into the text box, but maybe I can leave that as an exercise for the reader (try textWidget.get(“1.0″,END)).

Homework:  change the okClicked() function so that it prints the contents of the text box before exiting.  Use the hint in the previous paragraph.

In the previous tutorial we learnt that when we pack Tkinter objects, the order in which we pack them affects how they are displayed in the GUI.  We used a Text widget to enable the user to edit text in the GUI.  We also learnt to use a Frame widget to help with the layout of our GUI.  In particular, we put a Label and a Text widget together into one Frame, and put an ok and cancel Button into another.   For homework you needed to get data from the Text widget.

In order to display all of the configuration options we are going to go a bit nutty using Frames.   We will eventually (but not today) use one Frame to hold all of the configuration options, and another Frame to hold the Ok and Cancel buttons.  But that’s not all!  We will also use a Frame to house each configuration option (ie key and value pair).  Before we do that though, we need to remember where we were up to reading and parsing the server.properties file.

Here is the code we finished with two tutorials ago, excluding the last couple of lines (which printed out the results) for your reference if you need it (click to expand):

'''Minecraft config editor:
This is an editor for the Minecraft server.properties file.  It:
* opens the file server.properties
* reads, then closes the file
* parses each line by
-- stripping leading and trailing whitespace
-- if the line starts with "#", marks it as a comment
-- splits the line into a key, value pair, with the pair separated by a "=" sign
-- if the value of the pair is either "true" or "false", the entry is marked as a boolean (ie its only values are either true or false)
* displays each key, value entry on the screen allowing you to edit it
* renames the server.properties file to server.properties.bup (overwriting any existing file of that name from earlier edits)
* opens a new file called server.properties
* writes each of the entries to that new file
* closes the server.properties file.
'''

class configItem(object):  # name of the class, it is based on an object called 'object'
  def __init__(self, line):# this is called each time an instance of the class is created
    line = line.strip()  # this removes any white space at the start or end of the line
    # if it starts with a # it's a comment so check for it
    if line[:1] == "#":
       self.configKey = "#"
       self.configVal = line[1:]
    else:  # otherwise assume it's of the form x = y
       spam = line.split("=")
       self.configKey = spam[0]
       self.configVal = spam[1]
    # now check to see whether the config item takes only the values "true" and "false"
    if self.configVal.lower() in ["true","false"]:
       self.isTrueFalse = True
    else:
       self.isTrueFalse = False

# get data from the file

fileName = "server.properties"
fileObject = open(fileName,'rb')
fileData = fileObject.read()
fileObject.close()

configLines = []

for line in fileData.split('\n'):  # this splits it into individual lines
    if line.strip()=='':
      continue
    configLines.append(configItem(line))

If you remember we defined a class called configItem.  We read the lines from the config file and used each line to create instances of configItem.  We stored those in an array called configLines.  Each instance has two attributes – configKey and configVal (that is, the things on the left and right hand side of the equals respectively).  In the last tutorial for one key, value pair we:

• created a label and set it equal to configKey;
• created a text widget and set its value to configVal; and, finally,
• created a frame in which to pack each of these.

Now we have to do that for each and every entry in the array.  There are plenty of ways to do this.  However, I am going to do it by “subclassing” the configItem class.  That is, I am going to create a new class which is based on (“inherits from” or “is a subclass of”) the configItem class.  It has the features of the configItem class but will also store some stuff relating to the Tkinter widgets that we will need.  This is the new class which I’ve called guiConfigItem:

class guiConfigItem(configItem):
  def __init__(self,line):
    super(guiConfigItem,self).__init__(line)  # run configItem's __init__ method
    self.frame = Frame()
    self.keyLabel = Label(self.frame, text = self.configKey)
    self.valueEntry = Entry(self.frame, width="60")
    self.valueEntry.insert("0",self.configVal)
    self.keyLabel.pack(side=LEFT )
    self.valueEntry.pack(side=RIGHT)
    self.frame.pack(side=TOP)

Some things to note about this class:

• instead of the first line ending “(object):” like the other classes we’ve seen, this one ends “(configItem):”.  This means that guiConfigItem’s immediate parent is configItem.  However, since configItem is based on object, in the end, so is guiConfigItem.
• it takes the same initialisation parameters as configItem (that is, self and line)
• the first thing it does in initialising stuff is to call super(guiConfigItem,self).__init__(line).  This runs configItem’s __init__ method, so every guiConfigItem starts with the same initialisation that configItem would have
• it starts by creating a Frame, stores it in self.frame, then, inside the frame, it creates a label and an Entry widget.  An Entry widget is the single line version of the Text widget we used last time, and it should be good enough for our purposes.
• you can tell that the Label and Entry widgets are created inside the frame which has been created because the first parameter passed to them is self.frame.
• the Label widget is packed to the LEFT, and the Entry widget to the RIGHT.  the frame is also packed, but it is packed to TOP (ie it will make a list from top to bottom)

This class is added after the definition of the configItem class.  In order to get it working we just have to make a four changes to the program.  We will:

• import Tkinter – from Tkinter import *;
• create a root window in which to pack things - root =Tk();
• change the loop to create guiConfigItems rather than configItems – configLines.append(guiConfigItem(line=line)); and
• we will start the gui with a mainloop() – root.mainloop()

Here is the updated source code:

'''Minecraft config editor:
This is an editor for the Minecraft server.properties file.  It:
* opens the file server.properties
* reads, then closes the file
* parses each line by
-- stripping leading and trailing whitespace
-- if the line starts with "#", marks it as a comment
-- splits the line into a key, value pair, with the pair separated by a "=" sign
-- if the value of the pair is either "true" or "false", the entry is marked as a boolean (ie its only values are either true or false)
* displays each key, value entry on the screen allowing you to edit it
* renames the server.properties file to server.properties.bup (overwriting any existing file of that name from earlier edits)
* opens a new file called server.properties
* writes each of the entries to that new file
* closes the server.properties file.
'''

from Tkinter import *

class configItem(object):  # name of the class, it is based on an object called 'object'
  def __init__(self, line):# this is called each time an instance of the class is created
    line = line.strip()  # this removes any white space at the start or end of the line
    # if it starts with a # it's a comment so check for it
    if line[:1] == "#":
       self.configKey = "#"
       self.configVal = line[1:]
    else:  # otherwise assume it's of the form x = y
       spam = line.split("=")
       self.configKey = spam[0]
       self.configVal = spam[1]
    # now check to see whether the config item takes only the values "true" and "false"
    if self.configVal.lower() in ["true","false"]:
       self.isTrueFalse = True
    else:
       self.isTrueFalse = False

class guiConfigItem(configItem):
  def __init__(self,line):
    super(guiConfigItem,self).__init__(line)  # run configItem's __init__ method
    self.frame = Frame()
    self.keyLabel = Label(self.frame, text = self.configKey)
    self.valueEntry = Entry(self.frame, width="60")
    self.valueEntry.insert("0",self.configVal)
    self.keyLabel.pack(side=LEFT )
    self.valueEntry.pack(side=RIGHT)
    self.frame.pack(side=TOP)

# get data from the file

fileName = "server.properties"
fileObject = open(fileName,'rb')
fileData = fileObject.read()
fileObject.close()

root = Tk()
configLines = []

for line in fileData.split('\n'):  # this splits it into individual lines
    if line.strip()=='':
      continue
    configLines.append(guiConfigItem(line=line))

root.mainloop()

When I run this I get:

Wow, is that magic? The way we defined the class meant that each of the instances packed itself for us as we created them.  This is an example of why using classes can be so much fun.

Exercise: how might you do the same thing without using classes?

That said, the alignment is a little wonky.   This is because each of the individual frames (there is one on each line) are different sizes.  The overall window is big enough to fit the biggest, but that means that the smaller lines aren’t big enough.  This can be remedied by adding fill=”x” (that is, fill in the x (horizontal) direction if necessary to the pack command for each of the Frames:

self.frame.pack(side=TOP, fill="x")

Now the window looks much better:

Exercise: confirm that you can edit the values on the right.

Exercise 2: check through our docstring to see what we’ve done so far and what we’ve got left to do.

The complete source code with the final edit is below.

'''Minecraft config editor:
This is an editor for the Minecraft server.properties file.  It:
* opens the file server.properties
* reads, then closes the file
* parses each line by
-- stripping leading and trailing whitespace
-- if the line starts with "#", marks it as a comment
-- splits the line into a key, value pair, with the pair separated by a "=" sign
-- if the value of the pair is either "true" or "false", the entry is marked as a boolean (ie its only values are either true or false)
* displays each key, value entry on the screen allowing you to edit it
* renames the server.properties file to server.properties.bup (overwriting any existing file of that name from earlier edits)
* opens a new file called server.properties
* writes each of the entries to that new file
* closes the server.properties file.
'''

from Tkinter import *

class configItem(object):  # name of the class, it is based on an object called 'object'
  def __init__(self, line):# this is called each time an instance of the class is created
    line = line.strip()  # this removes any white space at the start or end of the line
    # if it starts with a # it's a comment so check for it
    if line[:1] == "#":
       self.configKey = "#"
       self.configVal = line[1:]
    else:  # otherwise assume it's of the form x = y
       spam = line.split("=")
       self.configKey = spam[0]
       self.configVal = spam[1]
    # now check to see whether the config item takes only the values "true" and "false"
    if self.configVal.lower() in ["true","false"]:
       self.isTrueFalse = True
    else:
       self.isTrueFalse = False

class guiConfigItem(configItem):
  def __init__(self,line):
    super(guiConfigItem,self).__init__(line)  # run configItem's __init__ method
    self.frame = Frame()
    self.keyLabel = Label(self.frame, text = self.configKey)
    self.valueEntry = Entry(self.frame, width="60")
    self.valueEntry.insert("0",self.configVal)
    self.keyLabel.pack(side=LEFT )
    self.valueEntry.pack(side=RIGHT)
    self.frame.pack(side=TOP, fill="x")

# get data from the file

fileName = "server.properties"
fileObject = open(fileName,'rb')
fileData = fileObject.read()
fileObject.close()

root = Tk()
configLines = []

for line in fileData.split('\n'):  # this splits it into individual lines
    if line.strip()=='':
      continue
    configLines.append(guiConfigItem(line=line))

root.mainloop()

There are two more things to do with our Minecraft config file editor before we’ve got the main part of it working (we may do some tweaking later).  We need to:

• add the Ok and Cancel buttons back; and
• when someone clicks Ok, we need to update the server.properties file

We’re doing the first of these today. We saw earlier how to do the Ok and Cancel buttons, although at the time we didn’t actually put any meat in the functions they called.  So, let’s fill that out now.  For Cancel, we are just going to quit the editor without making any changes – that’s pretty easy.  For the Ok button though, we’re going to have to:

1. somehow read all of the values from the screen (since we don’t know which ones have been changed we need to read them all);
2. make a backup of the server.properties file
3. write all of the key:value pairs to the new server.properties file.

Unlike variables, widgets are not the same as what is stored in them.  If we have an Entry widget called E and we want to store what has been typed there in a variable called text we can’t just write text = E.   This is because E is not a variable as we understand it.  Actually, E is an instance of a class.  This would just make another reference to the same Entry widget with the name text.  Rather, we want to “get” the current value of the text entered into E.  It turns out that the Entry widget has a method (called get()) which gets that text for you.

>>> from Tkinter import *
>>> E = Entry()  # this should pop up a Tkinter window
>>> E.pack()  # the widget should appear in your Tkinter window now
>>> type(E)
<type 'instance'>
>>> type(Entry)
<type 'classobj'>
>>> text = E
>>> type(text)
<type 'instance'>
>>> print text
.140543131462184
>>> # now type "Hi P4K!" in the entry widget
...
>>> text = E.get()
>>> print text
Hi P4K!
>>> # now add " - Again" to the end of the entry widget (leave the "Hi P4K!" there)
...
>>> text = E.get()
>>> print text
Hi P4K! - Again
>>> # you can also print the value which you get() without storing it first:
...
>>> print E.get()
Hi P4K! - Again

So what we’re going to do in our code is get() all these edited values when someone clicks “Ok”.  We could do that directly, for example by finding the relevant guiConfigItem and calling the get() method on the valueEntry attribute of that item.   That would also mean we’d have to make a copy of the key for that item and then combine them together with “=” before we wrote them to the server.properties file.  This would mean that logic which is relevant to the configItem class would be stored somewhere other than inside the class - which rather defeats the purpose of having a class to keep track of these things.  Instead, we’re going to add a method to the guiConfigItem class which updates the values it has stored.  That turns out to be pretty easy:

  def update(self):
    ''' Get the value which is currently in the Entry widget and save it to configVal'''
    if self.isTrueFalse:
      '''if isTrueFalse is True, then we should only have the values 'true' and 'false' in this
      Entry.  So, only update the configuration value if it is one of these two.  Otherwise, ignore it. '''
      spam = self.valueEntry.get()
      if spam in ['true','false']:
	self.configVal = spam
    else:
      '''this is not a variable which is limited to 'true' and 'false', so store the whole text'''
      self.configVal = self.valueEntry.get()

Note here that we are referencing the attribute configVal which is defined in the parent class.  Also note that we’ve included a bit of logic here to ensure that those configuration values which start as ‘true’ or ‘false’ can only be ‘true’ or ‘false’.  If you type something else into them it will be ignored.  It is sufficient here to just say if self.isTrueFalse rather than if self.isTrueFalse is True (the “is True” is redundant).

We also need a way to prepare the lines of the server.properties file to be printed or written to the file.  We do this by adding a method to the configItem class (since it doesn’t have anything to do with the graphical interface we don’t add it to the subclass):

  def item2ConfigLine(self):
    if self.configKey=="#":
      '''If the key is '#' then this is a comment, so don't include an '=' sign'''
      return "%s%s"%(self.configKey, self.configVal)
    else:
      '''otherwise, it has the form key=value'''
      return "%s=%s"%(self.configKey,self.configVal)

See this tutorial for an explanation of the %s stuff…

We don’t have a way to test these methods out yet. So let’s hook up the Ok and Cancel buttons. The Ok button will run through each of the guiConfigItems and update it, then print out the configuration line. After all items have been processed this way, the program will exit. The cancel button will just exit without doing anything. So, we need to:
1. create callbacks for each of the buttons,
2. create a frame for the buttons to go in
3. create the buttons, hooking up each of the buttons up to the callback
4. pack the buttons, then, finally,
5. pack the frame.
These go before the root.mainloop() line.

# 1. create callbacks for each of the buttons,
def okClicked():
  '''Get the edited values and write them to the file then quit'''
  for c in configLines:
    c.update()
    print c.item2ConfigLine()
  # have updated and printed each line, now exit
  exit()

def cancelClicked():
  '''Cancel edits and quit'''
  exit()

# 2. create a frame for the buttons to go in
bottomFrame = Frame(root)

# 3. create the buttons, hooking up each of the buttons up to the callback
okWidget = Button(bottomFrame, text= "Ok", command = okClicked)
cancelWidget = Button(bottomFrame, text="Cancel", command = cancelClicked)

# 4. pack the buttons, then, finally,
okWidget.pack(side=LEFT)
cancelWidget.pack(side=RIGHT)
# 5. then pack the frame:
bottomFrame.pack(side=BOTTOM)

At the moment, the Ok button just prints out the values of the items. This is so that we can test how it is working before we let it go editing the actual file.

Here is the complete source code:

# -*- coding: utf-8 -*-

'''Minecraft config editor:
This is an editor for the Minecraft server.properties file.  It:
* opens the file server.properties
* reads, then closes the file
* parses each line by
-- stripping leading and trailing whitespace
-- if the line starts with "#", marks it as a comment
-- splits the line into a key, value pair, with the pair separated by a "=" sign
-- if the value of the pair is either "true" or "false", the entry is marked as a boolean (ie its only values are either true or false)
* displays each key, value entry on the screen allowing you to edit it
* renames the server.properties file to server.properties.bup (overwriting any existing file of that name from earlier edits)
* opens a new file called server.properties
* writes each of the entries to that new file
* closes the server.properties file.
'''

from Tkinter import *

class configItem(object):  # name of the class, it is based on an object called 'object'
  def __init__(self, line):# this is called each time an instance of the class is created
    line = line.strip()  # this removes any white space at the start or end of the line
    # if it starts with a # it's a comment so check for it
    if line[:1] == "#":
       self.configKey = "#"
       self.configVal = line[1:]
    else:  # otherwise assume it's of the form x = y
       spam = line.split("=")
       self.configKey = spam[0]
       self.configVal = spam[1]
    # now check to see whether the config item takes only the values "true" and "false"
    if self.configVal.lower() in ["true","false"]:
       self.isTrueFalse = True
    else:
       self.isTrueFalse = False

  def item2ConfigLine(self):
    if self.configKey=="#":
      '''If the key is '#' then this is a comment, so don't include an '=' sign'''
      return "%s%s"%(self.configKey, self.configVal)
    else:
      '''otherwise, it has the form key=value'''
      return "%s=%s"%(self.configKey,self.configVal)

class guiConfigItem(configItem):
  def __init__(self,line):
    super(guiConfigItem,self).__init__(line)  # run configItem's __init__ method
    self.frame = Frame()
    self.keyLabel = Label(self.frame, text = self.configKey)
    self.valueEntry = Entry(self.frame, width="60")
    self.valueEntry.insert("0",self.configVal)
    self.keyLabel.pack(side=LEFT )
    self.valueEntry.pack(side=RIGHT)
    self.frame.pack(side=TOP, fill="x")

  def update(self):
    ''' Get the value which is currently in the Entry widget and save it to configVal'''
    if self.isTrueFalse:
      '''if isTrueFalse is True, then we should only have the values 'true' and 'false' in this
      Entry.  So, only update the configuration value if it is one of these two.  Otherwise, ignore it. '''
      spam = self.valueEntry.get()
      if spam in ['true','false']:
	self.configVal = spam
    else:
      '''this is not a variable which is limited to 'true' and 'false', so store the whole text'''
      self.configVal = self.valueEntry.get()

# get data from the file

fileName = "server.properties"
fileObject = open(fileName,'rb')
fileData = fileObject.read()
fileObject.close()

root = Tk()
configLines = []

for line in fileData.split('\n'):  # this splits it into individual lines
    if line.strip()=='':
      continue
    configLines.append(guiConfigItem(line=line))

# 1. create callbacks for each of the buttons,
def okClicked():
  '''Get the edited values and write them to the file then quit'''
  for c in configLines:
    c.update()
    print c.item2ConfigLine()
  # have updated and printed each line, now exit
  exit()

def cancelClicked():
  '''Cancel edits and quit'''
  exit()

# 2. create a frame for the buttons to go in
bottomFrame = Frame(root)

# 3. create the buttons, hooking up each of the buttons up to the callback
okWidget = Button(bottomFrame, text= "Ok", command = okClicked)
cancelWidget = Button(bottomFrame, text="Cancel", command = cancelClicked)

# 4. pack the buttons, then, finally,
okWidget.pack(side=LEFT)
cancelWidget.pack(side=RIGHT)
# 5. then pack the frame:
bottomFrame.pack(side=BOTTOM)

root.mainloop()

Exercise: run the code and confirm that: (a) your edits are captured and printed out; (b) if you enter anything but “true” or “false” for an item that takes only true and false, then the edit is ignored; and (c) that if you change a true to a false or vice versa, that that edit is captured.

This is our final instalment [sic] of our Minecraft config editor.   In the earlier tutorials we have done everything except actually updating the file.  Before we do update the file though, we need to make a backup of it, so it’s these two things that we’re going to do now.

All of the action will be in changing the behaviour of the ‘Ok’ button so that it makes a copy of server.properties into a new file called server.properties.bup and then writes the updated data to the server.properties file.  There is a small amount of work in making a copy of the file so we are going to do it in a separate function.  The function is not very smart.  It reads all of the data from the existing file and then just writes it out to the new file:

def backupFile(fileName):
  ''' Quick and dirty copy of file to fileName+".bup" - might also use os.rename(), but behaviour of os.rename is platform dependent '''
  fileObject = open(fileName,'rb')
  fileData = fileObject.read()
  fileObject.close()
  fileObject= open(fileName+".bup",'wb')  # will overwrite if it exists
  fileObject.write(fileData)
  fileObject.close()

The function could also use the rename() method from the os module.  Unfortunately, this behaves differently depending on the operating system you are using, so to keep it simple I have avoided it.

Exercise: Work out what this function is supposed to do, then confirm that it does it eg: start up a Python console, paste the definition in and then call the function with “server.properties” as a parameter.

Extra Points: If you are on a Unix based system, use the diff command to show the differences between the original and .bup (there shouldn’t be any).

With that done we can hook up the backup to the ‘Ok’ callback, and also write the new data:

def okClicked():
  '''Get the edited values and write them to the file then quit'''
  #TODO: add a confirmation dialog
  global fileName
  backupFile(fileName)
  dataToWrite = []
  for c in configLines:
    c.update()
    dataToWrite.append(c.item2ConfigLine())
  # have updated and printed each line, now exit
  fileObject = open(fileName,'wt')
  fileObject.write('\n'.join(dataToWrite)+'\n')
  # '\n' is technically not the newline character on Windows
  # but by default Python converts \n to the correct character on write
  # not sure if minecraft needs a final '\n', so included one just in case
  fileObject.close()
  exit()

Here we have introduced an array called dataToWrite.  Where, in the last tute we just printed out the line, in this tute we are appending those lines to the dataToWrite array.  Then, once they have been accumulated, we open the server.properties file (clearing it) then write our new data into it.  Only one generation of backup is saved.

We have used the global statement here to use the value of the fileName variable.  This is a little messy, but is a consequence of how the program has evolved.

One of the things that you might include here is a confirmation step.  Before the data gets overwritten we might ask the user to confirm that they are going to write over their data, giving them a second chance if they clicked “Ok” by mistake.

Complete code here:

# -*- coding: utf-8 -*-

'''Minecraft config editor:
This is an editor for the Minecraft server.properties file.  It:
* opens the file server.properties
* reads, then closes the file
* parses each line by
-- stripping leading and trailing whitespace
-- if the line starts with "#", marks it as a comment
-- splits the line into a key, value pair, with the pair separated by a "=" sign
-- if the value of the pair is either "true" or "false", the entry is marked as a boolean (ie its only values are either true or false)
* displays each key, value entry on the screen allowing you to edit it
* renames the server.properties file to server.properties.bup (overwriting any existing file of that name from earlier edits)
* opens a new file called server.properties
* writes each of the entries to that new file
* closes the server.properties file.
'''

from Tkinter import *


class configItem(object):  # name of the class, it is based on an object called 'object'
  def __init__(self, line):# this is called each time an instance of the class is created
    line = line.strip()  # this removes any white space at the start or end of the line
    # if it starts with a # it's a comment so check for it
    if line[:1] == "#":
       self.configKey = "#"
       self.configVal = line[1:]
    else:  # otherwise assume it's of the form x = y
       spam = line.split("=")
       self.configKey = spam[0]
       self.configVal = spam[1]
    # now check to see whether the config item takes only the values "true" and "false" 
    if self.configVal.lower() in ["true","false"]:
       self.isTrueFalse = True
    else:
       self.isTrueFalse = False
       
  def item2ConfigLine(self):
    if self.configKey=="#":
      '''If the key is '#' then this is a comment, so don't include an '=' sign'''
      return "%s%s"%(self.configKey, self.configVal)
    else:
      '''otherwise, it has the form key=value'''
      return "%s=%s"%(self.configKey,self.configVal)
      
   

class guiConfigItem(configItem):
  def __init__(self,line):
    super(guiConfigItem,self).__init__(line)  # run configItem's __init__ method
    self.frame = Frame()
    self.keyLabel = Label(self.frame, text = self.configKey)
    self.valueEntry = Entry(self.frame, width="60")
    self.valueEntry.insert("0",self.configVal)
    self.keyLabel.pack(side=LEFT )
    self.valueEntry.pack(side=RIGHT)
    self.frame.pack(side=TOP, fill="x")
 
  def update(self):
    ''' Get the value which is currently in the Entry widget and save it to configVal'''
    if self.isTrueFalse:
      '''if isTrueFalse is True, then we should only have the values 'true' and 'false' in this
      Entry.  So, only update the configuration value if it is one of these two.  Otherwise, ignore it. '''
      spam = self.valueEntry.get()
      if spam in ['true','false']:
	self.configVal = spam
    else:
      '''this is not a variable which is limited to 'true' and 'false', so store the whole text'''
      self.configVal = self.valueEntry.get()
    
    
# get data from the file

fileName = "ser_ver.properties"
fileObject = open(fileName,'rb')
fileData = fileObject.read()
fileObject.close()



root = Tk()
configLines = []

for line in fileData.split('\n'):  # this splits it into individual lines
    if line.strip()=='':
      continue
    configLines.append(guiConfigItem(line=line))

# 1. create callbacks for each of the buttons, 
def okClicked():
  '''Get the edited values and write them to the file then quit'''
  #TODO: add a confirmation dialog
  global fileName
  backupFile(fileName)
  dataToWrite = []
  for c in configLines:
    c.update()
    dataToWrite.append(c.item2ConfigLine())
  # have updated and printed each line, now exit
  fileObject = open(fileName,'wt')
  fileObject.write('\n'.join(dataToWrite)+'\n')
  # '\n' is technically not the newline character on Windows
  # but by default Python converts \n to the correct character on write
  # not sure if minecraft needs a final '\n', so included one just in case
  fileObject.close()
  exit()

def backupFile(fileName):
  ''' Quick and dirty copy of file to fileName+".bup" - might also use os.rename(), but behaviour of os.rename is platform dependent '''
  fileObject = open(fileName,'rb')
  fileData = fileObject.read()
  fileObject.close()
  fileObject= open(fileName+".bup",'wb')  # will overwrite if it exists
  fileObject.write(fileData)
  fileObject.close()

def cancelClicked():
  '''Cancel edits and quit'''
  exit()

# 2. create a frame for the buttons to go in
bottomFrame = Frame(root)

# 3. create the buttons, hooking up each of the buttons up to the callback
okWidget = Button(bottomFrame, text= "Ok", command = okClicked)
cancelWidget = Button(bottomFrame, text="Cancel", command = cancelClicked)

# 4. pack the buttons, then, finally,
okWidget.pack(side=LEFT)
cancelWidget.pack(side=RIGHT) 
# 5. then pack the frame:
bottomFrame.pack(side=BOTTOM)

root.mainloop()

Exercise: confirm that “Ok” saves your edits (open server.properties in a text editor or (extra points) write some Python to read and print the contents of the file) and that “Cancel” doesn’t.

Comments:

The code is a little messy because of how it has evolved in the course of explaining it.  Having code growing organically and getting messy is not unusual.  Every once in a while you need to stop and clean it up.  Cleaning it up can also allow you to restructure your code in ways you didn’t realise when you were writing it in the first place.

PS

My class names are naughty.  They should start with a capital letter.

Summary: id(), copy, copy.copy(), copy.deepcopy()

A short tutorial this week on some oddities with the way Python stores and references (“binds to”) data.   You might remember, a long time ago, we talked about how, when we store data in a variable, it’s like putting your stuff in a bucket so that you can access it later.  Variables in Python actually turn out to be references to objects.   A side effect of this is that, in some cases, Python doesn’t work out how you think it will – typically this is where your object is a list or dictionary (actually any object, but you only notice this effect with compound objects) that you think you have copied, but you actually haven’t.

In particular, you can do this with ‘plain’ variables:

 >>> a = 5
 >>> b = a
 >>> a = 6
 >>> b
 5

You can also do this with lists:

 >>> c = [1,2]
 >>> d= c
 >>> c =[5,6]
 >>> d
 [1, 2]

But there’s a gotcha with lists where you change one of the list’s entries:

 >>> c= [1,2]
 >>> d = c
 >>> d
 [1, 2]
 >>> c[0]=3
 >>> d
 [3, 2]

Can you see that, even though we only changed the first entry in the list c (that is, c[0]), the first entry of d has also changed?  That’s because there is an underlying list object that both c and d are pointing to.  That is, they are both pointing to the same thing.  In a sense they are both windows to the same room (the list object).  Looking in either window allows you to “see” the changes made in the room.   You can see that the objects are the same because you can check their location in memory using the id() function which is built in to Python (try help(id)):

 >>> id(c)
 139636641421288
 >>> id(d)
 139636641421288
 >>> id(c) == id(d)
 True

The number (139636641421288) is where in the computer’s memory the object is stored.  It will change, probably each time you run the program.  If we assign a different list to d, it will have a different id, even though the values in the list are the same:

 >>> d = [3,2]       # note this new list has the same values as the old one
 >>> id(c) == id(d)
 False
 >>> id(d)
 139636640593824
 >>>

We can see that this other list is stored in a different location because the id() of the lists is different.  It turns out that this referencing behaviour is actually what you want to happen in most cases.  However, every so often you want your lists to be separate.  For that there is a special module called copy.  The copy module has a method (also called copy) which allows you to copy across the values of an object, rather than simply referencing (called “binding“) to an existing object:

>>> import copy
>>> d = copy.copy(c)
>>> d
[3, 2]
>>> c[0]=1
>>> d
[3, 2]
>>> c
[1, 2]

When you use copy.copy() the two objects will be separate and can be used independently.  Changes to one won’t show up in the other.  Where a compound object like a list or a dictionary has values which themselves are compound objects – for example a list where each entry in the list is itself a list – use the copy.deepcopy() method.   Depending on the complexity of your objects deepcopy() is not guaranteed to work (objects which refer to themselves somehow can cause a problem), but generally you will be fine.

What You’ll Need

To get up and running as a developer of Python on Windows, you’ll need a Mercurial client to download Python, update and create patches. You can use a command-line tool or you can get TortoiseHg, a shell GUI. Once you have that configured correctly, you can do a


hg clone http://hg.python.org/cpython


Or use Tortoise to check out the repository. This will get you the latest Python 3.x version. If you want to help with a maintenance release, then you’ll want to read the documentation. The last major tool you’ll need is a compiler and the one that’s needed for the latest Python is Microsoft Visual Studio 2010. Fortunately, you don’t need to purchase the whole thing. In fact, you can just get the Express version of Visual C++ 2010 and you’ll be good to go. Note that this tool is not lightweight and ends up taking over one gigabyte of space on disc. Sad, but true.

Compiling Python on Windows 7

As you may have guessed, we’ll be doing the compiling of Python on Windows 7. I have Windows XP too, but that OS is practically dead now, so I’m not going to cover it. I doubt it’s much different anyway. Regardless, according to the dev guide documentation, you need to go into the repo that you just created on your machine and go into the PCBuild folder. Then find the pcbuild.sln file and run it with your new Visual Studio C++ application. You may see a warning from it about how the Express version doesn’t support Solutions, but just ignore that. Once the project is loaded, go into the Debug menu and select Build Solution. Oddly enough, the official docs say to go into the Build menu, NOT the Debug menu, but my copy doesn’t have a Build menu to choose from.

When I ran the build I got the following result at the end:


========== Build: 20 succeeded, 8 failed, 0 up-to-date, 3 skipped ==========


Looking through the log, it looks like it was unable to find the header files for sqlite3 and tcl and it had some issues with the bzip lib. It also complained that I don’t have ActivePerl / openssh installed. However, it still compiled Python and I had a fresh python_d.exe file in my PCBuild folder that I could run. I ran it by double-clicking it, but you can also run it from within Visual Studio using F5 or by going to the Debug menu and clicking Start Debugging.

I think at this point, I’m ready to figure out how to create a patch. If I do, I may try patching that messed up documentation so that people won’t spend time looking for a non-existent menu. Then I’ll write an article about how to submit a patch and use the issue tracker system for Python.

Related Reading

• Python 101: Setting up Python on Windows
• A (brief) introduction to Python-Core development
• Contribution article
• Python Core Mentorship program

To install NumPy and SciPy for use with IronPython, follow the instructions here. After installation, NumPy works as expected.

There is one small gotcha with SciPy. To use SciPy with IronPython, start ipy with the command line argument -X:Frames. Then you can use SciPy as you would from CPython. For example.

c:\> ipy -X:Frames
>>> import scipy as sp
>>> sp.pi
3.141592653589793

Without the -X:Frames option you’ll get an error when you try to import scipy.

AttributeError: 'module' object has no attribute '_getframe'

According to this page,

The issue is that SciPy makes use of the CPython API for inspecting the current stack frame which IronPython doesn’t enable by default because of a small runtime performance hit. You can turn on this functionality by passing the command line argument “-X:Frames” to on the command line.

Well, if you’re using Mercurial or Git, you’re lucky because both provide built-in methods for doing this.

With Mercurial, use hg grep.

With Git, you can either use git grep in conjunction with git rev-list, or git log -S (more details in this SO thread).

What about Subversion, though? SVN, to the best of my knowledge, does not have this functionality built-in. Moreover, SVN’s design makes this task inherently slow because no revisions past the last one are actually kept on your machine (unless the repository is local) and you have to ask the server for each revision. That’s a lot of network traffic.

That said, if you’re willing to tolerate the slowness (and sometimes there’s no choice!), then the following script – svnrevgrep – makes it as simple as with Git or Mercurial:

import re, sys, subprocess

def run_command(cmd):
    """ Run shell command, return its stdout output.
    """
    return subprocess.check_output(cmd.split(), universal_newlines=True)

def svnrevgrep(filename, s):
    """ Go over all revisions of filename, checking if s can be found
        in them.
    """
    log = run_command('svn log ' + filename)
    for ver in re.findall('r\d+', log, flags=re.MULTILINE):
        cmd = 'svn cat -r %s %s' % (ver.rstrip('r'), filename)
        contents = run_command(cmd)
        print('%s: %s' % (ver, 'found' if re.search(s, contents)
                                       else 'not found'))
if __name__ == '__main__':
    if len(sys.argv) != 3:
        print('Usage: %s <path> <regex>' % sys.argv[0])
    else:
        svnrevgrep(sys.argv[1], sys.argv[2])

It basically goes over all revisions of the file starting with the most recent one and looks for the pattern.

Note that while one could imagine using some kind of binary searching to find the first revision in which the regex appears (or doesn’t), this won’t work in the general case because code sometimes is added, then deleted, then re-added, then deleted again (this happens when refactoring or when reverting problematic commits).

Finally, if you find yourself doing the above frequently for a given repository, you may be better off with:

git svn clone <path>
git grep <...>

Related posts:

1. Python development switches to Mercurial source control The official CPython core development team has finally switched from...
2. Migrating my personal projects to Mercurial Introduction My first acquaintance with version control was soon...

This is a fairly technical post talking about the structural changes I would like to see in CubicWeb's near future. Let's call that CubicWeb 4.0! It also drafts ideas on how to go from here to there. Draft, really. But that will eventually turn into a nice roadmap hopefully.

I'm now back from Copenhagen were I attended the mercurial 2.3 sprint with twenty other people. A huge amount of work was done in a very friendly atmosphere.

Regarding mercurial's core:

• Bookmark behaviour was improved to get closer to named branch's behaviour.
• Several performance improvements regarding branches and heads caches. The heads cache refactoring improves rebase performance on huge repository (thanks to Facebook and Atlassian).
• The concept I'm working on, Obsolete markers, was a highly discussed subject and is expected to get partly into the core in the near future. Thanks to my employer Logilab for paying me to work on this topic.

• General code cleanup and lock validation.

Regarding the bundled extension :

• Some fixes where made to progress which is now closer to getting into mercurial's core.
• Histedit and keyring extensions are scheduled to be shipped with mercurial.
• Some old and unmaintained extensions (children, hgtk) are now deprecated.
• The LargeFile extension got some new features (thanks to the folks from Unity3D)
• Rebase will use the --detach flag by default in the next release.

Regarding the project itself:

• We now have a running buildbot for Windows (in addition to the runnable test suite itself).
• We migrated from round-up to bugzilla.

Regarding other extensions:

• hgsubversion got a lot of bug fix (the harmonious image of the Google staff reviewing patches from the Facebook staff).
• Edlund released their repoman tool to manage forests of repositories used in multiple "solutions".
• A new multiundo extension was written (thanks to Edlund and Wolfram Research).

And I'm probably forgetting some stuff. Special thanks to Unity3D for hosting the sprint and providing power, network and food during these 3 days.

Background

Some weeks ago, because of a problem in the client library we are using, I had to switch the http library we are using on the PS3 from using non-blocking IO to blocking. Previously, we were were issuing all the non-blocking calls, the “select” and the tasklet blocking / scheduling on the main thread. This is similar to how gevent and other such libraries do things. Switching to blocking calls, however, meant doing things on worker threads.

The approach we took was to implement a small pool of pyton workers which could execute arbitrary jobs. A new utility function, stacklesslib.util.call_async() then performed the asynchronous call by dispatching it to a worker thread. The idea of an call_async() is to have a different tasklet execute the callable while the caller blocks on a channel. The return value, or error, is then propagated to the originating tasklet using that channel. Stackless channels can be used to communicate between threads too. And synchronizing threads in stackless is even more conveninent than regular Python because there is stackless.atomic, which not only prevents involuntary scheduling of tasklets, it also prevents automatic yielding of the GIL (cPython folks, take note!)

This worked well, and has been running for some time. The drawback to this approach, of course, is that we now need to keep python threads around, consuming stack space. And Python needs a lot of stack.

The problem

The only problem was, that there appeared to be a bug present. One of our developers complained that sometimes, during long downloads, the http download function would return None, rather than the expected string chunk.

Now, this problem was hard to reproduce. It required a specific setup and geolocation was also an issue. This developer is in California, using servers in London. Hence, there ensued a somewhat prolonged interaction (hindered by badly overlapping time-zones) where I would provide him with modified .py files with instrumentation, and he would provide me with logs. We quickly determined, to my dismay, that apparently, sometimes a string was turning into None, while in transit trough a channel.send() to a channel.receive(). This was most distressing. Particularly because the channel in question was transporting data between threads and this particular functionality of stackless has not been as heavily used as the rest.

Tracking it down

So, I suspected a race condition of some sorts. But a careful review of the channel code and the scheduling code presented no obvious candidates. Also, the somehwat unpopular GIL was being used throughout, which if done correctly ensures that things work as expected.

To cut a long story short, by a lucky coincidence I managed to reproduce a different manifestation of the problem. In some cases, a simple interaction with a local HTTP server would cause this to happen.

When a channel sends data between tasklets, it is temporarily stored on the target tasklet’s “tempval” attribute. When the target wakes up, this is then taken and returned as the result from the “receive()” call. I was able to establish that after sending the data, the target tasklet did indeed hold the correct string value in its “tempval” attribute. I then needed to find out where and why it was disappearing from that place.

By adding instrumentation code to the stackless core, I established that this was happening in the last line of the following snippet:

PyObject *
slp_run_tasklet(void)
{
    PyThreadState *ts = PyThreadState_GET();
    PyObject *retval;

    if ( (ts->st.main == NULL) && initialize_main_and_current()) {
        ts->frame = NULL;
        return NULL;
    }

    TASKLET_CLAIMVAL(ts->st.current, &retval);

By setting a breakpoint, I was able to see that I was in the top level part of the “continue” bit of the “stack spilling” code

Stack spilling is a feature of stackless where the stack slicing mechanism is used to recycle a deep callstack. When it detects that the stack has grown beyond a certain limit, it is stored away, and a hard switch is done to the top again, where it continues its downwards crawl. This can help conserve stack address space, particularly on threads where the stack cannot grow dynamically.

So, something wrong with stack spilling, then.  But even so, this was unexpected. Why was stack spilling happening when data was being transmitted across a channel? Stack spilling normally occurs only when nesting regular .py code and other such things.

By setting a breakpoint at the right place, where the stack spilling code was being invoked, I finally arrived at this callstack:

Type Function
PyObject* slp_eval_frame_newstack(PyFrameObject* f, int exc, PyObject* retval)
PyObject* PyEval_EvalFrameEx_slp(PyFrameObject* f, int throwflag, PyObject* retval)
PyObject* slp_frame_dispatch(PyFrameObject* f, PyFrameObject* stopframe, int exc, PyObject* retval)
PyObject* PyEval_EvalCodeEx(PyCodeObject* co, PyObject* globals, PyObject* locals, PyObject** args, int argcount, PyObject** kws, int kwcount, PyObject** defs, int defcount, PyObject* closure)
PyObject* function_call(PyObject* func, PyObject* arg, PyObject* kw)
PyObject* PyObject_Call(PyObject* func, PyObject* arg, PyObject* kw)
PyObject* PyObject_CallFunctionObjArgs(PyObject* callable)
void PyObject_ClearWeakRefs(PyObject* object)
void tasklet_dealloc(PyTaskletObject* t)
void subtype_dealloc(PyObject* self)
int slp_transfer(PyCStackObject** cstprev, PyCStackObject* cst, PyTaskletObject* prev)
PyObject* slp_schedule_task(PyTaskletObject* prev, PyTaskletObject* next, int stackless, int* did_switch)
PyObject* generic_channel_action(PyChannelObject* self, PyObject* arg, int dir, int stackless)
PyObject* impl_channel_receive(PyChannelObject* self)
PyObject* call_function(PyObject*** pp_stack, int oparg)

Notice the “subtype_dealloc”. This callstack indicates that in the channel receive code, after the hard switch back to the target tasklet, a Py_DECREF was causing side effects, which again caused stack spilling to occur. The place was this, in slp_transfer():

/* release any objects that needed to wait until after the switch. */
Py_CLEAR(ts->st.del_post_switch);

This is code that does cleanup after tasklet switch, such as releasing the last remaining reference of the previous tasklet.

So, the bug was clear then. It was twofold:

1. A Py_CLEAR() after switching was not careful enough to store the current tasklet’s “tempval” out of harms way of any side-effects a Py_DECREF() might cause, and
2. Stack slicing itself, when it happened, clobbered the current tasklet’s “tempval”

The bug was subsequently fixed by repairing stack spilling and spiriting “tempval” away during the Py_CLEAR() call.

Post mortem

The inter-thread communication turned out to be a red herring. The problem was caused by an unfortunate juxtaposition of channel communication, tasklet deletion, and stack spilling.
But why had we not seen this before? I think it is largely due to the fact that stack spilling only rarely comes into play on regular platforms. On the PS3, we deliberately set the threshold low to conserve memory space. This is also not the first stack-spilling related bug we have seen on the PS3, but the first one for two years. Hopefully it will be the last.

Since this morning, the fix is in the stackless repository at http://hg.python.org/stackless

What I didn't describe was the mental effort it took to develop this program. This is the second time I've written code to find the multiple structure MCS, and both times it took a couple of months and put my head in a very strange place. You would think the second time is easier, but it means that I spent more time adding features and doing things that my first version couldn't begin to handle. (Did someone just mutter "second system effect"? Pshaw!)

This time too I had a better understanding of the development process. I think I know why it's so much harder than most of the software I develop.

In this essay, I reflect on some of the reasons why this is a hard problem to test and I consider how unit tests, or any other incremental-based testing approach, are not well-suited to a certain class of complex algorithm development. While unit tests can provide a basic sanity check during development, they fail to provide the test coverage which one might expect from applying them to algorithmically simpler modules. Further, development methodologies built primarily on unit tests, like test-first style test-driven development, aren't that applicable. Instead, other methods, like system testing and a deep understanding of the algorithm and possible problems, are required. This strengthens my view, explored in Problems with TDD that "TDD is a weak method for developing those tests of confidence."

How would you implement an MCS algorithm?

Think about the problem for a moment. How would you write an algorithm to find the largest common subgraph between two graphs?

Take your time. This really is a hard problem. I dug up some of the early papers from the Journal of Computer Documentation. People ended up simplifying the problem by, for example, not supporting rings.

Finished thinking?

You probably came up with a graph walking algorithm which tries different match pairings and uses backtracking or lazy evaluation to search all of graph space. The more mathemtically inlined might have converted this into a maximum clique algorithm.

In both cases there are a lot of arbitrary decisions to make. Which pairings should you investigate first? Are there times when you cann prune the search space? Did you make any assumptions about the nature of the chemical graph (eg, that it's topologically a planar graph)?

How would you test your MCS algorithm?

Back in the late 1990s, I almost never wrote automated tests, and never wrote an extensive test suite. Nowadays I'm pretty thorough, and use coverage-guided techniques to get good, extensive tests through some semi-permanent API layer that will allow me to refactor the internals without changing the tests.

I couldn't do that here.

There are a lot of heuristics, and some of them are only triggered under unusual circumstances, after a bunch of combinitorial possibilities. Outside of a few minor components, I couldn't figure out how to write unit tests for the code.

I ended up pushing most of the testing into validation testing of the complete system, which meant I wrote some 1,000+ lines of code without strong testing. Moreover, I used a new approach to the MCS problem, so the algorithm I was working on doesn't have the track record of the standard clique-based or backtracking approaches.

So stress factors included not knowing if the algorithm would work, and not being able to develop enough test cases to provide good validation during development.

As a rule of thumb, it's easiest to fix bugs which are caught early. Unit tests and evolutionary prototyping are two of the techniques that people use to tighten the feedback loop between specification, implementation, and valiation. I think another stress factor is propotional to the size of the feedback loop.

What testing did I do?

I mean, I did have tests during development. I came up with a few examples by hand, I did a substructure search of a large data set and I verified that the MCS code found that substructure, but I know that's not enough tests. I know this because after six weeks of development and over 1000 lines of code, I spent another three weeks doing a large amount of post-development testing, and found several bugs. In the process of writing this essay I also found that four days of that development work ended up making things slower, so I'll have to remove it.

I did most of my tests based on the ChEMBL data: 10,000 random pairs of structures, the k=2, k=10, and k=100 nearest neighbors, and the k<=100 neighbors with similarities of at least 0.95, 0.9, and 0.8. I also did, at the end, tests based on the ChEBI structure ontology. There were easily 20,000 different machine-generated test cases, although in most cases I didn't know the expected results beforehand.

What bugs did I find?

What bugs did I find? I think it's educational to characterize a few of them.

Typo caught by an assertion check

One of the simplest bugs was a poorly formatted string. I used "%d" when I should have used ""%%%d". A bit of jargon for those in the know; I generate SMARTS strings for the intermedate subgraphs. If there are more than 9 open rings then the syntax goes from a single digit closure number to a closure number like "%10". I forgot to include the '%' for that case, and probably because the '%' was already there for the format string.

This wasn't triggered by my random-pairs test nor my various similarity-search based tests. Only when I ran through the ChEBI data, did I get an assertion failure when RDKit refused to accept my SMARTS string. That was the first time where I had a SMARTS with 10 unclosed rings.

As it happens, this error could have been caught by a unit testing, as some people practice unit tests. It's a four line function which takes a string and a number. Testing it is trivial. I didn't test it because I feel that testing directly against internal functions inhibits refactoring. I prefer to test against higher-level, more stable APIs.

My view is that it's usually easy to come up with high-level test cases which trigger a certain function call. But not in this case. The MCS search algorithm, while deterministic, uses so many arbitrarily defined values that I couldn't on my own come up with a test case with at least 10 open ring closures. And even if I did, a new search heuristic might change things so that only, say, 7 open ring closures were needed.

I felt that my system testing and the assertion check would be enough to identify if there was a problem, and it did. A low-level unit test might have helped, especially as I still don't have a specific test for that failure.

I think the right thing to do is add that failure as a specific test case, and use monkey-patching to insert wrong code for a repeat of that test case. The first one tests that the code is correct, and the second tests that the test case is still exercising the code under test.

Cross-validation testing

The first MCS papers are about as old as I am. Many people have written implementions, although relatively few are are available to me both at no cost and with no prohibitions on using it to develop a new MCS algorithm. (Some commercial companies don't like people using their software to write new software which is competitive to it, or even to use their software for benchmark comparisons.)

I tested pairs of structures against SMSD and I did more extenstive tests against Indigo's MCS implementation.

This is cross-validation testing. It's a relatively rare technique because the cost of producing multiple identical implementations usually isn't worth the benefits. Even here the results aren't exactly identical because of differences in how the toolkits perceive chemistry, and more specifically, aromaticity. I ended up spending a lot of investigation time staring at cases with different answers and trying to figure out if it was a chemistry problem or an MCS algorithm implementation problem.

I found the SMSD had a bug in one of its options, which I reported back to the author. The code had been fixed internally but not pushed to the outside world. Its default mode and fmcs matched quite well, except for a couple of chemistry differences. The new version is out now - I need to test it again.

The only problem I found in the Indigo code was a part of their setup code which didn't check the timeout. That's also been fixed after I reported it.

The cross-validation with Indigo found problems in my code. For example, I was often getting smaller MCSes then Indigo. After looking at them, I figured out that my code didn't correctly handle the case when a molecule was fragmented after a bond was removed because its bondtype wasn't in all of the structures.

Why didn't my hand-written test cases find it? None of them had a case where there was a bond in the "middle" of a structure chosen as the reference structure, and where the MCS was not in the first fragment.

My code usually got the right answer when using highly similar structures, for obvious reasons. It was only the random pair testing where the problem really stood out.

Could I create a simple unit test for this error? Perhaps, but it's not easy. I don't know which of the two fragments will be first - it depends on so many arbitrary decisions which could change as the algorithm is improved. The only test I can think of for this was to generate a diverse set of tests, make sure some fail if the code isn't implemented correctly, record the results, and make sure that future tests never find a worse (or better?) test case.

Bad heuristic to determine the maximum possible subgraph growth

Most of the MCS implementation is heuristics. There's a branch and bound search, there's subgraph canonicalization to reduce the number of substructure tests, and so on. Each of these is supposed to help make the code faster.

One of the tests takes the current subgraph and the list of "outgoing" bonds to see how much of the remainder of the graph is accessible for growth from the current subgraph. The rest of the molecule might not be accessible because it's on another fragment, but it also might not be accessible due to an earlier decision to exclude certain parts of the molecule from future growth. (My algorithm tests all subgraphs which include a given bond, then all subgraph which don't include the given bond.)

It took a couple of days to think of the algorithm, write the code, and get it working. I then did the timing tests to find out it was 1% faster, to get the same answers.

Does that mean my code worked? I only had a suspicion that the new algorithm should be faster. Perhaps the overhead of searching for the accessible atoms was too costly?

After looking at my implementation - a lot - I finally realized that I told the algorithm to exclude the subgraph from consideration for growth but had forgotten to also exclude the set of previously excluded bonds from consideration. With two changed lines, the overall performance doubled for the random-pairs case. Most of the time it's faster and sometimes its slower, so it takes an aggregate of tests to measure this correctly.

I don't think this heuristic could have been written as a unit test. No, I take that back. It could have, but it would have required some careful thought to set up. Not only was this part of the code not "built for test", but setting up the right conditions requires a mental understanding of the problem which I know I didn't have when I was doing the development.

BTW, I am not saying that unit tests can't be used to measure performance. Some algorithms have simple timing characteristics where it's easy to say that the code must complete by a given time, or that it must be at least twice as fast as a reference implementation. Occasionally these tests will hiccup due to unusual loads on the test machine, but not usually enough to be a problem.

Indeed, the fmcs code has some unit tests for the timeout code. I found a pair of structures which takes over 10 seconds to find the MCS, I set the timeout to 0.1 second, and assert that no more than 0.5 seconds elapsed during the function call. (I don't require a high precision for this code.) I do worry though that future changes to the MCS search code might speed things up by a faster of 20. On the other hand, the effect of broken timeout code is very obvious when running the validation suite, so I'm not going to worry about it.

As is widely acknowledged, this stretches the idea of a "unit test." Unit tests are supposed to be fast; preferably several hundred or more per second. The goal is that these tests run often, perhaps even after every save. Stick in a few 0.1 second timeouts into the system and it bogs everything down, which discourages people from running the tests so often.

But let's go back to this new code. On average, over a large number of tests, the performance is 50% faster. What's a good test case? Can I pick one structure or a small set of structures? Does the speedup occur only when there are large molecules? Only for molecules with lots of internal symmetry? Only for those which are easily fragmented?

Only now, when writing this essay, did I find good test cases. When I use a 10 second timeout using the k=10 nearest neighbors tests, I get 8 timeouts in the first 32 records using the old algorithm, and only 1 timeout using the new algorithm. The time for the very first test goes from over a minute (I finally gave up) using the old algorithm to 0.08 seconds using the new one.

Obviously (in retrospect) the right solution is to pull out a couple of those from my validation suite and turn them into test cases.

I never would have come up with these test cases before implementing the new code - up until ten minutes ago I thought that the new code only added a factor of two the code, not possible factors of 1,000!

Canonicalization: an in-depth analysis

I'm not finished with testing. I haven't fully characterized all the parts of the implementation. For that matter, there are heuristics I can still add. Here I'll describe what I did to analyze subgraph SMARTS canonicalization. I conclude by finding that while it works, it slows down the code and several days of development work ought to be removed.

My MCS algorithm enumerates subgraph of one structure, converts that into a SMARTS string, and tests if the pattern exists in other structures.

There are many alternate ways to make a SMARTS string from a subgraph. Given the linear chain of C O N, there's the reasonable CON or NOC variations, the more confusing O(C)N and O(N)C, and crazy variations like C1.N1O and O%987.N7.C%98. It's easy to make the non-insane versions be generating a spanning tree. The question is where to start the search and how to handle branches.

I believe that there are many duplicate patterns in a query. For example, a benzene ring will generate many "ccc" queries. I can minimize the number of substructure matches by caching earlier SMARTS match results. But caching doesn't work if sometimes I get a CON and sometimes I get a NOC. The solution is to generate a "canonical" SMARTS for the subgraph - a unique representation for all of the variations.

I implemented the CANGEN algorithm from SMILES. 2. Algorithm for generation of unique SMILES notation by Weininger, Weininger, and Weininger J. Chem. Inf. Comput. Sci., 1989, 29 (2), pp 97-101 in order to choose a canonical SMARTS. (But see Assigning Unique Keys to Chemical Compounds for Data Integration: Some Interesting Counter Examples, Neglur, Grossman, and Liu, 2nd International Workshop on Data Integration in the Life Sciences (DILS 2005), La Jolla.)

Canonicalization is notoriously hard to get right. It too has a lot of tricky corner cases. For example, in the late 1990s Daylight tracked down a reported bug in their canonicalization implementation. They algorithm requires a stable sort algorithm, but they used an unstable sort. Their own internal testing never found the problem - it was a customer who found the Solaris and IRIX boxes occasionaly returned different values.

There could be similar bugs in my canonicalization algorithm. I don't know. The usual solution is to generate large numbers of test cases, randomize the order of the atoms and bonds, and verify that all of them are the same. Here too it may be that only certain rare topologies, or rare bond patterns, triggers an error. I'm not convinced that I could come up with the right set of test cases for it.

What makes it harder is that the underlying search algorithm doesn't need a canonical SMARTS, so a canonicalization failure doesn't ever show up in the output. I could have an infrequent bug and not notice it! Canonicalization is only there for performance reasons, but my implementation is in Python, while the substructure matching is in C++, so it might even be that the canonicalization time might be more than the time saved.

I can make that a bit more nuanced. Canonical SMARTS generation has three phases: 1) initial ranking, 2) canonicalization/tie-breaking, and 3) SMARTS generation. I can disable the canonicalization step and get a "semi-canonical" SMARTS (my initial ranking is more like a circular fingerprint of diameter 3 than the atom characteristic from Weininger et al.). I can also disable caching. Both of these are doable with only a couple of changes to the code.

This leads to three cases: canonical SMARTS with caching, semi-canonical SMARTS with caching, and semi-canonical SMARTS without caching. (There should be a fourth step, which is arbitrary SMARTS, but that requires more extensive changes.)

What I want to understand most is the effect of canonicalization on my code. Again, I have to resort to aggregate timings and statistics across a set of benchmarks. I used the ChEMBL-13 data set and generated various benchmarks. One is based on computing the MCS between 10,000 pairs selected at random, another contains the k=2, k=10, and k=100 nearest neighbors of randomly chosen fingerprints (timing 500 data sets each time), and the last is the total of 500 tests of the k<=100 compounds with Tanimoto score of at least 0.95 to randomly selected fingerprints. My results (times are reproducible to within a few percent) include the number of unique SMARTS (canonical or non-canonical) generated and the total number of substructure tests ("SS") which were carried out.

Random pair timings		# unique SMARTS	# SS tests
no cache	Total: 10000/467.3s (21.4/s) Complete: 9997/437.3s (22.9/s) Incomplete: 3/30.0s	921666	1339292
semi-canonical	Total: 10000/421.3s (23.7/s) Complete: 9997/391.3s (25.6/s) Incomplete: 3/30.0s	921666	921666
canonical	Total: 10000/442.1s (22.6/s) Complete: 9997/412.0s (24.3/s) Incomplete: 3/30.0s	709636	709636
k=2 nearest neigbhors		# unique SMARTS	# SS tests
no cache	Total: 500/287.3s (1.7/s) Complete: 484/127.1s (3.8/s) Incomplete: 16/160.2s	264057	320238
semi-canonical	Total: 500/276.7s (1.8/s) Complete: 484/116.6s (4.2/s) Incomplete: 16/160.2s	264057	264057
canonical	Total: 500/298.6s (1.7/s) Complete: 483/128.4s (3.8/s) Incomplete: 17/170.2s	186682	186682
k=10 nearest neigbhors		# unique SMARTS	# SS tests
no cache	Total: 500/520.5s (1.0/s) Complete: 471/230.0s (2.0/s) Incomplete: 29/290.5s	447648	3525563
semi-canonical	Total: 500/490.1s (1.0/s) Complete: 472/209.5s (2.3/s) Incomplete: 28/280.6s	447648	2872040
canonical	Total: 500/509.3s (1.0/s) Complete: 471/218.7s (2.2/s) Incomplete: 29/290.6s	330010	2109332
k=100 nearest neigbhors		# unique SMARTS	# SS tests
no cache	Total: 500/414.4s (1.2/s) Complete: 486/271.4s (1.8/s) Incomplete: 14/143.0s	128781	9932263
semi-canonical	Total: 500/363.5s (1.4/s) Complete: 487/230.8s (2.1/s) Incomplete: 13/132.7s	128781	7456877
canonical	Total: 500/361.8s (1.4/s) Complete: 488/239.5s (2.0/s) Incomplete: 12/122.4s	107648	6196764
k<=100 at or above Tanimoto threshold of 0.95		# unique SMARTS	# SS tests
no cache	Total: 500/642.1s (0.8/s) Complete: 458/220.3s (2.1/s) Incomplete: 42/421.9s	468661	4388074
semi-canonical	Total: 500/624.5s (0.8/s) Complete: 461/232.8s (2.0/s) Incomplete: 39/391.7s	468661	3828813
canonical	Total: 500/640.9s (0.8/s) Complete: 460/239.2s (1.9/s) Incomplete: 40/401.8s	364802	3222366

Whew! That's a lot of data to throw at you. Please believe me when I say that it took two days to generate correctly. BTW, "complete" means that the MCS search algorithm went to completion, the "incomplete" means that it timed out - here after 10 seconds - and gave a partial solution.

What does this tell me? Obviously, caching is good. As expected, the "semi-canonical" solution is always better than the "no cache" case, and the canonicalization always reduces the number of substructures and substructure tests. This means the canonicalization code is doing something right.

Unfortunately, it seems like canonicalization has a big performance impact. In the pairwise tests I do 922,000 canonicalizations in Python to save 212,000 substructure tests, and I lose 21 seconds in doing so. For the k=100 nearest neighbor benchmark, which is the only one where the canonicalization code saves time, I do 129,000 canonicalizations to save 1,260,000 comparisons, and gain about 2 seconds. This suggests that each canonicalization takes as long as 10 substructure matches, and that RDKit can do 60,000 SMARTS matches per second. A quick checks using one SMARTS gives 100,000 SMARTS matches per second, which helps validate this estimate.

This means I should take out the canonicalization until it can be implemented in C++. Granted, there may be a bug in the code, like there was with an earlier hueristic. But there would have to be an order of magnitude performance increase for this to be effective, and I don't think that's likely.

Unit tests aren't enough to drive development

Which leads me back to my thesis. It would have been possible to develop a few more unit tests for the canonicalization code. There would have been some extra scaffolding in order to do that, but that's a minor cost. I suspect that many of the tests would be very implementation-dependent, and tied to specific internal function calls. I don't like this because it means that a re-implementation of this internal component in C++, which submerges many of the internal helper functions and places them out of reach of Python-based unit tests, would cause the unit tests to be un-runnable. And I am certain that those unit tests would not be rigorous enough to be confident that the code was working correctly.

Think about the question "should I write this code in Python or C++?". It's a development-time question. Test Driven Development (TDD) is a development methodology which uses unit tests to help make development-time decisions. I think TDD is most helpful when used to establish the minimum requirements for a project.

I don't see how TDD is helpful here. I can't think of any unit tests which would guide this development decision. Sometimes you can write a "spike solution" (also called a "prototype"):

A spike solution is a very simple program to explore potential solutions. Build the spike to only addresses the problem under examination and ignore all other concerns.

If not TDD, what methodology do I use?

Read Peter Seibel's Unit testing in Coders at Work. The author interviewed various famous programmers and learned more about how they tested their software. You may also want to read the related comments on Hacker News.

Seibel recounts that Joshua Bloch "wrote a monstrous 'basher'" which found "that occasionally, just occasionally, the basher would fail its consistency check." Bloch then wrote tests to pin down the failure, eventually finding the fault in a system mutex. Note that under normal unit test development you assume that your underlying components are correct, so this isn't something you would normally code for.

Seibel also reports that others assemble a large amount of test cases and uses that to check their code. This is of course the technique I used for the MCS problem.

It feels as trite as saying that the secret to losing weight is diet and exercise, but my method for programming is to understand the problem, implement it, and test it until you are confident that it's good enough. That knowledge of how to do that comes from experience, practice, reflection, and discussion.

For those who scoff and call this "big design up front," I merely point you to earlier parts of this essay. When I started this code I had a rough idea that it would work. I had previously implemented substructure enumeration, and the Weininger et al. canonicalization algorithm and SMARTS generation, so I knew that the components were possible, but I didn't know how they went together. Instead, I let the code development itself guide me. I thought some, I implemented code, I "ran the code in my head", I reflected on what the tricky parts were, I thought about how to make them more clear, and I tried various was to improve the code.

That was a lot to keep in my head, I wasn't sure if the result would be fast enough, and I had no way good way to test its usefulness until most of the code was in place. No wonder it was mentally taxing!

I believe tests - including unit tests implemented during development - are important. I don't believe that unit tests are good enough to guide complex algorithm development. However, most programming (over 95%?) is not complex algorithm development; complicated? yes, but not complex.

I once jokingly said that TDD is not useful if there's two or more embedded for loops. That's a bit of a simplification, but not far from my feelings. Some classes of problems, like complex algorithm development and security analysis, require a different attitude towards programming, emphasizing "what can go wrong?" and "how can I improve my confidence that the code is working correctly?" It's my view that this doubt-based philosophical attitude is missing from most discussions of software development practices.

And perhaps as fundamental, when I have to be that critical of my own work and probe it for failures and be open to the possibility of nasty gotchas lurking in the deep dark corners, then some of that doubt passes over into my personal life. I empathize with the idea of not living in that "strange place" all the time, and I can see why this isn't a common attitude.

Comments

This essay is meant to be a thoughtful reflection on the difficulties of programming, using the MCS problem to provide specific structure. If you want to leave comments about the MCS portion then use the MCS comment site. Otherwise, leave a comment on testing hard algorithms.

Good old log files are still the most reliable, versatile, and useful sources of information.
When they are also human-readable and easy to use with standard Unix tools like tail and grep they are even more useful.

We all know that especially in time of stress when things with a system are going south having a quick and easy way to grep for useful information is critical. The purpose of a small Python utility Logsna is to provide a sane log output format that makes some grepping a bit easier.

Logsna offers a custom formatter class logsna.Formatter that can be used in a logging config file, for example:

# sanefmt.py
import logging
import logging.config
from StringIO import StringIO

CONFIG = """\
[loggers]
keys=root

[handlers]
keys=console

[handler_console]
class=logging.StreamHandler
args=(sys.stderr,)
formatter=sane

[formatters]
keys=sane

[logger_root]
level=DEBUG
handlers=console

# Our custom formatter class
[formatter_sane]
class=logsna.Formatter
"""

config = StringIO(CONFIG)
logging.config.fileConfig(config)

log = logging.getLogger('mylogger.component1')

log.debug('debug message')
log.info('info message')
log.warning('warning message')
log.critical('critical message')
try:
    1 / 0
except:
    log.exception('Houston we have a problem')

The Log Format

Here is an output from the above program:

DEBUG    [2012-05-21 01:59:23,686] mylogger.component1: debug message
INFO     [2012-05-21 01:59:23,686] mylogger.component1: info message
WARNING  [2012-05-21 01:59:23,686] mylogger.component1: warning message
CRITICAL [2012-05-21 01:59:23,686] mylogger.component1: critical message
ERROR    [2012-05-21 01:59:23,686] mylogger.component1: Houston we have a problem
! Traceback (most recent call last):
!   File "/home/alienoid/python/sanefmt.py", line 67, in
!     1 / 0
! ZeroDivisionError: integer division or modulo by zero

The Log Format Notes

- All timestamps are in ISO8601 and UTC format

- To grep for messages of a specific level

$ tail -f sanefmt.log | grep '^INFO'

- To grep for messages from a particular logger

$ tail -f sanefmt.log | grep 'component1:'

- To pull out full exception tracebacks with a corresponding log message

$ tail -f sanefmt.log | grep -B 1 '^\!'

Installation

$ [sudo] pip install logsna

We just released LFS 0.6.17. This is a yet another bugfix release of the 0.6 branch.

I only vaguely remember my first encounter with a parser generator which must by dated back to the late 1990s. I guess it was Spark by John Aycock, an Early parser. What puzzled me back then was the need to be explicit down to the character level.  Regular expression engines, albeit cryptic, were a revelation because one could specify the structural information one needed and match the rest using a ‘.’ which is the wildcard pattern that matches any character.

I came back to Any in the Trail parser generator lately. I was motivated by writing a reversible C preprocessor. Unlike conventional C preprocessors which are used in the compilation chain of C code, a reversible C preprocessor can used to refactor C code, while retaining the preprocessor directives and the macro calls. This is basically done by storing the #define directive along with the code to be substituted and the substitution. The substituted code and the substitution are exchanged after the refactoring step, such that it looks like no substitution happened at all.

A comprehensive C preprocessor grammar can be found on the following MSDN site. What is most interesting to us are the following two EBNF productions:

# define identifier[( identifier_opt, ... , identifier_opt )] token-string_opt

token-string :

String of tokens 

The String of tokens phrase this is Any+.

Bon appetite

Suppose one defines two macros

#define min(a,b) ((a)<(b)?(a):(b))

#define max(a,b) ((a)<(b)?(b):(a))

Obviously the defining string of the min macro can be recognized using token-string but how can we prevent that token-string eats the max macro as well? Once in motion token-string has a sound appetite and will eat the rest. The solution to this problem in case of regular expressions is to make Any non-greedy. The non-greediness can easily be expressed using the following requirement:

If S | Any is a pattern with S!=Any. If S can match a character, S will be preferred over Any.

In the production rule

R: ... Any* S ...

R: ... (Any* | S) ...

Non greediness in grammars

Grammars are more complicated than regular expressions and we have to take more care about our greediness rules. To illustrate some of the problems we take a look on an example

R: A B | C
A: a Any*
B: b
C: c

Detecting the trailing Any in A is easy though. We solve the follow/first conflict with a trailing Any by embedding A into R. Embedding strategies are the centerpiece of Trail and they shall not be recapitulated here. Just so much: embedding A in R doesn’t destroy any information relevant for parsing. If A has been embedded Any* will be delimited by B to the right and we can safely apply R without the danger of Any consuming a token ‘b’.

Eventually we have to re-apply our embedding strategy: if A is a rule with a trailing Any and A is embedded in B and B has a trailing Any after this embedding  then B will be embedded wherever possible.

A short experiment

Here is a mini-Python grammar is used to detect Python class definitions.

file_input: (NEWLINE | stmt)* ENDMARKER
classdef: 'class' NAME ['(' Any+ ')'] ':' suite
suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT
simple_stmt: Any* NEWLINE
stmt: simple_stmt | compound_stmt
compound_stmt: classdef | Any* suite

Unlike a real Python grammar it is fairly easy to build. All rules are taken from the Python 2.7 grammar but only file_input, suite and stmt remained unchanged. In all other cases we have replaced terminal/non-terminal information that isn’t needed by Any.

The Problem

It isn’t easy to start contributing to FOSS. Some projects seem even not to be particularly interested in new contributors – because everyone wants to contribute for fame and glory. Others could use help but aren’t eager to get new patches because of ample amounts of code rotting in the bug tracker. But there are plenty of projects that want you to help.

And yes, even CPython — i.e. the most widespread Python implementation as of 2012, I’ll call it just “Python” henceforth — needs your help! That’s the reason it’s active in its outreach to new contributors. Please note I said “contributors”, you don’t have to be a programmer – or do programmer’s work – to help.

Python core development has always been a welcoming place. But Jesse tried to push things even farther. And he succeeded at that – it’s easily one of the best places to hang out with people way smarter than you (or at least me) without being patronized.

But it’s hard to get started with new code and often even harder to get started with a new community. So what’s the most important personal attribute to get into a high profile open source project like Python, Django or Twisted? Hint: it’s like with everything important in life you can’t buy.

The Solution

Perseverance.

People sense that but don’t believe it. It’s just like kids don’t believe they’ll ever become adults and their parents were where they are now at some point in prehistoric times. And just like that, the now famous CPython developers once started submitting patches to a tracker and prayed for someone to notice.

I’m in no way famous but I think that’s exactly what it makes easier for you to relate to me.

Please compare the dates between my first patch (2011-08-08) to the bug tracker and the day I got my push rights (2012-05-14): nine months. What’s even worse, my first real code patch rotted for four months in the tracker. I’ll be honest with you: I walked away from Python core development at this point.

“If they don’t want my help, I’ll look elsewhere.”

Fortunately, Antoine gave me my “christmas presents” and committed it at last out of nowhere. That made me come back and I’m really happy about that. But was it really necessary for me to leave my ambitions? Not really. I’ll try to draw a picture of the real situation.

The problem is: the delay didn’t indicate that Python isn’t interested in my help. It proved it needs my help very much.

Although there are lots of committers, only few of them are active (try running hg churn -c -d '2012 to 2013' if you don’t believe me). Committing code to a project like Python is different than dabbling on some small project. We have a high responsibility to not break stuff livelihoods depend on. Someone who commits your code has to fully understand what it’s doing and takes responsibility for your patch (although breaking stuff isn’t really a crime as long as it breaks loudly, that’s what we have buildbots for).

So, logically few active committers means few checkins means long delays on patches.

So in the beginning, you really need some patience. It also helps to hang out and socialize on Freenode’s #python-dev or the Python mailing lists (although I’m rather shy on posting on mailing lists myself; it doesn’t feel good to ask basic stuff that gets archived forever). Your also gain good karma but reviewing code of others and generally being active on the bug tracker. Once some of your patches have been committed, other developers will be much more open to look at your code. We’re all human in the end and you would also rather help out a friend, no?

And that would have been the better way for instead of retiring. While I was on IRC, I wasn’t really active on the tracker building up reputation. After my first patches were committed and I started triaging and reviewing, people even started asking me whether I need to have something reviewed.

When you arrive at this point, you’re over the hump. Contributing becomes much less of a insular patience game and much more of a social experience. You’re part of a community hacking on something cool – and while there’ll always be friction where people gather and you’ll still have to wait for reviews – it’s a lot of fun.

And one day – out of the blue – you’ll get extended privileges offered. At this point it’s important to keep your cool; too many people started doing way too much, burning out and stopping doing anything at all.

What Python Needs Most

We need triagers that weed through tickets – new and old – and help to close them. Be it by reviewing code, checking spelling, grammar and wording, trying to reproduce the problems – we’re especially in need of capable Windows users – or just pinging stale tickets and adding suitable developers to the watch lists. There’s a lot to do and the current team is glad to be able keep up with new stuff, but there are 1,500 and growing tickets with patches open – and I’d like very much to change that.

Let me stress at this point that it’s unlikely that you get commit rights if you just add own patches and wait for them to be committed. It’s expected from you to take part at the whole development process.

So my foremost priority for the next months is doing exactly that: cut back on own coding and weed through tickets and make people happy by getting simple stuff merged or at least noticed. I know from personal experience that ignored patches are worse than rejected ones.

But I could use the help of people like you, who speak Python or English and are willing to learn Python’s development process! Get an account on the tracker, familiarize yourself with our triaging process and start triaging! Meanwhile, feel free to ask your questions on Freenode’s #python-dev or our core-mentorship mailing list.

Your efforts will be noticed. And nothing’s as fun as breaking all buildbots with one commit. :)

1. modifying the model layer, so that the field becomes available as an attribute on retrieved objects, and can be queried against etc.
2. creating a database migration script that can be run immediately on the development box, and later for staging and production.
3. running the migration against the development DB.
4. updating any admin screens for editing the field.
5. checking the changes and scripts into source control.
6. deploying - including pushing source code and running migration scripts etc.

Using Django, from a cold start (no editor/IDE open), this just took me 1 minute 45 seconds of work for steps 1 - 4, and an additional 45 seconds waiting for step 5, total 2 minutes 30 seconds, and I wasn't rushing.

Step 1 is a one line code addition. Pretty much everything else can and should be generated automatically.

Step 2 is taken care of by a one line command using South, as is step 3 and the database part of step 6 (which is run de-rigueur from my deployment scripts).

Step 4 is taken care of by Django's admin, which introspects the model and generates the right form for you.

This is one of the reasons I love Django. It's not so much the time it saves, although that is pretty awesome, it's the tedium it saves.

This is also one of the reasons I'm not very tempted by schema-less or schema-light databases, because with Django a nice strict schema brings so little administrative overhead. I was going to have to add something about the change to the model anyway, even if it was only documentation, and having done that in one place, the other additional changes required by a relational DB with strong schema placed virtually no burden on me.

(Of course, things could be more complex on bigger apps, especially if the table is large or sharded. But then again, there's no reason why rolling out your DB change shouldn't be just as automated - it's only the 'waiting' stage that has to take longer for a simple change like adding a column. If the coding/work part is taking much longer than the above example, your tools probably need fixing or replacing.)

Twisted is excited to be supporting 4 full-time summer interns from around the world through 2 internship programs this summer.

Google Summer of Code internships

Expanded Endpoints Support, by Ashwini Oruganti (IRC nick ashfall)

Ashwini joins us from the Manipal Institute of Technology in Manipal, India. She has already worked on and closed out a number of Twisted tickets and has previously contributed to Evolution and Sugar Labs.

Her project:

Recently, two new APIs, IStreamServerEndpoint and IStreamClientEndpoint were added to Twisted, for specifying what address the servers should listen for connections and what address a client should connect to, respectively. But not all of the addresses that Twisted supports have this endpoint support added to them; presently endpoint support has been implemented for TCP, SSL and UNIX domain sockets. My project deals with adding more endpoint implementation to Twisted, some involving wrappers around the existing APIs (e.g. serial ports, standard I/O), others involving making fresh APIs where setting up connections was difficult before the addition of the endpoints (e.g. SOCKS and HTTPS proxies).

Python 3 preparation, by Vladimir Perić (IRC nick vperic)

Vladimir joins us from Czech Technical University in Prague. Last year he was a Google Summer of Code student with SymPy.

His project:

Python 3 is the future of Python. If Twisted is to see continued usage in the future, it will have to be ported, and rather sooner than later. As Twisted is a large and complicated code-base, this process needs to be done with care, ensuring that any code written remains compatible with the currently supported versions. The test-driven development methodology Twisted uses will ensure no regressions happen and will ease the maintenance of the code-base.

Automatic Coding Standard Enforcement, by Raphael Shu (IRC nick zomux)

Raphael joins us from Tsukuba University in Japan, where he uses Python daily in his NLP research.

His project:

Twisted applies certain naming and style standards to all contributed code. Currently, a human reviewer needs to check all of these things. The purpose of this project is to develop a tool which can automatically make these simple, mechanical checks, freeing up human reviewer time to focus on more important aspects of proposed changes. Finally, it will speed up the review process.

Software Freedom Conservancy / GNOME Outreach Program internship

We are also excited to be working with the Software Freedom Conservancy and the GNOME Outreach Program for Women this summer. You can read more about the initiative and our work to encourage diverse participation in open source communities here. Through this initiative we have a 4th paid, full-time internship this summer:

Improving Twisted Mail and Twisted Core, by Fei Tan (IRC nick argonemyth)

Fei joins us from Grand Bay, Mauritius, where she works as a freelance web developer.

She will improve Twisted Mail on a number of fronts, including improving API documentation, adding more examples, adding more HOWTOs, and improved test coverage.

Please join me in welcoming Ashwini, Vladimir, Raphael, and Fei, whose internships start next week. Expect a torrent of code reviews and some record-breaking high scores list stats this summer!

Thank you Google for giving us this paid mentorship opportunity, and thank you to the Python Software Foundation for supporting us as our Google Summer of Code umbrella organization.

• python-qrcode on github
• pyqrcode on sourceforge and
• pyqrnative on Google code

I tried python-qrcode and pyqrnative since they worked on Windows as well as Mac and Linux. They also didn’t require anything except the Python Imaging Library. The pyqrcode project requires several other prerequisites and didn’t work on Windows, so I didn’t want to mess with it. I ended up taking some old code based on my Photo Viewer application and modified it slightly to make this a QR Code viewer. If I’ve piqued your interest, then read on!

Getting Started

As I noted at the beginning, you will need the Python Imaging Library. We’ll be using wxPython for the GUI part, so you’ll need that as well. And you’ll want to download python-qrcode and pyqrnative. The main difference I have found is that python-qrcode is much faster at generating the images and it generates the type you’ve probably seen the most of. For some reason, pyqrnative take a lot longer to run and it creates a much denser looking QR code. There may be options for both of these projects that allow you to generate different kinds of codes, but the documentation for either project is abysmal. I ended up using the source and Wingware’s IDE to traverse the code more than anything else.

Generating QR Codes

Anyway, once you have all the prerequisites, you can run the following code and see what Python can do:

import os
import wx
 
try:
    import qrcode
except ImportError:
    qrcode = None
 
try:
    import PyQRNative
except ImportError:
    PyQRNative = None
 
########################################################################
class QRPanel(wx.Panel):
    """"""
 
    #----------------------------------------------------------------------
    def __init__(self, parent):
        """Constructor"""
        wx.Panel.__init__(self, parent=parent)
        self.photo_max_size = 240
        sp = wx.StandardPaths.Get()
        self.defaultLocation = sp.GetDocumentsDir()
 
        img = wx.EmptyImage(240,240)
        self.imageCtrl = wx.StaticBitmap(self, wx.ID_ANY,
                                         wx.BitmapFromImage(img))
 
        qrDataLbl = wx.StaticText(self, label="Text to turn into QR Code:")
        self.qrDataTxt = wx.TextCtrl(self, value="http://www.mousevspython.com", size=(200,-1))
        instructions = "Name QR image file"
        instructLbl = wx.StaticText(self, label=instructions)
        self.qrPhotoTxt = wx.TextCtrl(self, size=(200,-1))
        browseBtn = wx.Button(self, label='Change Save Location')
        browseBtn.Bind(wx.EVT_BUTTON, self.onBrowse)
        defLbl = "Default save location: " + self.defaultLocation
        self.defaultLocationLbl = wx.StaticText(self, label=defLbl)
 
        qrcodeBtn = wx.Button(self, label="Create QR with qrcode")
        qrcodeBtn.Bind(wx.EVT_BUTTON, self.onUseQrcode)
        pyQRNativeBtn = wx.Button(self, label="Create QR with PyQRNative")
        pyQRNativeBtn.Bind(wx.EVT_BUTTON, self.onUsePyQR)
 
        # Create sizer
        self.mainSizer = wx.BoxSizer(wx.VERTICAL)
        qrDataSizer = wx.BoxSizer(wx.HORIZONTAL)
        locationSizer = wx.BoxSizer(wx.HORIZONTAL)
        qrBtnSizer = wx.BoxSizer(wx.VERTICAL)
 
        qrDataSizer.Add(qrDataLbl, 0, wx.ALL, 5)
        qrDataSizer.Add(self.qrDataTxt, 1, wx.ALL|wx.EXPAND, 5)
        self.mainSizer.Add(wx.StaticLine(self, wx.ID_ANY),
                           0, wx.ALL|wx.EXPAND, 5)
        self.mainSizer.Add(qrDataSizer, 0, wx.EXPAND)
        self.mainSizer.Add(self.imageCtrl, 0, wx.ALL, 5)
        locationSizer.Add(instructLbl, 0, wx.ALL, 5)
        locationSizer.Add(self.qrPhotoTxt, 0, wx.ALL, 5)
        locationSizer.Add(browseBtn, 0, wx.ALL, 5)
        self.mainSizer.Add(locationSizer, 0, wx.ALL, 5)
        self.mainSizer.Add(self.defaultLocationLbl, 0, wx.ALL, 5)
 
        qrBtnSizer.Add(qrcodeBtn, 0, wx.ALL, 5)
        qrBtnSizer.Add(pyQRNativeBtn, 0, wx.ALL, 5)
        self.mainSizer.Add(qrBtnSizer, 0, wx.ALL|wx.CENTER, 10)
 
        self.SetSizer(self.mainSizer)
        self.Layout()
 
    #----------------------------------------------------------------------
    def onBrowse(self, event):
        """"""
        dlg = wx.DirDialog(self, "Choose a directory:",
                           style=wx.DD_DEFAULT_STYLE)
        if dlg.ShowModal() == wx.ID_OK:
            path = dlg.GetPath()
            self.defaultLocation = path
            self.defaultLocationLbl.SetLabel("Save location: %s" % path)
        dlg.Destroy()
 
    #----------------------------------------------------------------------
    def onUseQrcode(self, event):
        """

https://github.com/lincolnloop/python-qrcode

        """
        qr = qrcode.QRCode(version=1, box_size=10, border=4)
        qr.add_data(self.qrDataTxt.GetValue())
        qr.make(fit=True)
        x = qr.make_image()
 
        qr_file = os.path.join(self.defaultLocation, self.qrPhotoTxt.GetValue() + ".jpg")
        img_file = open(qr_file, 'wb')
        x.save(img_file, 'JPEG')
        img_file.close()
        self.showQRCode(qr_file)
 
    #----------------------------------------------------------------------
    def onUsePyQR(self, event):
        """

http://code.google.com/p/pyqrnative/

        """
        qr = PyQRNative.QRCode(20, PyQRNative.QRErrorCorrectLevel.L)
        qr.addData(self.qrDataTxt.GetValue())
        qr.make()
        im = qr.makeImage()
 
        qr_file = os.path.join(self.defaultLocation, self.qrPhotoTxt.GetValue() + ".jpg")
        img_file = open(qr_file, 'wb')
        im.save(img_file, 'JPEG')
        img_file.close()
        self.showQRCode(qr_file)
 
    #----------------------------------------------------------------------
    def showQRCode(self, filepath):
        """"""
        img = wx.Image(filepath, wx.BITMAP_TYPE_ANY)
        # scale the image, preserving the aspect ratio
        W = img.GetWidth()
        H = img.GetHeight()
        if W > H:
            NewW = self.photo_max_size
            NewH = self.photo_max_size * H / W
        else:
            NewH = self.photo_max_size
            NewW = self.photo_max_size * W / H
        img = img.Scale(NewW,NewH)
 
        self.imageCtrl.SetBitmap(wx.BitmapFromImage(img))
        self.Refresh()
 
 
########################################################################
class QRFrame(wx.Frame):
    """"""
 
    #----------------------------------------------------------------------
    def __init__(self):
        """Constructor"""
        wx.Frame.__init__(self, None, title="QR Code Viewer", size=(550,500))
        panel = QRPanel(self)
 
if __name__ == "__main__":
    app = wx.App(False)
    frame = QRFrame()
    frame.Show()
    app.MainLoop()

The code for changing and showing the picture is explained in the previous article I wrote (and linked to above), so the only parts that you’ll probably care about are the two methods for generating the QR codes: onUseQrcode and onUsePyQR. I just took some examples from their respective websites and modified them slightly to create the QR code images. They’re very straight-forward, but not well documented, so I can’t really tell you what’s going on. Sadly at the time of this writing, the code for these projects is seriously lacking in docstrings, with only a few here and there. Still, I was able to generate some decent QR codes. The following was done using python-qrcode:

As you can see, it’s a pretty standard code. The next one is created with PyQRNative and is much denser looking:

I tried scanning both images with my Android cell phone’s barcode scanning application and both QR codes were read correctly by it. So if you’re in need of generating QR code images for your project, I hope one of these projects will fit your needs!

Source Code

• QRCodeViewer.zip

I got into a discussion about code reuse with my co-worker, Jarek. I remember reading books on object oriented programming from a few decades ago in which OOP promised to increase code reuse. A lot of people doubted that serious code reuse would ever happen. OOP suggested that code reuse would be possible by reusing and subclassing existing classes. I even remember hearing C++ programmers talk about purchasing libraries of useful classes.

To some degree, this did actually happen. For instance, QT is a wonderful library for C++, and it certainly makes coding in C++ a lot easier. It was even proprietary up until a few years ago. Library reuse also happened via language-specific package sites such as CPAN (for Perl), PyPI (for Python), and a range of sites for Ruby. Furthermore, most languages come with a lot more stuff built in these days. For instance, compare Python's standard library ("batteries included") with that of C.

However, compared to decades ago, there is less of an emphasis on class-level reuse these days. For instance, rather than trying to code classes that will be flexible enough to solve tomorrow's problems, agile programmers suggests you write your code in such a way that you will be able to modify it to solve tomorrow's problems (with an emphasis on tests that will prevent regressions when you modify the existing code).

These days, rather than focusing on class-level reuse, we've achieved reuse via RESTful web services (and variations thereof) as well as via open source stacks. RESTful web services are usually closed source and proprietary. However, they have a lot of benefits. For instance, they're language agnostic. It's also a lot easier to control access to a RESTful Web Service than it is to a class library. This permits good authorization controls and various monetization strategies.

As I mentioned before, another place where we see a lot of reuse is in the open source stack. Just as Taco Bell has a million things on the menu that are all made from the same 10 ingredients, there are a million startups out there all based on the same stack of open source software. We even have a word for it--LAMP. Sure, there are variations of this--PostgreSQL instead of MySQL, Nginx instead of Apache, Python and Ruby instead of PHP--however, the fact remains that a lot of startups are only possible because of the tremendous amount of code that is available for reuse.

Hence, it's clear that code reuse arrived during the last 10-20 years. It didn't come exactly as we expected, but it's definitely here.