This is my weblog.

Custom TreeModel in Gtk3

2013-01-05T00:00:00+01:00

While porting CrunchyFrog to Gtk3 I came across a tricky issue. Porting the GenericTreeModel I use to display query results wasn't easy.

The first problem was that GenericTreeModel doesn't exist in the Python bindings for Gtk3 anymore. This class was specific to PyGTK 2. In Gtk3 you'll have to implement the TreeModel interface (and implementing an existing interface looks much cleaner).

But the I came across a bunch of issues. After asking on the mailing list I came up with the following solution:

The important thing is that python-gi (pygobject) >= 3.4 is required to make this work. On Debian Wheezy you'll have to install it from experimental ATM. The latest Ubuntu should already have the correct version.

Slides from my pdb presentation

2012-08-08T00:00:00+02:00

I've just uploaded the slides from my presentation of pdb, the builtin Python debugger, from yesterday's pyCologne meeting. The slides are in German, if this isn't a problem for you, you can download the PDF version here. Otherwise Doug Hellmann's PyMOTW article about pdb was the main source of this presentation and is a must-read.

The purpose of this presentation was to showcase the most basic features of the builtin debugger everyone has at hand. Too often I forget about this feature and add print statements everywhere instead of just jumping right into a pdb session and find out what's actually happening at a certain piece of code.

In most cases it's the right approach to add import pdb; pdb.set_trace() instead of another print. For me it turned out that adding a second or third print statement most likely means that I'm looking at the wrong piece of code and jumping into a debug session is much more efficient to find the real cause of a misbehavior.

pdb is such a handy module and distributed in the standard library. So why not use just it :)

Running flake8 in Emacs

2011-09-26T00:00:00+02:00

Some time ago I've discovered python-pylint.el, a minor mode for Emacs to run pylint in compilation mode. Running pylint in compilation mode is really nice since you can jump between errors with the usual Emacs keyboard shortcuts M-g n / M-g p.

But when focussing more on style then I want to run flake8 instead. Here's a thin wrapper (gist.github.com) around python-pylint that runs flake8 in compilation mode and parses the output. Nothing special, just the right commands and a regex to parse the output - but helpful :)

For the context switchers - git branch I/O

2011-08-11T00:00:00+02:00

At work I'm switching contexts quite often. While working on a long-term feature, I'll have to switch to do a bug fix, then I'll have to add a feature request that's done in a few hours and so on…

Usually I end up with several git branches in my working copy. For example, currently I have three branches:

$ git branch
  issue3521
  master
 *work

One goes upstream ("master"), one for instant fixes ("work"), one feature branch ("issue3521").

To get a quick overview of the current state of all branches I've added a very simple git extension to my setup that shows incoming and outgoing commits for each branch. I found the script (besides the issues it has) quite useful. It gives me just as little information as needed to orient myself after a couple of changes here and there. It just visualizes incoming and outgoing commits relative to a branch.

The following screenshot illustrates it's usage. The colors have the following meanings:

green	everything's fine
blue	almost ok
yellow	take care
red	do something

(Note, git bio is just an alias to git branch-io.)

So in that case I should keep an eye on issue3521 to keep it in sync with master, should merge back one commit from work to master and definitely should to something about the branch called outofdate.

If you want to give it a try, here's the extension (gist.github.com).

sqlparse 0.1.3

2011-07-29T00:00:00+02:00

I've just released a new bug fix release for sqlparse.

Changes since 0.1.2

Bug Fixes

Improve parsing of floats (thanks to Kris).
When formatting a statement a space before LIMIT was removed (issue35).
Fix stripcomments flag (issue38, reported by ooberm…@gmail.com).
Avoid parsing names as keywords (issue39, reported by djo…@taket.org).
Make sure identifier lists in subselects are grouped (issue40, reported by djo…@taket.org).
Split statements with IF as functions correctly (issue33 and issue29, reported by charles….@unige.ch).
Relax detection of keywords, esp. when used as function names (issue36, nyuhu…@gmail.com).
Don't treat single characters as keywords (issue32).
Improve parsing of stand-alone comments (issue26).
Detection of placeholders in paramterized queries (issue22, reported by Glyph Lefkowitz).
Add parsing of MS Access column names with braces (issue27, reported by frankz…@gmail.com).

Other

Replace Django by Flask in App Engine frontend (issue11).
Documentation moved to Read the Docs. Thanks for this great service!
Special thanks to Andriy Senkovych for packaging sqlparse for Debian/Ubuntu and giving some insights in all that packaging stuff. Thanks to all other package maintainers too!

Please file bug reports and feature request on the issue tracker.

A Selenium and Python appetizer

2011-07-12T00:00:00+02:00

Selenium is a browser automation framework and was released in version 2.0.0 just this week. In this post I'll show you how easy it is to get started with using this fine framework in Python.

Peparing the kitchen

Let's create and jump into a virtualenv. You don't really need it, but at least we don't need no sudo for anything :)

$ virtualenv demo
New python executable in demo/bin/python
Installing distribute..................................................done.
$ source demo/bin/activate
(demo) $

Installing the Selenium Python package is as easy as possible. Just use your favorite package manage and install it from PyPI:

(demo) $ pip install selenium

This installs the selenium Python package which includes all necessary drivers to run your browser automation using Firefox, Chrome or IE locally or anything that the Selenium Remote Control provides. We come back to that later.

If you want to automate Google's Chrome on your local machine some little setup is needed. If you want to follow the examples in this post and if you want to use an installed Firefox feel free to skip this section. To automate Chrome make sure Chrome is installed and then download and unzip the chromedriver for your system from the Selenium download page. Place the chromedriver binary somewhere in your $PATH. That's it.

Is it Google?

Now that Selenium is up and running, let's write a simple script that verifies that the title of the Google page actually contains the word "Google". That's simple and obvious, but the script illustrates how to use Selenium in UnitTests.

And you can do all this fancy stuff interactive in the Python shell:

#!/usr/bin/env python

import unittest

from selenium import webdriver


class TestTitle(unittest.TestCase):

    def setUp(self):
        # Toggle comments to test with a different browser
        self.driver = webdriver.Chrome()
        #self.driver = webdriver.Firefox()
        #self.driver = wevdriver.Ie()

    def tearDown(self):
        self.driver.close()

    def test_title_tag(self):
        self.driver.get('http://google.com')
        title_tag = self.driver.find_element_by_tag_name('title')
        self.assertEqual(title_tag.text, 'Google')


if __name__ == '__main__':
    unittest.main()

This script has one little drawback. It initializes a new webdriver (aka browser) for each tests. That's very accurate to start with a fresh browser for each test, but very time consuming too. As an alternative you can modify this script to use a global webdriver instance that is initialized before unittest.main() and closed after that line.

Since webdriver uses a network protocol to communicate with the browser, it's painless to use it in an interactive shell:

(demo) $ python
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
>>> from selenium import webdriver
>>> driver = webdriver.Chrome()
>>> driver.get('http://google.com')
>>> driver.find_element_by_tag_name('title')
<selenium.webdriver.remote.webelement.WebElement object at 0x29c2510>

That's a nice and easy way to figure out what webdriver commands are available and how they work.

Going headless

When using Selenium as part of your test suite, you don't have always an X environment around to start Firefox or Chrome. As you can see in the sample script above, it's easy to switch between different browsers without the need to change anything in your actual tests.

As the name suggests the remote webdriver talks to a Remote Control Server that let's you choose a browser. The nice thing is that the standalone server comes with a headless browser that even supports JavaScript execution much like Firefox or Chrome.

Download the standalone server JAR on the Selenium download page and simply run java -jar selenium-server-standalone-2.0.0.jar to start it. When the server is started you can initialize a headless browser in your scripts like this:

from selenium import webdriver
driver = webdriver.Remote(
    command_executor= 'http://127.0.0.1:4444/wd/hub',  # that's the default
    desired_capabilities={'browserName': 'htmlunit',
                          'version': '2',
                          'javascriptEnabled': True})

That's all to run your tests using a headless browser. Ready for your continuous integration system running on a server somewhere.

Testing web pages is fun, isn't it?

Edit 2011-07-12: Corey Goldberg posted a very similar article on his blog a few hours before I published this post.

Back from PythonCamp

2011-04-18T00:00:00+02:00

I'm back from a phantastic weekend at the Python BarCamp in Cologne. We've had two great days with lots of very interesting sessions (I'm still wondering how it's possible to stuff that many topics with such depth in two days, but it worked!), lightning talks, discussions, pizza, tweets, coffee, and and and … We're still working on the wiki pages to merge all notes taken during the sessions on various Etherpads.

To get an impression search for #pycamp on flickr.

Last but not least, big, big thanks to everyone hanging around at the PythonCamp this year for making it a very interesting and relaxing weekend!

Regex to parse pylint's parseable output

2011-04-13T00:00:00+02:00

Just in case someone (including myself) is looking for that: Here's the regex I'm using to parse the output from Pylint to extract error messages. Pylint needs to be called with the -f parseable flag.

Try to match each line of Pylint's output with this regex. If it matches call groupdict() to access to information bits easily.

Blog Moved

2010-04-21T00:00:00+02:00

Finally I've had some time last weekend to move my blog from wordpress.com over here. The main reason for doing this was the missing support for ReStructured Text on wordpress.com. I prefer plain text over WYSIWYG and I prefer to use my favorite editor for writing posts.

This blog now runs on App Engine with the Bloggart engine and Disqus for comments and this post is just an excuse to all my subscribers that all old posts popped up as new during migration :)

hgsvn 0.1.8 released

2010-01-24T00:00:00+01:00

A new maintenance release of hgsvn is available on PyPI. Go and grab it while it's hot :) Besides a lot of bug fixes this release includes a few new features and improvements:

Matt Fowles added the so-called "airplane mode", when using hgimportsvn with the --local-only flag no network access is required to convert a already checked out Subversion repository to a hgsvn controlled Mercurial repository.
Commit messages when pushing back to SVN can be edited before committing using the -e/--edit command line flag. Patch by eliterr.
sterin provided a patch that takes care that no mq patches are applied when running hgpushsvn or hgpullsvn.

Thanks to all contributors for reporting bugs and providing patches!

Pluggable App Engine e-mail backends for Django

2009-11-04T00:00:00+01:00

Yesterday support for pluggable e-mail backends has landed in Django's trunk and today I'm happy to announce two e-mail backend implementations to be used with Django-based App Engine applications.

The e-mail backends allow you to use native Django functions for sending e-mails like django.core.mail.send_mail() on App Engine.

To use pluggable e-mail backends you'll have to use a recent version of Django. Support for pluggable e-mail backends was introduced in rev11709 and will be released in Django 1.2.

Using the e-mail backend

The e-mail backends are available as a single package on bitbucket. To check out the most recent sources run:

$ hg clone http://bitbucket.org/andialbrecht/appengine_emailbackends/

or got to the downloads page and grab a zip file.

To use the e-mail backend for App Engine copy the appengine_emailbackend directory to the top-level directory of you App Engine application and add the following line to your settings.py:

EMAIL_BACKEND = 'appengine_emailbackend'

If you prefer asynchronous e-mail delivery use this line in your settings.py instead:

EMAIL_BACKEND = 'appengine_emailbackend.async'

When using the async backend e-mails will not be sent immediately but delivered via a taskqueue.

Running the demo application

There's also a demo application included in this repository. Run this application using

$ dev_appserver.py .

and open http://localhost:8080 in your web browser. Remember to include a current checkout of the Django SVN repository in the top-level directory of the demo application.

Dominion Set Designer

2009-10-11T00:00:00+02:00

If you're a regular Dominion player like me, you're always in search for new card sets. Fortunately I've had a little time this weekend to write a simple App Engine application that gives me a random card set on demand.

Screenshot of the Dominioin Set Designer

You can see it in action at http://dominionsets.appspot.com That's it, just a short post on this, now back to the next match :)

P.s.: Some technical stuff... There's Django with it's i18n support under the hood. The UI is powered by JQuery.

CrunchyFrog 0.4.1 released

2009-10-01T00:00:00+02:00

A new bugfix release for CrunchyFrog is out in the wild. There are not much visible changes, but a few bug fixes under the hood. One visible change is the option to give the SQL editor much more space by hiding all other UI elements in the main window. Here's a screenshot:

Pure SQL editing pleasure

The second screenshot shows the new highlighting of errors in the editor. In this case there's no table called "actor". When the execution of a statement results in a SQL error, CrunchyFrog now tries to find the position of the error in the SQL editor, highlights the corresponding part in your SQL statement and moves the cursor to the right place. I found it very handy to have the cursor in place to correct the error and run the statement again without searching the error before.

SQL editor with error message

If you're interested in the other, non-visible changes, please read the full list of changes or just go straight to the project page and give it a try :) Thanks to everyone who contributed to this release by sending patches and submitting issues on the tracker!

"Thank you, Mark"

2009-07-03T00:00:00+02:00

The quote in the title is something that Dana Colley said after giving credits to the members of Orchestra Morphine on some show circulating through the internet about nine years ago. I think this show is still available on the HI-N-DRY website and I strongly recommend to buy it, it's a fascinating recording.

I've choosen this quote because it expresses what I'm thinking today, the 10th anniversary of Mark Sandman's passing. His music shaped me more than any other music I'm listening too. He had that idea of a unique sound, not to mention the great lyrics. It's not easy to explain to someone who never listened to Morphine what this idea is about. It's not even easy to explain it to people that had to hear quite a lot of Morphine the past ten (and more) years (many, many thanks to my family for their patience with my musical taste they call "a bit strange" from time to time :)...

There's always something good about unique and innovative ideas: They make their way through time. Without a doubt, Mark died way too early. But his musical concept is still present today. Just have a look at Twinemen, A.K.A.C.O.D., Orchestra Morphine and many other artists and bands doing covers and claiming Marks musical ideas as a major influence for their own ideas.

I discoverd Morphine through a good friend of mine. He visited me after a travel to Canada somewhere in the late 90's and had those records in his hand, just saying "Listen to this." That fundamentally changed the way I was listening to music up to today. It was this mixture of blues, jazz, rock and poetry carried through a suprising soundscape of a 2-string slide bass, baritone saxophone and a straight and driving drum. Unfortunately I've never had the chance to see them live. We were about to see them in Brugge in July '99, but that show never happened.

I've met a few people since that who found the Morphine sound and lyrics "so charmingly depressive". But, hey, I'm not sure if they really listened... :) One thing I really like about the lyrics in Sandman songs is that there's always something that spontaneously comes to your mind in certain situations. Here, it is to just say "Yes" (in terms of "get in your go-cart and go, little sister").

A funny side-note regarding the lyrics: A few years ago I wrote my thesis about fortune in medieval Arthurian literature and after a busy day reading manuscripts, making notes und trying to understand the different concepts of fortune that can be found, I've put the B-Sides into my player and found myself laughing out loud, Mark summarizes them all in "Lucky Day":

Now I'm down a little in fact I'm down a lot I'm on a roller coaster ride that I can't stop My luck has changed but she'll come back That's the beauty of a game of chance I can't loose forever but I'm doomed to try Keep on hearing a voice inside Players win and winners play, have a lucky day There's both the concept of being at almighty Fortune's mercy and the concept of getting rid of her by using your own virtue condensed in a few lines while I was reading verse over verse to find a subtle phrase for it.

BTW, the version of "Lucky Day" found on the B-Sides is one of my favorites. The instrumental part preceeding these lines is incredibly powerfull and one of the most astonishing bits of music I've ever heard.

So, let me close with saying thanks too. Thanks to the folks at HI-N-DRY and the Mark Sandman Music Project for spreading a musical idea over years now, keeping up the memory of Mark Sandman and of course for the honest and good music they make. And thanks to Mark for giving his potion of uniqueness and sharing his thoughts and insights in an fascinating musical way. I'm looking forward to the first downloads on HI-N-DRY today featuring covers of Sandman songs, poetry and unreleased songs from the depths of the HI-N-DRY archive!

With mistakes yea mistakes and sudden inspirations Edges corners explosions convections All fast through a slow motion landscape

P.S.: The lyrics are copied from The Other Side, kudos to Ian Hadfield for keeping this site up!

Working on hgsvn

2009-06-20T00:00:00+02:00

Yesterday I've taken over maintenance of hgsvn from it's original author Antoine Pitrou. Many thanks to Antoine at this point for doing a great job on this tool!

In case you don't know, hgsvn is "a set of scripts to work locally on Subversion checkouts using Mercurial."

My aim is not to make these scripts a full-blown Subversion extensions for Mercurial. hgsubversion or the convert extension will probably do a much better job with tighter integration as it's in the scope of hgsvn. So the main focus is on fixing bugs and to finish and release the hgpushsvn script that commits changes in your Mercurial repository back to Subversion.

The reason why I decided to work on this project is pretty simple: I've made good experiences with it and I still want to keep using it.

With hgpushsvn the tool provides three scripts for the most basic tasks:

hgimportsvn initializes a hg repository and fetches the sources from svn, optionally starting at a specific revision.

hgpullsvn pulls new change sets from svn to your local hg repository.

hgpushsvn pushes back your local commits to the svn repository.

Using hgsvn for a while now, I'm feeling very comfortable with these scripts and they fit very well into my workflow.

The next things I'm about to do are:

fix the unittests, at least one is broken ATM
setup a buildbot to have some automated tests
finish hgpullsvn

If you're using hgsvn, please file bug reports or submit patches on the issue tracker.

How Do You Look When Merging Fails ;-)

2009-05-09T00:00:00+02:00

There was a Simpsons episode, I can't recall correctly, but I think Bart recorded Lisa when her heart broke and he watched it in slow motion to stop exactly at that point.

I thought of this episode yesterday while playing around with my laptop's webcam and a Python shell. Finally I wrote a little fun script that does almost the same: Just register it as a hg hook and it takes a picture of you exactly at the unique moment when merging fails and it sends it directly and without any further questions to Twitpic and Twitter:

#!/usr/bin/env python

import os
import sys
import tempfile
import time

from CVtypes import cv
from twitpic import TwitPicAPI


DEVICE = 0
TWITTER_USER = 'xxx'  # CHANGE THIS!
TWITTER_PWD = 'xxx'   # CHANGE THIS!
# This is the time in seconds you need to realize that the merge has
# failed. When setting this consider that it already takes about a second
# for the camera to take the picture. &quot;0&quot; means no delay ;-)
EMOTIONAL_SLUGGISHNESS_RATE = 0.0


def grab_image(fname):
    camera = cv.CreateCameraCapture(DEVICE)
    frame = cv.QueryFrame(camera)
    cv.SaveImage(fname, frame)


def how_do_you_look():
    failed = bool(os.environ.get('HG_ERROR', 0))
    if not failed:
        return  # hmpf, maybe next time...
    fd, fname = tempfile.mkstemp('.jpg')
    if EMOTIONAL_SLUGGISHNESS_RATE &gt; 0:
        time.sleep(EMOTIONAL_SLUGGISHNESS_RATE)
    grab_image(fname)
    twit = TwitPicAPI(TWITTER_USER, TWITTER_PWD)
    retcode = twit.upload(fname, post_to_twitter=True,
                          message='Another merge failed.')
    os.remove(fname)


if __name__ == '__main__':
    how_do_you_look()

You'll need the CVtypes OpenCV wrapper and this Twitpic Python module. I've patched the twitpic module to support messages. Have a look at this issue if it's already supported, otherwise a diff that adds the message keyword is attached to the issue. To use it as a Mercurial hook just add to .hg/hgrc:

[hooks]
update =  /path/to/the/above/script.py

and make the script executable.

The results are pretty good :)

http://twitpic.com/4txls

Have fun!

BTW, the way how to access the camera is inspired by this nice blog post about face recognition using OpenCV.

Edit (2009-05-12): It was Ralph, not Lisa. Thanks Florian!

App Engine Tracebacks via E-Mail

2009-04-30T00:00:00+02:00

Edit 2009-09-04: Starting with App Engine SDK 1.2.5 released today there's now a much smarter approach. See the docstring of the ereporter module for details!

Django's error reporting via e-mail is something I've missed for my App Engine applications. If you've followed the instructions on how to use Django on App Engine errors are already logged. But honestly, I don't check the logs very often and sometimes it's just a silly typo that's fixed in a minute ;-)

So if you've followed the instructions you can simply extend the log_exception function to send an email after the exception is logged:

def log_exception(*args, **kwds):
  """Django signal handler to log an exception."""
  excinfo = sys.exc_info()
  cls, err = excinfo[:2]
  subject = 'Exception in request: %s: %s' % (cls.__name__, err)
  logging.exception(subject)

  # Send an email to the admins
  if 'request' in kwds:
    try:
      repr_request = repr(kwds['request'])
    except:
      repr_request = 'Request repr() not available.'
  else:
    repr_request = 'Request not available.'
  msg = ('Application: %s\nVersion: %s\n\n%s\n\n%s'
         % (os.getenv('APPLICATION_ID'), os.getenv('CURRENT_VERSION_ID'),
            ''.join(traceback.format_exception(*excinfo)),
            repr_request))
  subject = '[%s] %s' % (os.getenv('APPLICATION_ID'), subject)
  mail.send_mail_to_admins('APP_ADMIN@EXAMPLE.COM',
                           subject,
                           msg)

Make sure to replace "APP_ADMIN@EXAMPLE.COM" with the email address of an admin for this application and to add these two imports somewhere at the top of your main.py:

import traceback
from google.appengine.api import mail

You don't need to comment this when developing your app to avoid lots of useless mails. The dev server doesn't fully implement send_mail_to_admins() most likely as there's no real concept of an designated admin user in the SDK. It just writes a short message to the log, even if the sendmail option is enabled.

P.S.: If you use webapp instead of Django have a look at the seventh slide of this presentation by Ken Ashcraft. He shows a similar implementation for webapp based applications.

SQL Parsing with Python, Pt. I

2009-03-29T00:00:00+01:00

Some time ago I was in search for some kind of SQL parser module for Python. As you can guess, my search wasn't really successfull. pyparsing needs a grammar, but I'm not really interested in writing a full-blown grammar for SQL with it's various dialects. AFAICT Gadfly implements a customized SQL parser, but I was not able to figure out how to use it. And there's sqlliterals, but as the name suggests it's just for identifying literals within statements.

I expect such a module to do the following:

It should be fast ;-)
scalable - what the parser needs to know about a string containing SQL statements depends on what I want to do with it. If I just want to split that string in separate statements I don't need to know as much as when I want to know what identifiers occur in it.
non-validating - Parsing shouldn't fail if the statement is syntactically incorrect as I want to use it for an SQL editor where the statements are (naturally) incorrect most of the time.
some beautifying is a nice-to-have - We all know SQL generated by a script and copied from a logging output to a database front-end - it always looks ugly and is pretty unreadable.

The only thing I've found that comes close to my needs is Pygments (yes, the syntax highlighter). It does a really good job highlighting SQL, so it "understands" at least something. To be concrete, it has a rock-solid and fast lexer. There's a FAQ <http://pygments.org/faq/#understand-code> entry if it's possible to use Pygments for progamming language processing. The answer is:

The Pygments lexing machinery is quite powerful can be used to build lexers for basically all languages. However, parsing them is not possible, though some lexers go some steps in this direction in order to e.g. highlight function names differently.

Also, error reporting is not the scope of Pygments. It focuses on correctly highlighting syntactically valid documents, not finding and compensating errors.

There's a nice distinction between lexers and parser in this answer. The former does lexical and the latter syntactical analysis of a given stream of characters. For my needs this led to a two step process: first to do the lexing using Pygments mechanism (BTW, read "focuses on [...] syntactically valid documents, not finding [...] errors" as "non-validating") and then to add parsing on top of the token stream.

So I stripped the lexing mechanism for SQL out of Pygments to get rid of some overhead not needed for my purposes (e.g. loading of extensions). The only changes to the Pygments tokenizer was to replace a huge regex for finding keywords by a dictionary-based lookup and to add a few new tokens to make live easier in the second processing stage. The achieved performance improvement by replacing the regex doesn't play a significant role for Pygments, it just speeds up the lexing a bit.

In the second step the very efficient token-type/value stream generated in step 1 is grouped together in nested lists of simple classes. Instantiating a bunch of classes results in a performance loss, but with the benefit of some helpful methods for analyzing the now classified tokens. As the classes are pretty simple most of the performance loss was recaptured by using slots.

The grouping mechanism is again non-validating. It tries to find patterns like identifiers and their aliases or WHERE clauses and their conditions in the token stream, but it just leaves the tokens unchanged if it doesn't find what it's looking for. So it actually defines some sort of "grammar", but very unrestrictive. That means that the quality of answers to questions like "Is there a WHERE clause?" or "Which tables are affected?" heavily depends on the syntactical clearness of the given statement at the time when it's processed and it's up to the user or application using the module to intrepet the answers right. As the module is seen as a helper during the develoment process of SQL statements it doesn't insist on syntactical correctness as the targeted database system would do when the statement is executed.

All in all these are the two (simple) processing steps:

Lexing a given string using regular expressions and generate a token-type/value stream (in fact, this is the Pygments lexer).
Parsing
1. Instantiate a simple token class for each item in the stream to ease syntactical analysis.
2. Group those tokens in nested lists by using simple functions by finding patterns.

Depending on what should be done with the statements, only the first step is required. For example, splitting statements or simple formatting rules don't require parsing.

Pt. II of this blog post will cover the two remaining points: scalibality and the beautifiying goodie. And it will give a closer look at the module I came up with, including some performance stats. For the impatient: the source code of this module is available here.

SQL Parsing with Python, Pt. II

2009-03-29T00:00:00+01:00

After I've described some basics in part I let's have a closer look to the actual Python module called sqlparse. First off, have a look at the project page on how to download and install this module in case you're interested. But now, let's have some fun...

The API of the module is pretty simple. It provides three top-level functions on module level: sqlparse.split(sql) splits sql into separate statements, sqlparse.parse(sql) parses sql and returns a tree-like structure and sqlparse.format(sql, **kwds) returns a beautified version of sql according to kwds.

As mentioned in my previous post, what effort needs to be done to build the return values depends on what lexing and parsing work is needed to find the result. For example sqlparse.split() does the following:

Generate a token-type/value stream basically with a copy of Pygments lexer
Apply a filter (in terms of a Pygments stream filter) to find statements
Serialize the token stream back to unicode

sqlparse.parse() does all the grouping work and sqlparse.format() runs through those groups, modifies them according to the given formatting rules and finally converts it back to unicode.

Here's an example session in a Python shell:

>>> import sqlparse
>>> # Splitting statements:
>>> sql = 'select * from foo; select * from bar;'
>>> sqlparse.split(sql)
>>> # Formatting statemtents:
>>> sql = 'select * from foo where id in (select id from bar);'
>>> print sqlparse.format(sql, reindent=True, keyword_case='upper')
SELECT *
FROM foo
WHERE id IN
  (SELECT id
   FROM bar);

>>> # Parsing
>>> sql = 'select * from "someschema"."mytable" where id = 1'
>>> res = sqlparse.parse(sql)
>>> res
<<< stmt = res[0]
>>> stmt.to_unicode()  # converting it back to unicode
<<< u'select * from "someschema"."mytable" where id = 1'
>>> # This is how the internal representation looks like:
>>> stmt.tokens
[<DML 'select' at 0x2a8dba8>,
 <Whitespace ' ' at 0x2a8dc00>,
 <Wildcard '*' at 0x2a8dc58>,
 <Whitespace ' ' at 0x2a8dcb0>,
 <Keyword 'from' at 0x2a8dd08>,
 <Whitespace ' ' at 0x2a8dd60>,
 <Identifier '"somes...' at 0x2a8f628>,
 <Whitespace ' ' at 0x2a8dec0>,
 <Where 'where ...' at 0x2a8f518>]

Now, how does the grouping work? Grouping is done with a set of simple functions. Each function searches for a simple pattern and if it finds one a new group is built. Let's have a look at the function that finds the WHERE clauses.

def group_where(tlist):
    [group_where(sgroup) for sgroup in tlist.get_sublists()
     if not isinstance(sgroup, Where)]
    idx = 0
    token = tlist.token_next_match(idx, T.Keyword, 'WHERE')
    stopwords = ('ORDER', 'GROUP', 'LIMIT', 'UNION')
    while token:
        tidx = tlist.token_index(token)
        end = tlist.token_next_match(tidx+1, T.Keyword, stopwords)
        if end is None:  # WHERE is at the end of the statement
            end = tlist.tokens[-1]
        else:
            end = tlist.tokens[tlist.token_index(end)-1]
        group = tlist.group_tokens(Where, tlist.tokens_between(token, end))
        idx = tlist.token_index(group)
        token = tlist.token_next_match(idx, T.Keyword, 'WHERE')

tlist is a list of tokens, possible subgroups are handled first (bottom-up approach). Then it grabs the first occuring "WHERE" and looks for the next matching stop word. I'm pretty unsure if the stop words approach is right here, but at least it works for now... If it finds a stop word, a group using the class Where is created, the tokens between WHERE and the stop word are attached to it and - still within the while loop - the next WHERE keyword is used as the next starting point.

So why not use a grammar here? At first, this piece of code is pretty simple and easy to maintain. But it can also handle grammatically incorrect statements more lazily, e.g. it's no problem to add an if clause that - when for example an unexpected token occurs - the function just jumps to the next occurance of WHERE without changing anything or even raising an exception. To achieve this the token classes provide helper functions to inspect the surroundings of an occurrence (in fact, just simple list operations). There's no limitation what a grouping function can do with the given token list, so you could even "guess" a proper group with some nasty algorithm.

The current problem with this approach is performance. Here are some numbers:

Size	split()	parse()
100kb (20600 tokens)	0.3 secs.	1.8 secs.
1.9MB (412500 tokens)	5.53 secs.	37 secs.

Most of the performance is lost when giving up the stream-oriented approach in the parsing phase. The numbers are based on the first unrevised working version. I expect performance improvents especially in the way how token lists are handled behind the scenes with upcoming versions. For real life statements the parser behaves quite well. BTW, the Pygments lexer takes about 6 seconds (compared to 5.5 secs. for splitting) for the 1.9MB of SQL.

The non-validating approach is a disadvantage too. You'll never know if a statement is valid. You can even parse middle high german phrases and receive a result:

>>> sqlparse.parse('swer an rehte güete wendet sîn gemüete')[0].tokens
[<Identifier 'swer an' at 0x2a8f0d8>,
 <Whitespace ' ' at 0x2a8d788>,
 <Identifier 'rehte ...' at 0x2a8f1e8>,
 <Whitespace ' ' at 0x2a8d8e8>,
 <Identifier 'wendet...' at 0x2a8f2f8>,
 <Whitespace ' ' at 0x2a8da48>,
 <Identifier 'gemüete' at 0x2a8f408>]
>>>

It's up to the user of this module to provide suitable input and to interpret the output. Furthermore the parser only supports not every nifty edge of an SQL dialect. Currently it's mostly ANSI-SQL with some PostgreSQL specific stuff. But it should be easy to implement further grouping functions to provide more SQL varieties.

The splitting feature is currently used by CrunchyFrog and does a pretty good job there. I assume that SQL splitting works stable and reliable in most cases. Beautifiying and parsing is very new in the module and full functionality needs to be proven with time. Luckily the top-level API with it's three functions is damn simple and keeps the doors open for significant changes behind the scenes if they're needed.

The sources of the sqlparse module are currently hosted on github.com but may move to Google Code anytime soon. Refer to the project page on Google Code for downloads and how to access the sources.

Edit: The current repository is once again on http://github.com/andialbrecht/sqlparse.

In addition there's a simple AppEngine application that exposes the formatting features as an online service.

Creating Man Pages Using optparse and distutils

2009-03-17T00:00:00+01:00

This blog post describes how to generate a simple man page during build time for Python applications using distutils and optparse. Technically this post describes how to write a custom HelpFormatter and a custom build command.

For GUI applications command line options usually bother me not much. Just in some rare cases, mainly when debugging or feeling unlucky with the startup behavior, I'm interested in these options. Even then I prefer to use "--help". BTW, for the "real" command line tools like sed, ls or grep this is totally different.

So, due to my lack of interest in man pages for GUI applications I've shipped the previous release of CrunchyFrog along with a totally out-dated man page. Moreover I really don't like writing man pages or keeping them in sync with the actual command line options already documented by OptionParser's help features...

Did I say "already documented"? I'm not a friend of doing things twice, reusing that "already documented" should be very pleasant to go to get rid of my problem.

What To Put On A Man Page?

The short answer is: It should mention the application's name, it's purpose, how to start it, who wrote it, where to find additional information and what command line options it provides. Or, in terms of man page sections: "NAME", "SYNOPSIS", "DESCRIPTION", "OPTIONS", "SEE ALSO" and "AUTHOR" (as you may already have noticed, I'm focussing just on man page (1) in this post). Of course, there are more possible sections, but after inspecting some randomly choosen man pages for GUI applications on my system, it seems that those sections are the most common ones.

(If you are interested in the full answer, read the Linux Man Page Howto.)

Now back to our Python application that uses optparse to provide command line options.

optparse comes with a pretty good out-of-the-box help feature. As stated in the docs the primary purpose is to assist in creating user-friendly command-line interfaces and not to generate help documents (like man pages). But the ability to write a custom HelpFormatter makes it easy to generate such documents.

Writing a Custom HelpFormatter

Let's have a look at the custom formatter:

class ManPageFormatter(optparse.HelpFormatter):

    def __init__(self, indent_increment=2, max_help_position=24,
                 width=None, short_first=1):
        """Constructor. Unfortunately HelpFormatter is no new-style class."""
        optparse.HelpFormatter.__init__(self, indent_increment,
                                        max_help_position, width, short_first)

    def _markup(self, txt):
        """Prepares txt to be used in man pages."""
        return txt.replace('-', '\\-')

    def format_usage(self, usage):
        """Formate the usage/synopsis line."""
        return self._markup(usage)

    def format_heading(self, heading):
        """"Format a heading.
        If level is 0 return an empty string. This usually is the string "Options".
        """"
        if self.level == 0:
            return ''
        return '.TP\n%s\n' % self._markup(heading.upper())

    def format_option(self, option):
        """Format a single option.

        The base class takes care to replace custom optparse values.
        """"
        result = []
        opts = self.option_strings[option]
        result.append('.TP\n.B %s\n' % self._markup(opts))
        if option.help:
            help_text = '%s\n' % self._markup(self.expand_default(option))
            result.append(help_text)
        return ''.join(result)

The three methods starting with format_* do all the formatting for us. The base class has additional formatting methods, but there's no need to overwrite them as they do nothing special except wrapping lines to the desired width.

To use a custom formatter either give it as an parameter to the OptionParser constructor or use

parser.formatter = ManPageFormatter()
parser.formatter.set_parser(parser)

if your option parser is already instantiated.

Now, when you call parser.format_option_help() you'll see a man page fragment with all options (including nicely formatted option groups). Maybe you wonder why not use parser.print_help() here, which normally adds a nice usage line and the description if given to the output. The reason for that is quite simple: Many of the documentation features of optparse are optional. If there's no usage defined, no usage line will be formatted. So using it for the SYNOPSIS section and eventually the header of the man page isn't a good idea. Everything surrounding the OPTIONS section of the man page should be generated by something else, that takes care to include a proper header and footer. We are using the custom formatter just for the options themself.

Adding a Custom Build Command

To write the rest of the man page we're using a custom build command for distutils. This command is executed when you run python setup.py build or install that depends on build. There's a nice side-effect when generating the man page on build time: You can simply skip this step when the setup.py commands are run on an OS that doesn't know what a man pages is ,-)

The skeleton of a distutils command looks like this:

# -*- coding: utf-8 -*-

"""build_manpage command -- Generate man page from optparse/dist metadata."""

from distutils.command.build import build
from distutils.core import Command


class build_manpage(Command):

    description = 'Generate man page.'
    user_options = []

    def initialize_options(self):
       pass

    def finalize_options(self):
       pass

    def run(self):
       # Do something useful!


build.sub_commands.append(('build_manpage', None))

As you can see, "build_manpage" is added as a sub-command to the build command and the build_manpage class has some methods for option handling and a run() method.

To use this command add the following to your setup.py:

from distutils.core import setup

from build_manpage import build_manpage

setup(
    [...]
    cmdclass={'build_manpage': build_manpage}
)

Adding Command Options

Generating the man page needs some options. The build_manpage commands needs to know where to write the generated man page and which option parser it should use. To achieve this we have to modify the user_options attribute and the two corresponding methods in our skeleton class:

user_options = [
    ('output=', 'O', 'output file'),
    ('parser=', None, 'module path to optparser (e.g. mymod:func'),
    ]

def initialize_options(self):
    self.output = None
    self.parser = None

def finalize_options(self):
    if self.output is None:
        raise DistutilsOptionError('\'output\' option is required')
    if self.parser is None:
        raise DistutilsOptionError('\'parser\' option is required')
    mod_name, func_name = self.parser.split(':')
    fromlist = mod_name.split('.')
    try:
        mod = __import__(mod_name, fromlist=fromlist)
        self._parser = getattr(mod, func_name)()
    except ImportError, err:
        raise
    self._parser.formatter = ManPageFormatter()
    self._parser.formatter.set_parser(self._parser)
    self.announce('Writing man page %s' % self.output)
    self._today = datetime.date.today()

user_options is a list of 3-tuples (long option, short option, help), initialize_options() sets default values and finalize_options() makes sure that the given options are sane (nb, the Command class has some validation functions, unfortunately you'll have to read the source to find them...). Run setup.py build_manpage --help to see how to use your options on the command line. Besides setting these options on the command line you can set them in a file called setup.cfg too:

[build_manpage]
output=data/mymanpage.1
parser=myapp.somemod:get_parser

Both options are required. output tells the command to which file it should write the generated man page, parser is a string pointing to a function that returns an OptionParser instance (a dotted module path with a function name separated by a single colon).

Putting It All Together

This is done by the run() method, of course. In finalize_options() the option parser was imported and our ManPageFormatter was attached to it. The run() method now does the following:

def run(self):
    manpage = []
    manpage.append(self._write_header())
    manpage.append(self._write_options())
    manpage.append(self._write_footer())
    stream = open(self.output, 'w')
    stream.write(''.join(manpage))
    stream.close()

Let's have a look at the _write_header() helper function (_write_options() is basically a call to the formatter and _write_footer() just adds the author and a link to the homepage):

def _write_header(self):
    appname = self.distribution.get_name()
    ret = []
    ret.append('.TH %s 1 %s\n' % (self._markup(appname),
                                  self._today.strftime('%Y\\-%m\\-%d')))
    description = self.distribution.get_description()
    if description:
        name = self._markup('%s - %s' % (self._markup(appname),
                                         description.splitlines()[0]))
    else:
        name = self._markup(appname)
    ret.append('.SH NAME\n%s\n' % name)
    synopsis = self._parser.get_usage()
    if synopsis:
        synopsis = synopsis.replace('%s ' % appname, '')
        ret.append('.SH SYNOPSIS\n.B %s\n%s\n' % (self._markup(appname),
                                                  synopsis))
    long_desc = self.distribution.get_long_description()
    if long_desc:
        ret.append('.SH DESCRIPTION\n%s\n' % self._markup(long_desc))
    return ''.join(ret)

As you can see, this method makes use of another advantage of using a distutils command. It fetches some information not covered by optparse from distutils' meta data and in addition takes care of optparse's optional usage and description attributes.

The complete implementation of the build_manpage command is available here.

This is how the resulting man page for CrunchyFrog looks like:

man crunchyfrog

I'll have to admit, it's a very simple approach to have a man page without writing it and - even worse - maintaining it. But at least it solved my problem to distribute a GUI application with an up-to-date man page. This approach has no extra requirements as everything needed can be found in Python's stdlib and the man page is generated when needed. Luckily the build command is extensible, so it's no problem to add additional sections (what about connection the script to a bug tracker and add a hopefully not too long BUGS section?).

P.S.: In case your interested in another formatter example, I've recently written another HelpFormatter that creates a Google Code Wiki page for Rietveld's upload.py. The source are available here.

Edit (2009-03-25): There's a nice patch written by René Neumann that adds a SEE ALSO option to the distutils command.

Something completely different

2009-02-06T00:00:00+01:00

Spent half of the evening crawling through sites I've rarely visited in the last few months.

There are at least two "new" Sandman songs on MySpace. Ok, they're not new at all, but mystic... ;-) BTW if you haven't did so already, support the Mark Sandman Music Project in any way...

While clicking links back and forth I came across a Hi-N-Dry photostream on Flickr. hm.. the new rooms look way too clean compared to the legendary old ones (but that's maybe just because they're new...)

While talking about fresh music, you definitely have to listen or to view the Chip Smith Project! What a refreshing music, ol' Chip Smith must have been a very honorable man!

A cleaner UI for CrunchyFrog

2009-01-29T00:00:00+01:00

Today I've released a new version of CrunchyFrog. Besides beeing mainly a bug fix release, CrunchyFrog 0.3.2 introduces some enhancements under the hood.

The most visible is of course a cleaner user interface. The more or less experimental dock implementation was removed in favour of a - at least in my opinion - cleaner interface heavily inspired by GEdit. Along with this changes I've cleaned up some requirements: Removed modules that where only used in a few places and made others optional. The latter includes the various GNOME bindings.

This changes were made to provide initial support for running CrunchyFrog with other desktop environments and on different platforms. With this release a new package archiv on launchpad opened it's doors. If you're an Ubuntu (or Debian) user you can use this PPA to update your CrunchyFrog installation from this repository.

Read the full announcement for more information about what has changed since the last release. Any further enhancements like a terminal plugin to access native database command lines within the application or a somewhat improved schema browsing will go into the next major release.

Thanks to everyone for giving feedback, reporting bugs and for the good ideas during the last months! Have fun with it and - as always - feel free to report any bugs or feature requests... ,-)

Experiences with Rietveld on Django

2008-11-14T00:00:00+01:00

It's been a while now that we're running a Rietveld instance on Django at our company for internal reviews. Besides the basic setup as described in my article on Google's App Engine site we've made some more modifications to integrate Rietveld into our infrastructure. Here's a short description of what we've done and what problems we've had (and still have):

First off, we're using PostgreSQL as the database backend. Thanks to Django the database setup was quite simple: We've just changed the database settings in settings.py, ran "manage.py syncdb" and that's it. The first time we've ran this setup a few minor bugs regarding type conversion in gae2django came up, but they were fixed soon. For authentication we've added a simple backend that talks to our company's ActiveDirectory (using python-ldap). Implementing that was again quite easy, just follow Django's documentation on how to write a custom authentication backend and let Django do the rest. The most time consuming change was the integration with an existing web server. We've already had a lighttpd server running and Rietveld should live inside a subfolder on this server. Because Rietveld assumes to sit in a top-level directory we need to change all absolute URLs in Rietveld's sources, templates and in the JavaScript

We still have some broken links in our setup, e.g. using the keyboard shortcuts to navigate between deltas still doesn't work and a few weeks ago there was a change in Rietveld (collapsed patch sets on the issue page are loaded with an AJAX call when expanded) that is still not ported to our setup yet. It's just a matter of time to have this fixed. Django runs in fcgi mode (threaded) and lighttpd acts as a proxy as described in the "Deployment" section of Django's documentation. In the first few days we've had some strange problems with sending emails. Some emails were lost. We had some troubles to find the actual bug, but after I've done some improvements to gae2django the problem was almost gone. Currently only the sender of an email doesn't receive his own mail but this might be an issue related to the setup of our mail servers. We're using upload.py to create and update issues. To avoid confusions we've renamed it and changed the default settings for the login URL and the address of the review server. The master template has some slight modifications too: We removed the link to the issue creation form and to all repository related pages, added a little question mark pointing to a help page and we've added a nice header image to match our CI.

Rietveld running on local infrastructure

Updating this customized installation of Rietveld is done using a Makefile-driven patch system. First we fetch the latest Rietveld sources from it's Subversion repository, then we update gae2django and apply the patches from this project required to get Rietveld running on Django. On top of that we apply the changes I've described above. Of course, this procedure still needs some work, but at least it works and keeping those patches in sync isn't that complicated as you might think ,-)

But besides the email issue and some broken links due to our customized setup Rietveld runs pretty stable on Django. Currently the Django server runs 3 weeks without any troubles even without a traceback ;-)

Finally a few notes on how code reviews were integrated in our workflow. When we've decided to use a code review tool, there were some concerns about the time needed to review other peoples code. But after a short period of getting comfortable with this tool - and now we feel very comfortable with it ;-) - it turned out that it's less effort to do a review than expected. Mostly one or two iterations are sufficient enough to eliminate the most obvious things and to give a LGTM. Only changes that have to go immediately into the code base are reviewed afterwards.

Besides that, the knowledge about special parts of the code has spread and as another side-effect there's now much more discussion about best-practices for some application specific tasks and the process of finding and defining them has accelerated noticeably.

Edit 2009-01-12: The email issue seems to be solved. Our Postfix had some troubles with multiple addresses given as in a single, comma separated string. Splitting this string into a list of string (addresses) solved the problem.

CrunchyFrog 0.3.0 has landed

2008-10-24T00:00:00+02:00

A new version of CrunchyFrog is out. Besides a bunch of bug fixes and UI improvements this release adds support for GNOME keyring and finally the SQL editor supports multiple statements, the most visible change in this version.

Option to split SQL statements before actually executing them.

Splitting of SQL statements for various dialects is very tricky and may fail under certain circumstances. Fortunately the SQL editor is clever enough to mark statements with a little arrow besides the line numbers while you're hacking. So if you're not lucky with the splitting of your queriesm you can always disable this feature either by using the context menu (see highlighted entry in screenshot) to temporarily disable it or open the preferences menu and disable it permanently. (Keep in mind, it's still possible to execute only parts by doing a text selection in the editor.) Getting CrunchyFrog up and running is simple four step process (assuming all required Python modules are already installed on your system):

Download the source archive
Run tar xvfz crunchyfrog-0.3.0.tar.gz
cd crunchyfrog-0.3.0/
Run ./crunchyfrog

Please file bug reports and feature requests on the issue tracker and feel free to contribute ;-)

A Google Code Sphinx Theme

2008-10-21T00:00:00+02:00

The wiki feature on Google Code is a great feature to provide documentation for a project. In my opinion it has to big benefits: You can version your documentation and (thanks to the wiki markup) it's easy to maintain. But on the other hand writing your projects documentation using the wiki has some limitations.

Especially when you know Sphinx, the documentation generator used by Python, you will miss some features like Python-specific markup, automagic index generation, sections and chapters and last but ot least the power of restructured text ;-)

Unfortunately Google Code Hosting offers no official way to serve static files. So if you want to use Sphinx for your documentation, you need to publish it somewhere else and put a link on your project page. Wouldn't it be great to use Sphinx and integrate it with everything else? So let's do it, but first off here's how it looks like (visit this project page to see it in action):

And two other screenshots. The first one shows an index...

... and the second shows some search results using the Sphinx search engine:

Here's how to use this theme for your project and how to "publish" it on Google Code.

Read the Sphinx documentation if you don't know anything about Sphinx yet. To make this work, your documentation files and the generated HTML output must live somewhere in your Subversion repository. The theme consists of three files: Grab the CSS from here and place it in the directory for static files (usually DOCROOT/source/.static where DOCROOT is the root directory of your Sphinx documentation), then grab the two templates (layout.html, search.html) and place them in the templates folder (DOCROOT/source/.templates).

Now open the configuration file of your documentation (conf.py) and make sure it has this line html_file_suffix = '.html'. It's required to build the correct URL for search results. The layout template has some configuration options. Open layout.html in your favorite editor and customize them for your needs: gcprj --> set this to match your project name (the one in the URL...). gcusedownloads --> 1 = show "Downloads" tab, 0 = don't show it gcusewiki --> 1 = show "Wiki" tab, 0 = don't show it gcuseissues --> 1 = show "Issues" tab, 0 = don't show it gcusespurce --> 1 = show "Source" tab, 0 = don't show it Then build the HTML version of your documentation by running make html. Make sure that all generated files are under version control (e.g. by running svn status build/html) as we will deliver all content straight out of the repository... ,-) Finally you need to set the correct mime types for all generated files. Otherwise they're delivered as plain text which is not very useful in our case. You can easily set the correct SVN properties by running the following commands:

svn propset -R svn:mime-type text/css `find build/html/ -name .svn -type f -prune -o -name *.css`
svn propset -R svn:mime-type text/javascript `find build/html/ -name .svn -type f -prune -o -name *.js`
svn propset -R svn:mime-type image/x-png `find build/html/ -name .svn -type f -prune -o -name *.png`
svn propset -R svn:mime-type text/html `find build/html/ -name .svn -type f -prune -o -name *.html`

That's it. Now "publish" your documentation by running svn commit... ,-) To view your documentation point your browser to http://<YOUR_PROJECT>.googlecode.com/svn/<DOCROOT>/build/html/index.html.

Some Thoughts on CrunchyFrog 0.3

2008-09-11T00:00:00+02:00

In the last few months some people asked me how to execute single statements in CrunchyFrog. It's possible, just select the text you want to execute and hit F5. If no text is selected all text in the SQL editor is executed in whole. This is not really satisfying so I played around with a SQL splitter. It's inspired by Google's GQL parser found in their App Engine SDK and currently it only tries to find statement boundaries in the SQL editor. The statements are marked with a little arrow and that's all for the moment. The backend that executes the statements still needs some work. If everything works as expected some simple SQL parsing will be part of the next major release. If anyone knows a good SQL parser module for Python, leave a comment! Even a solid SQL splitter would be enough in the first run. Another topic for the next major release is the plugin system, especially the install and update mechanism. I plan to use Capucchin. It should be easy to integrate in the current sources and it's definitely better than another homegrown plugin updater and installer.

Using ReStructuredText on App Engine

2008-08-14T00:00:00+02:00

To use docutils in your App Engine application to render ReStructuredText <http://docutils.sourceforge.net/rst.html> _first download the module here <http://docutils.sourceforge.net/> and place the docutils directory and roman.py in the top-level directory of your application:

$ cd my_appengine_app/
$ wget http://prdownloads.sourceforge.net/docutils/docutils-0.5.tar.gz?download
$ tar xvfz docutils-0.5.tar.gz
$ mv docutils-0.5/docutils/ docutils-0.5/extras/roman.py .
$ rm -rf docutils-0.5*

By default, docutils tries to read configuration files from various locations. But this will ail in the App Engine environment. So you have to disable it by overwriting the default settings:

from docutils.core import publish_parts
def someview(request):
    parts = publish_parts(source=some_rst, writer_name='html4css1',
                          settings_overrides={'_disable_config': True})
    return parts['fragment']

If you are using the Django helpers, add

RESTRUCTUREDTEXT_FILTER_SETTINGS = {'_disable_config': True}

in settings.py to use the restructuredtext filter. Refer to the Django documentation on how to activate and use this filter.

CrunchyFrog 0.2.1 released

2008-06-10T00:00:00+02:00

Just some minor bug fixes and improvements for this release:

Support for recent files.
UI improvements and fixes.
SQL server and httplog backend.
Improved plugin handling.
New and updated translations.

The two new backends (MSSQL server, HttpLog backend for combined log files) are very basic for now. Database queries with these backends are ok but overall they should be seen as experimental (forgot to mention that in the release notes ;-).

Visit the GoogleCode pages for additional information and downloads.

CrunchyFrog on Arch Linux

2008-05-17T00:00:00+02:00

To all Arch Linux users out there, CrunchyFrog is now available in the user-community repository. Thanks to angvp and anyone else involved for packaging and debugging!

Debugging gobject signals

2007-09-05T00:00:00+02:00

Here's a little snippet I oftenly use to debug GObject signals:

import gobject

def test_signals(obj):
    """Helper function to test signals

    It connects all `obj` signals to a dummy function
    which simply prints out all given arguments.

    Args:
      obj: A ``gobject.GObject``

    """
    if not isinstance(obj, gobject.GObject):
        print "test_signals failed: obj must be a GObject"
        return
    def foo(*args):
        print "Signal %r emitted by %r:" % (args[-2],
                            args[-1].__class__.__name__), args[:-2]
    for signame in gobject.signal_list_names(obj):
        obj.connect(signame, foo, signame, obj)

In fact, it does nothing special, but I found it very useful during development. Example usage:

from somewhere import test_signals
test_signals(mywidget)

Pulsing ProgressBar with text

2007-08-20T00:00:00+02:00

Here's a little snippet for a pulsing ProgressBar with changing text.

import gtk
import gobject

win = gtk.Window()
win.connect("delete_event", lambda x, y: gtk.main_quit())
pb = gtk.ProgressBar()
pb.set_pulse_step(.1)

win.add(pb)
win.show_all()

count = .0

def do_pulse(pb, count):
    pb.pulse()
    pb.set_text("Step: %.2f" % count)
    count += .01
    gobject.timeout_add(200, do_pulse, pb, count)

gobject.timeout_add(200, do_pulse, pb, count)

gtk.main()

Nothing special, anyway...