Side note: You need Python 2.4 or above to use generators, you’ll probably have this anyway, but it might pay to check before wondering why this stuff doesn’t work.

A Quick Introduction

At it’s heart a generator is a function which can return a value several times within it’s body. Each time the function is called it resumes from where it terminated in the previous call, with all it’s internal state intact (i.e. variables are held between calls). In Python a generator is defined by creating a function which uses the ‘yield’ keyword. Each time that a yield is hit the function will return the specified value. Let’s start with an example which defines a generator implementation of the built-in ‘range’ function:

def genrange(start, end):
    i = start
    while i < end:
        yield i
        i += 1

As you can see this is incredibly simple and behaves in the same way as the standard range function. The only difference is that when the generator method is called it will return a generator object rather than a value (in contrast to range, which will return a list of values in the range). These generator objects are iteratable so calling the ‘next’ method on the object will run your code and return the next yielded value. Because the generator object is iteratable, the way in which you are likely to use it should look familiar:

for i in genrange(0, 10):
    print i

If you want to go ahead and create a list from the sequence as you generate it, you can use a list comprehension, like so:

l = [i for i in genrange(0, 10)]

Of course, as hist comprehensions can be much more complicated than this you can use it to achieve more than just creating the equivalent list as calling the built-in range function. For example, you could filter the output elements according to some criteria, or call a method on each element returned.

A Trade Off

Although the above example is incredibly simple, it presents an interesting trade off between memory use and the speed of iteration through the ‘list’. The trade off exists because you could just build the list of elements in a separate function and then iterate through it. This is a perfectly valid approach but if there are a large number of elements you will use a significant quantity of memory to store the list. You are also more likely to double handle the data, unless you are building the list up over a period of time. Using a generator in this case is likely to be more efficient, as you are just creating the elements as you need them, so only storing a minimal amount of data at any one time.

The other side of the trade off is performance. If the process that produces your iterative data is computationally expensive, it makes no sense to calculate more values than you will need. So if you have a data processing loop, that can potentially exit early (i.e. before reaching the end of the ‘list’) a generator will be more efficient. But equally, if you require a tight loop which iterates quickly, you would want to pre-compute the values.

When considering using generators you should weigh up these factors and think which method is going to be most efficient for your situation. I think in a lot of cases, you will end up coming down in favour of the generator solution.

Another Example (Dan Brown is going to love me)

I’m going to leave you with another quick example, namely a very efficient way of generating Fibonacci numbers using a generator. Without further ado:

def fibonacci():
    p1 = 0
    p2 = 1
    while True:
        f = p1 + p2
        p2 = p1
        p1 = f
        yield f

There you go, easy. Many other Fibonacci implementations use a recursive method, which is horribly inefficient. Most other implementations will require either a class to store persistent data in, or the use of static variables, both of which are more tricky to handle from a programming point of view.

Well that’s it. I think I’ve demonstrated that generators are pretty cool. The examples I’ve presented are very simplistic, but I’m sure you can think of awesome uses for them. Go forth and code!

Here are a few photos from some tramping trips I organised over the last few weeks, the first was to the Dome Forest Park and the other was to Pohuehue Scenic Park (the last two photos).

Anyone that follows my Identi.ca feed will probably be aware that I recently switched away from Ubuntu and Archlinux and over to Fedora. This was mainly due to frustrations with Arch (I need a system which lets me get stuff done without too much overhead in looking after it) and my increasing feeling that Ubuntu isn’t going in the right direction for me as a power user/developer.

I recently wrote an article on Fedora 13 for our University LUG – UALUG and I thought I’d post a link for readers of this blog. The article focuses on Fedora 13 as a platform for developers and basically details my own Fedora 13 setup. It’s written with the aim of advocating Fedora for people new to Linux, but it also serves of my review of the latest Fedora release.

Anyway here’s the link: https://ualug.ece.auckland.ac.nz/archives/246.

Hopefully I should be posting some more interesting content here soon as I’ve been playing around with some interesting stuff. It’s just a matter of me finding time to write it up! Bye for now.

Intent i = new Intent(Intent.ACTION_SEND);
i.setType("text/plain");
i.putExtra(Intent.EXTRA_TEXT, "Testing intents on android");
startActivity(i);

These four lines of code hook into a key feature of the Android platform: Intents. The intent system is a neat way of communicating via apps and enables different apps to integrate together without knowing about each other specifically.

In this case, we are sending the ACTION_SEND intent, which Mustard is set up to handle. We set the mime-type of the message to ‘text/plain’ and put the contents of our message into the EXTRA_TEXT field. We then start the intent, and we’re done!

When the intent is started Android presents the user with a list of all the possible applications which can handle the ACTION_SEND intent. In my case this is Email, Gmail, Messaging and Mustard, when I select mustard the new message screen pops up and is populated with my message content. Hitting send posts the message and returns control to my app. Neat!

There are probably a whole host of ways you can integrate your apps with other apps on the phone using the intents system, I guess it’s just a matter of knowing what intents are received by each app and what they do.

Thanks to @macno (main developer of Mustard) and @bugabundo on identi.ca for their help working this out!

Anyway, I hope someone finds this useful. This is the first time I’ve posted about Android development, but I’ll hopefully write more about it in the future. Bye for now!

Is it just me or has there been an increasing amount of Apple fanboyism around recently? This is really beginning to annoy me. Principally, because I just don’t care. Apple are completely irredeemable in my eyes, they are worse than Microsoft, but no-one seems to see this, whatever they do!

This is really hacking me off in the Open Source media, particularly on podcasts (I’m looking at you Shot of Jaq!). Frankly, I don’t give a <censored> what Apple are doing, or what they might do in the future. I don’t listen to Open Source podcasts for this.

Then there’s Ubuntu, which seems to be moving closer and closer to the Mac in terms of looks (this isn’t necessarily a bad thing, but why can’t we be more inventive?). The Ubuntu community seems to be going
this way too. In a recent episode of the Ubuntu UK Podcast the words ‘the Mac just works’ were uttered. Hello! So does Ubuntu! (of course I can only speak from my experience, but I rarely if ever have problems).

As a solution to this growing epidemic, I propose a simple test which you should use if you are tempted to praise Apple in public, particularly in the Open Source media.

Simply answer the following five questions:

1. Name five pieces of Open Source software that Apple have contributed to the community.

2. Would you buy an iPhone over a Nexus One? Seriously?

3. Is the iPad even an original idea? (clue: No.)

4. If the Mac is so great, why is it so damned hard to use? Why can’t it be configured to work they way I want?

5. Do you like DRM? Apparently, neither does Steve. Slightly hypocritical, don’t you think?

If you can’t answer these simple questions in Apple’s favour, KEEP STUM!

Before I start: NO, UNLIKE EVERY OTHER ARTICLE ON THE WEB, PUBLISHED TODAY, THIS IS NOT A JOKE, K?!?

I’ve been looking for a new distro recently. I do this from time to time, principally because I get bored of what I’m currently running. Last time it was Crunchbang which I settled on. This time I wanted to go more advanced, so I started researching Arch Linux.

For those that don’t know, Arch Linux describes itself as:

…a lightweight and flexible Linux® distribution that tries to Keep It Simple.

I’d heard about Arch in the past from several sources and had heard that you basically have to install and configure everything yourself, but that the package manager (awesomely named Pacman!) manages software without having to compile from source (unless you want to!).

The following series of posts will be a record of my experiences installing and configuring Arch on my home desktop machine. This isn’t intended to be an exhaustive installation guide, more just a record of where I tripped up in order to aid those who come next. If you are searching for an installation guide, try the excellent article on the Arch Wiki.

I’ve separated the post out into days. Note: it didn’t actually take me a full day for each part, I work during the day and only really had a couple of hours each evening to spend on this.

Day 1: Backing Up

Before installing I wanted to make sure I didn’t trash my existing Ubuntu system and all my personal data, as I still need to do all the stuff I usually do with my machine. So I made a backup.

I’m not really going to go into how. Suffice to say I used LVM snapshots and rsync, I might write about this in a future post.

This took a while, as I have quite a lot of data. I thought it best to have a beer in the mean time, so I did.

Day 2: Making Space, Starting the Installation and Various Adventures with LVM

The next thing to do was to resize my existing LVM partition containing Ubuntu so that I had space for Arch. I couldn’t work out how to do this at first as none of the partition tools I tried (GParted and Cfdisk) could resize the partition. I eventually worked out how to do it.

First, on my running Ubuntu system I resized the physical volume with:

$ pvresize --setphysicalvolumesize 500G /dev/sda1

This shrank the space used by LVM down to 500GB (from about 1000GB on my machine).

I then rebooted into the Arch live CD (64-bit edition in my case), and ran:

$ fdisk /dev/sda

Now what you have to do next is slightly alarming. You actually have to delete the partition and recreate it in the new size. This works, without destroying your data, because fdisk only manipulates the partition table on the disk, it doesn’t do any formatting of partitions, etc.

I did this through fdisk so that the partition was 501GB (making it a little bigger than the PV just to make sure). I then rebooted back into Ubuntu and ran:

$ pvresize /dev/sda1

To allow it to use all the space. This probably isn’t necessary but I wanted to be safe.

Next, I proceeded to the installation. For some reason the Arch boot CD was really slow to boot and gave me loads of read errors, I think this might have something to do with my drive as I’ve been experiencing the same with other disks. Eventually it booted and dropped my at the default prompt.

From then I basically followed the installation guide for setting up the package source (CD) and the date and time.

I then set about partitioning the disks. The Arch installer uses Cfdisk, which is fine. I just added two partitions to my disk, a small (255Meg) one for my /boot partition and a large LVM one for the rest of the system (I like LVM and wanted to use it again on Arch).

This was fine, but I had some problems setting up the LVM through the installer, even though the user guide seems to think it can do it. Every time I tried, it would just fail on creating the Volume Group, weird.

I gave up for the evening and (you guessed it) went for a beer!

Day 3: Successful Installation

The next day I thought I’d try googling for LVM on Arch, luckily when I got in to work @duffkitty on identi.ca had seen one of my posts complaining about having problems and had given me a link to the LVM article on the Arch Wiki.

This advocated setting up the whole LVM setup manually (and guides you through it) and then just setting the partitions to use in the installer. It also gives you some important things to look out for when configuring the system. Following these instructions worked like a charm and I was able to format everything correctly and install the base system.

I then moved on to configuring the system, following the install guide and taking into account the instructions in the LVM article. Everything went pretty much fine here and I eventually got to installing the bootloader. Here I replaced the Ubuntu Grub version with the one installed by Arch. This left me having to add an entry for Ubuntu, which wasn’t difficult, I just copied the Arch one and changed the partition and file names.

Then it was time to ‘type reboot and pray’ as the Arch installation guide puts it.

So I did.

When I rebooted the bootloader came up with the Arch and Ubuntu entries. I selected Ubuntu just to check everything was OK.

It didn’t work.

Panicking and Swearing Ensued.

I rebooted and selected Arch.

That worked (thankfully).

When it had booted I logged in and opened up the Grub config file again. it turned out I mis-typed the name of the Ubuntu initrd file, that was easily fixed. Rebooting got me safely back to Ubuntu.

So now I have a functioning dual boot between my original Ubuntu install and a very basic Arch install, I think I might need some software there!

But first… beer.

So What’s Next???

Well, firstly I need to get my network connection up and running as I didn’t do that during the install. It’s a Wifi connection over WPA so that’s going to be fun. Then I can start installing software. I’ll probably follow the Beginners Guide on the Wiki (from Part 3). I was also recommended Yaourt by @duffkitty, so I’ll give that a try.

I’ll be continuing to play with Arch over the next few days and reporting my progress in follow up posts here. I’ll also be denting as I go along and you can follow all of these on my #archrob hash tag.

There’ll probably be beer too.

We’ll see how it goes, but eventually I hope to have a system I can use full time.

Bye for now! Happy Easter!

In the Electrical and Computer Engineering Deaprtment of the University of Auckland, where I work we have a little device, internally known as the ‘DonKey’. The purpose of this is to allow easy programming of Atmel AVR based microcontrollers via USB, rather than the simpler serial interface. We also have some internally developed software to program microcontrollers via the device, unfortunately this software is pretty much windows only (we did have a successful attempt to compile it for Linux, but this was quite a while ago, a better solution would be to use a native Linux application).

Internally the DonKey uses an FTDI based USB to UART chip (specifically the FT232R) to communicate with the microcontroller. This presents some problems as, despite being the basis of the programmer on incredibly popular Arduino boards, the main Linux programming tool (AVRdude) has no official FTDI support. I think this is largely due to the use of a bootloader on the Arduino boards, which negates the need of the programming tool to directly flash the board. If however you brick the AVR on the Arduino, you would be out of luck and would need a physical programmer (more on this below).

The DonKey in all it's glory.

In this howto I’ll cover how to get the DonKey working on Linux with AVRdude. Luckily, while researching how I might go about this I found that a large part of the work had been done for me, due to the fact that the Arduino also uses these chips. I found instructions on doswa.com on how to patch and compile AVRdude for just this purpose (so you could flash a bootloader to a new AVR).

These instructions work quite well for the DonKey, up until you get to running the ‘./configure’ command, I replaced this with:

$ ./configure --prefix=$HOME/.local

to setup the code to do a local install in my home directory (as I want this to be my primary version of AVRdude, but not to screw with things on the root filesystem).

Next I followed the instructions on modifiying the makefile and compiling AVRdude via the ‘make’ command. After ‘make’ I also typed:

$ make install

to install into the directory setup earlier. Now AVRdude is installed, the next thing to do is a bit of configuration, firstly you’ll want to make sure it’s on your $PATH so add the following to your ~/.bashrc file:

export PATH=$HOME/.local/bin:$PATH

and run the command:

$ source ~/.bashrc

to re-read the file.

The next issue is that you may wish to remove any copy of AVRdude that is otherwise installed (I found that sometimes my shell would run the wrong one – especially if you use ‘sudo’ to run it):

sudo apt-get remove --purge avrdude

Now, I just mentioned above that you might use ‘sudo’ to run AVRdude, well according to the doswa article you do need to use sudo when using the FTDI based programmers. I’m not sure why this is, but it’s not very useful if you want to be able to call AVRdude from a Makefile or the like.

I solved this by setting a ‘suid root’ on my AVRdude binary. For those that don’t know what this is, the suid bit is a Unix permission setting that makes any program with it run under it’s owning user rather than the user who called it. If the owner happens to be root, the program runs as root even if the user who calls it isn’t. This is probably really insecure if you do it a lot, but you should be OK in this case.

WARNING: Despite what I say, it might not be OK. Allowing any program unrestricted root access has the potential to hose your system and scatter all your data to the winds. FOLLOW THESE INSTRUCTIONS AT YOUR OWN RISK!!

So here we go:

$ sudo chown root:root ~/.local/bin/avrdude
$ sudo chmod u+s ~/.local/bin/avrdude

Now you should be able to successfully run AVRdude on FTDI based devices without resorting to using sudo every time.

But, what of the DonKey I hear you cry! Well all we have to do to support the DonKey is give AVRdude a little bit of configuration which tells it what the DonKey actually is. This can go in ~/.avrduderc, and looks a bit (well exactly) like this:

programmer
id = "donkey";
desc = "University of Auckland ECE DonKey";
type = ft245r;
miso = 1; # D1
sck = 2; # D2
mosi = 3; # D3
reset = 4; # D4
;

OK, now you should be able to successfully use the DonKey with AVRdude, using a command similar to this:

avrdude -c donkey -p m8 -P ft0 -U myawesomeavrproject.hex

Note: this command is for the ATMega8 as denoted by the ‘-p m8′, check the AVRdude manual page for the correct -p option if you are using a different type of AVR.

OK, well that’s pretty much it, I’ll post back soon regarding my other progress with some microcontroller stuff. Bye for now!

I’m very pleased to announce the release of my pet project Mu-Feeder, or at least a very early version of it. I actually released version 0.1.0 unofficially yesterday, but then found a bug which was tickled by Python 2.4, so fixed that and re-released as version 0.1.1.

If you want to try it out you can get it here, or you can download the code with:

$ bzr branch lp:mu-feeder/0.1

If you have a bug report, please post it on the Launchpad bug tracker, I’m also seeking feedback which you can send to me in a variety of ways (comment here, identi.ca or Launchpad answers page).

I hope someone finds this useful! Enjoy!

The purpose of this entry is to announce my personal Free Software project. Mu-Feeder (pronounced ‘mew-feeder’) is a project to replace the proprietary web service Twitterfeed with a piece of Free Software. Basically the idea is to take entries from an RSS or Atom feed and post them to a microblog. I figured this could be quite easily done and it would let me get used to the tools, etc involved in publishing Free Software. Basically, it’s not meant to be a complicated project, just to fill a gap and give me some experience.

At the moment the software is designed to run as a Python script executed via Cron. When it gets called it should build a summary of the top items in the feed (along with shortened URL’s) and post them to the configured microblog. I already have identi.ca support as well as (untested) twitter and StatusNet support, principally bacause they all use the same API. I’m aiming to have support for a single feed in the 0.1 release with support form multiple feeds in a subsequent release.

All the details are up on the Launchpad page at https://launchpad.net/mu-feeder and you can get the code via bzr with:

$ bzr branch lp:mu-feeder

Bug reports are welcome at https://bugs.launchpad.net/mu-feeder.

That’s about it, I’ll update on progress here as I go. Bye for now!

PS. Ironically this blog entry will be posted to identi.ca with the very service that Mu-Feeder aims to replace!

The following commands will resize an ext2, ext3, or ext4 filesystem running on LVM while it is mounted:

$ sudo lvresize -L +XXG <path to fs device>
$ sudo resize2fs <path to fs device>

In the above command you need to replace XX with the number of GB you want the filesystem to grow by and <path to fs device> by the device node (typically /dev/mapper/something).

An there you have it, done! Obviously there is a huge amount more you can do with the two tools above, take a look at their man pages for more info.

Hopefully this post will save me from having to work out how to do this every time!