Sometimes a man’s worst ideas lead to his finest moments: Ben Franklin decides to fly a kite in a storm and we get electricity (I’m paraphrasing that story), Alexander Graham Bell shacks up with his 15-year-old apprentice (who was deaf) and we get the telephone (paraphrasing again).

Now, I don’t fly kites and my wife is neither 15 years old nor deaf, but I feel you and I might be on the cusp of solving the world’s data storage needs as I, too, have a terrible idea: let’s put everything in Twitter. Everything. Music, photos, tax forms—you name it, we should be able to store the data in a Twitter stream.

Brilliance begets ‘tootfiles’

So yes, you probably have realized that storing all of your binary data in Twitter is one of the top five or six ideas of this decade, so you’re as eager as I am to begin implementation. Luckily, I’ve already gone to the trouble and written a Python script that does just this: I call it tootfiles.

Note: What follows is a detailed look at how this project was built and the various choices I made along the way. If the details of character encodings and data compression bore you and you’re only interested in seeing the final code or using the script yourself, you can check out the tootfiles project page over at my Github account. I promise I won’t judge you.

A Quick (and faulty) Look at the Numbers

Twitter allows us to post individual text messages to its service with up to 140 bytes per message. Assuming for a moment that each binary byte can be represented as one character byte in a toot, it is fairly simple to gauge how many toots it will take to represent a file given its size in bytes. For example, the standard sized Google search logo weighs in at 8,558 bytes. Divided up into 140 byte segments, you might determine that we could represent this file with 62 individual toots. That sounds like a lot of bit-sized messaging, but consider that internet celebrity Robert Scoble has over 20,000 toots to his name and still has over 90,000 followers fawning over whatever it is that he actually does.

A more amusing case study might be trying to store a music file in your Twitter stream. Doing a quick search for some legal music to test with, I’ve found a punchy little number from the year 1909 called “John, Go and Put Your Trousers On.” With this short piece weighing in at 3,586,134 bytes, it would only take 25,615 individual toots to represent it as a tootfile. I’ll admit that this is a disturbingly high number of messages to store a small music file, but life is about trudging through dead end projects and never letting it break your spirit, so we’re just going to pretend it’s not a problem. Did you hear that? It’s not a problem.

Reality Bytes

So the truth is we can’t just slice the files up and stick the pieces into a toot. Binary files are a different beast from the short, nearly plaintext messages of Twitter. To fit any possible binary byte into a string that we can store in Twitter, we’ll need to store these bytes using some type of standard encoding system.

Choosing an encoding system

The two candidate encoding systems I evaluated for this project were Base64 and Base85, with the trailing number indicating the size of the usable character set. To understand this a little better, remember that binary is base 2 (it can represent any number using only 1 or 0) and the standard numerical representation is base 10 (representing any number using 0-9). You could theoretically encode anything using any base, but the higher the base the tighter you can pack the data in.

Size Matters…

I said before that the higher the base, the more efficiently you can pack data into your string. To examine this a little closer, let us look at the real world difference between Base64 and Base85 encodings.

Base64 uses the character ranges A–Z, a–z, and 0–9 for its first 62 character slots. Depending on the implementation (and there are a few), the remaining two characters differ. In mapping binary to this encoding scheme, Base64 uses four characters to represent three bytes of data. This leads to a roughly 33% increase in size when representing a binary file in Base64.

Base85 (as defined in RFC1924) uses the character ranges 0–9, A–Z, a–z, and the 23 characters !#$%&()*+-;<=>?@^_`{|}~ to represent data. By using five characters to represent four bytes of data, Base85 typically can represent a binary file with a 25% increase in size.

To compare these encodings using our two example files from above, I wrote a script that encodes the files with Base64 (as it’s included in the standard lib) and used this pure Python implementation which does the same with Base85. You can compare the file sizes in the chart below.

There seem to be no surprises there, as the numbers back up our earlier claims with regard to encoding efficiency.

… But Character Sets Matter More

So yeah—Base85 is much more efficient; that’s the pick, right? I’ll now ask that you hold on for just one moment, as there is one more issue that we must consider: the characters used to represent our data in the encodings.

You see, Twitter has a little known ‘feature’ that does a bit of post-processing to your messages to prevent bad guys from inserting scripts into a Twitter stream. This sanitization of Twitter messages takes certain special characters and converts them into HTML entities, meaning that characters like <, >, and & get expanded into safer representations of themselves.

Sanitizing messages is great because it keeps you and I safe when we’re tooting away, but this has serious implications for our data storage scheme. You see, converting these single characters into HTML encodings means the characters now take up four bytes instead of one. Base85 includes a number of characters that Twitter translates into HTML entities, so our 140 byte messages might actually end up taking up much more than that and cause problems in the decode process.

For the above reason, its smarter for us to stick with good old Base64 encoding. Even without Base85’s character issues (get it?), there’s something to be said for having a standard Base64 implementation baked right in to most standard libraries as well.

Implementation Details

Instead of walking through every line of the code, I want to just highlight some of the other choices I made and what it means for the finished product. This is broken down into two sections—one for posting to and one for decoding files from Twitter—as the situations require different approaches. The code snippets are shown as Python functions for brevity, but you’ll notice that the final code handles things with an object-oriented design.

Posting Data to Twitter

Compression

Before encoding our data to be posted to Twitter, we should really employ some type of simple data compression. Using the standard zlib package that ships with Python, I ran our two example files from before through a quick script to see how much compression really helps us. The snippet below (shortened at the expense of PEP 8) should give you an indication of how to do this on your own.

import os, sys, zlib, base64

f = open('somefile', 'rb')
fcompressed = zlib.compress(f.read())
fencoded = base64.b64encode(fcompressed) 

The results from the run are shown in the following chart.

This chart actually surprised me a bit, as I had only hoped to shrink the file down enough to save some of extra cruft we saw after encoding. While the compressed Google logo is only slightly smaller than the Base64 encoded file, our rousing hit song about trousers from 1909 has been reduced to a startling 82% of the original file’s size. Your mileage will vary on this, as some file formats compress better than others while many formats are compressed by default, but it’s clearly a huge win to include a bit of fast compression to our code.

Get Your toot in the Door

So now that we have a decent handle on our compressed and encoded data, it’s time to think about getting it tooted. Before posting the messages to Twitter, we’ll first need to split the large string of data into 140 byte chunks. The below function does just this, given a data string and defaulting the slice size to 140.

def segment(data, n=140):
  ''' Given the encoded string, slice it into twitter ready array elements '''
  tootcount = int(math.ceil(len(data)/float(n)))

  slices = range(tootcount)
  slices.reverse()
  
  return [data[i*n:(i+1)*n] for i in slices]

You’ll notice that we reverse the range in the segment() function, as we want to build the list from tail to head. This reverse order makes reading the individual toots easy later on in the decode section, as we can simply iterate through the stream grabbing the toots in order.

Including some type of header helps signify the start of a file, along with a bit of metadata. Header information is appended to the list (to be posted last) using the following format:

tootlist = segment(data)

# Header Information
header = "|Tootfile:'%s' MD5:'%s' Count:'%s'|" 
         % (filename, md5hash, tootcount)

tootlist.append(header) # Insert the header

With this header, we’re helping future decode runs by including three important pieces of information: the name of the file, the MD5 hash of the file data (for integrity checking), and the number of data segments that will follow the header.

Publishing the Segments

Publishing the toots to Twitter is accomplished using the Twitter-python library. It’s a fairly simple library to use (despite the uppercase method names), so I won’t get into the details of this other than to provide a quick look at the publish() method:

def publish(data, username, password):
    api = twitter.Api(username, password)
    for toot in data:
        for retries in range(5):
            try:
                status = api.PostUpdate(toot)
                break
            except:
                if retries == 4:
                    raise Exception('Unable to post a segment. Quitting.')
                else:
                    time.sleep(1)
    print "Finished."

There’s some rudimentary fault tolerance built in, as I encountered some timeout issues when tooting a larger file. The run will fail if it cannot post a single toot with five attempts.

Retrieving Data from Twitter

This part of the project was a bit tricky, as the Twitter API does not seem to provide a good way to access a single user’s stream of toots for more than a single page. My preferred method was for a user to only need to know the ID of a tootfile header when retrieving files.

The first issue I ran into was not knowing the username to grab all of the toots from. I ended up using simplejson and using a call to http://twitter.com/statuses/show/<TOOTID>.json to grab enough info given a single Twitter message ID to do the rest of our business. The key piece of info I needed was the Twitter username that owns the toot. I could have settled and required a full URL path to the header (which includes username), but this was a choice I made—right or wrong—as to how I wanted this implemented.

After reading a blog post from Scott Carpenter, I decided to just use the fantastic Python HTML scraping library BeautifulSoup to grab the tweets out of the given users stream. While Scott’s goal was to archive all of his Twitter messages, I was able to simplify the script to grab just the info we needed. The modified section is below.

def walk(username, headerid, tootcount):
    tootlist = []
    grabbedtoots = 0
    
    url = 'http://twitter.com/%s?page=%s'
    re_status_id = re.compile(r'.*/status/([0-9]*).*')
    
    # find the max number of pages, based on 20 per page
    maxpages = int(math.ceil(int(tootcount)/20.0))

    for page in range(1, maxpages+1):
        f = urllib.urlopen(url % (username, page))
        soup = BeautifulSoup(f.read())
        f.close()

        toots = soup.findAll('li', {'class': re.compile(r'.*\bstatus\b.*')})
        if len(toots) == 0:
            break

        for toot in toots:
            # Do we need more toots? If so, keep going
            if grabbedtoots < int(tootcount):
                m = re_status_id.search(toot.find('a', 'entry-date')['href'])
                status_id = m.groups()[0]

                # Look for the message directly after our header
                if(int(status_id) < int(headerid)):
                    data = str(toot.find('span', 'entry-content').renderContents())
                    tootlist.append(data)
                    grabbedtoots += 1
            else:
                break

        # one second delay between pages 
        time.sleep(1)

There’s nothing too fancy going on here—just some pulling down of the Twitter pages and scraping out the messages. This implementation of walk() is naive and assumes that you only have one tootfile in your stream and that it is located at the front of your stream. I consider this a bug, so it will be fixed eventually in the actual project.

Reassembling the data is fairly trivial once you have the data in a list. The following snippet works backwardly compared to the encryption, peeling back the layers of compression and encoding before leaving you with a string representing the file data.

def decode(tootlist):
    """Decodes the raw data given a list of toots"""
    data = "".join(tootlist)
    compressed_data = base64.b64decode(data)
    rawdata = zlib.decompress(compressed_data)
    md5hash = md5.new(self.rawdata).hexdigest()

With that, we’re done. The decode process can now check the md5 hash of the assembled data against what’s in the header and write out the rawdata to either a file or standard out.

Grab the source and fork it for your own needs

If you’re interested in the full (and slightly more proper) Python implementation of tootfiles, you can grab the source code and documentation over at its Github project. The script can either be used as a standalone command line encoder, or you can use it as a Python library.

I’m releasing the script with an MIT license, which basically means you can do whatever you want with it.

This is mostly a proof of concept release, as the whole idea of storing meaningful amounts of data in 140 byte segments is absolutely silly. Even as I write this, I’m aware of a few bugs that would occur if someone were to try and use it heavily. With that said, I’ll continue to iron things out and I will gladly accept any patches as well.

If you’re interested in seeing some actual postings from the script, check out tootfiles on Twitter. Have fun, and please don’t spam your followers with a tootfile—especially if you’re someone that I’m following. ;)

As someone who enjoys finding better ways to do things, I wanted to share a new tip I learned this weekend. Do you find yourself wrangling a handful of all too similar sprites, only varying by color? A useful pattern in game development is to keep the number of sprites in your project to a minimum by performing what’s called a palette swap of the colors in the images. Fortunately for us, Ricardo Quesada’s cocos2d iPhone game engine makes it a piece of cake to color your textures on the fly.

Credit for this idea goes to Codemattic from the cocos2d-iphone discussion group, but I thought it was worth expanding upon to spread the knowledge and throw some code around while I’m at it.

Let’s gather our tools

Note: Code snippets in this article assume that you’re using version 0.7.1 or later of cocos2d. If you’re interested in grabbing the final product, you can download the finished project from GitHub. You can also browse the source tree or fork it for yourself at the GitHub project page.

To implement this demo, you will need to have a working cocos2d project to hack with. My code snippets will live in the SimpleGame project that I co-maintain over at GitHub. If you’d like to play along, grab a copy of SimpleGame and have it handy.

Also, grab a copy of the simple humanoid sprite to the right—we’ll be adding it to our project to demonstrate the effect. You can either right-click on this link or save the image using your browser.

Implementation

Import your sprite and draw it

The first thing you’ll want to do is open up the XCode project you’ll be working with. Feel free to use your own cocos2d 0.7.1+ project, or use the provided one from above. From this point on, I’m going to assume you’re using the provided project. If you have your own, I expect you’ll know more about where the code should go than I do.

Once you’re in XCode, right click on Resources in the project pane to the left and choose Add -> Existing Files... from the context menu. Browse to find your copy of the humanoid sprite from above and click Add. On the next dialog, click the checkbox next to Copy items into destination group's folder (if needed) and then choose Add again.

Now that the sprite image is added to your project, let’s write some code to display it on the screen. We’ll be working in the GameLayer portion of the GameScene, so open up GameScene.m and find the init method on or around line 27.

Modify init to look like this, changing “ColorSprite.png” to whatever name you might have saved the image as:

- (id) init {
    self = [super init];
    if (self != nil) {
        isTouchEnabled = YES;
        
        // Add our original sprite to the layer
        Sprite *plainSprite = [Sprite spriteWithFile:@"ColorSprite.png"];
        [plainSprite setPosition:cpv(150, 160)];
        [self addChild:plainSprite z:0];
    }
    return self;
}

Assuming no typos, and that you’ve referenced the sprite filename correctly, hit Build and Go in the XCode toolbar to run the project. Once the app is running, choose the Start Game option on the menu and you’ll something like what I have below. (I took the liberty of changing the background of the layer to make the screenshots clearer)

Color away

We’re in a good spot now to see the power of coloring our sprite. To accomplish this, we’ll add another copy of our sprite to the layer and employ the setRGB::: method to it.

Add the following block right below the three lines you added previously to the init method.

        // Add a red sprite onto the layer
        Sprite *redSprite = [Sprite spriteWithFile:@"ColorSprite.png"];
        [redSprite setPosition:cpv(200, 160)];
        [redSprite setRGB:255 :0 :0];
        [self addChild:redSprite z:0];

Notice the only difference in the above code is our use of the setRGB method on the sprite. Using standard RGB color values, you can theoretically tint your sprite into 16,581,375 colors. For this demo, we’re sticking to the primary colors.

Also, and this is important, the math behind this method is ruthless—it works nicely with a white (RGB 0,0,0) pixel, but might do scary things to your orange boot sprite. I trust that you will use this tastefully.

Before you run the above code, let’s add two more sprites right below it to really spruce things up.

        // Add a green sprite onto the layer
        Sprite *greenSprite = [Sprite spriteWithFile:@"ColorSprite.png"];
        [greenSprite setPosition:cpv(250, 160)];
        [greenSprite setRGB:0 :255 :0];
        [self addChild:greenSprite z:0];
        
        // Add a blue sprite onto the layer
        Sprite *blueSprite = [Sprite spriteWithFile:@"ColorSprite.png"];
        [blueSprite setPosition:cpv(300, 160)];
        [blueSprite setRGB:0 :0 :255];
        [self addChild:blueSprite z:0];

These two blocks will draw a green and blue copy of our sprite. If you’ve typed everything correctly, you can run the project now and see the brightly colored fruits of our labor.

Conclusion

I hope you find this trick as useful as I have. While we’ve covered a simple example where the entire sprite is tinted to one color, one could (and should) use this approach to create complex, layered sprites with interchangeable parts.

Trouble? Let me know in the comments, but feel free to grab a copy of the finished project if you’re interested.

Update 4/29/2009: Cocos2d-iphone v0.7.2 has been released which includes two new actions, TintBy and TintTo, which might be of interest to readers of this post.

This release is a joint venture between Will Larson and myself. If you’re not familiar with Will’s work, just know that he’s an ace coder with a reputation for quality.

The touchDefense website is located at http://touchdefense.com. The game is available for purchase at the introductory price of $3.99 on the App Store.

So what’s it all about?

touchDefense is a tower defense game for the iPhone OS. If you’re not familiar with the genre, this Wikipedia entry should give you the rundown. The game features touch controls, allowing you to simply drag towers onto the playing grid. There are five levels included in 1.0, but we expect to add more in an upcoming free update.

Monocle Studios LLC

Preceding the release of touchDefense, Will and I formed a software development company, Monocle Studios LLC. We’re still working up plans for the future of Monocle Studios, but rest assured that includes both exciting new releases as well as free content updates to touchDefense.

My history with version control

I have used Subversion for nearly all programming projects at my day job for the past few years. I manage all of my repositories on an offsite server (my Dreamhost server, actually) and it serves me well as both a version control system and a great offsite backup solution. Working on a development team of one, all of my changesets are simple (no merging necessary).

As for git, I first began working with it a few months back after a some friends set up a Github repository for an excellent javascript tower defense implementation. After coming up with a few small patches, I setup my first Github account and struggled through the process of learning an entirely new set of commands to merge what amounted to a few lines of code. A few projects and many more commits later and I’m much more comfortable with the various git commands.

Why distributed matters

My needs are vanilla enough that Subversion should work well for anything I have going on, yet I now prefer working with git. Instead of giving you a huge narrative, I deem it proper to list off the reasons why a DVCS works best for me:

• No need to setup a server - While the initial setup of git might be complicated, I no longer have to login to my remote server to initialize repositories. This low barrier to entry is a huge win for me.
• You always have the whole tree at your disposal - No need to go back to the server for diffs and the like—each team member (or at least each of your development machines) has the entire repository sitting on the file system. This is also useful in that a server can fail, temporarily or permanently, and you’ll still have a complete copy of your repository. It also matters because…
• Local commits (aka offline commits) - You might not realize you need these, but you do. Why should it matter whether you’ve paid TMobile $10 when you want to add a changeset? Just add and commit whenever and push the stuff up when you get back to a wired location. I find myself breaking commits up into smaller and more specific changesets.
• Branching works - The last thing I would want to suggest is that branching in Subversion does not work, but it always felt like too much work for me. In git, the concept of branching is key, with every copy of the repository basically representing a branch. Having multiple branches in the repository is also a snap, and since you have the whole repository at hand, it is easy to switch between branches as well.

So while I’m a bit late to this conversation, that’s my take on the matter.