Software used (on Mac OS X 10.5):

• tsMuxer
• Disk Utility
• newfs_udf
• hdiutil

Hardware used:

• Macbook Pro
• Blank CD/DVD Media
• PS3

Files used: * MKV file with AC3 audio stream and H264 video stream

Attempt 1 (successful): Load the MKV with tsMuxerGUI, select M2TS muxing. If the video profile is above 4.1, lower it to 4.1 (as the PS3 cannot support higher than an H264 level 4.1). Generate the m2ts and meta file, burn them both to a CD/DVD. This will be readable by the PS3 as a data disk – it will not autoplay, but you can access it and play it. To me, this is not an ideal solution as it does not support menus, chapters, or seeking.

Attempt 2 (unsuccessful): Same deal as before, but select “AVCHD disk”. This option will create a BD friendly file structure (folders named BDMV and CERTIFICATE). The trick here is to burn the disk as UDF 2.5 (this is not super easy on OS X or Linux). I wasted many CDs trying to regular ISO9660. Following the instructions here, you must create the image and format it as UDF 2.5.

dd if=/dev/zero of=myfile.img bs=1k count=716800
newfs_udf -r 2.5 myfile.img -v volume_label
hdiutil mount -nobrowse myfile.img
cp -R /path/to/avcdh-files/ /Volume/volume_label/
hdiutil unmount /Volume/volume_label

In plain-speak, create an empty (large) image, format it to UDF 2.5, mount it, copy the BD-compatible files into the volume, and unmount. You then use Disk Utility to burn the resulting image. No success here, but at this point I was so close, I could taste it.

Attempt 3 (successful): Identical procedure as Attempt 2, with one important exception – the image you create with dd must be sized in even increments of 1GB – that is, count==N*1024*1024.

A few things to try next:

• Menus
• DTS audio stream
• Subtitles

Anyone wanting to test this out with a super high quality 1080p rip, I recommend Big Buck Bunny. You’ll need Handbrake to convert it to a compatible container format if you download the AVI (tsMuxer doesn’t like AVIs).

Warning pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.

Now there are plenty of things you can do to improve the security of the unpickling process. Python lets you subclass pickle.Unpickler to give the user finer grained control over what gets unpickled. This is a fine approach (a nice example here), and will work for most, but I will give my take on the issue.

For most of the applications I write that use pickle, I’m just looking for a way to store arbitrary Python data as a string. One example might be storing small data objects on S3, or perhaps implementing user sessions for a webapp. Either way, I should be able to trust my own data for unpickling, but it’s always best to be double-extra-sure when dealing with something where you can blindly execute arbitrary bits of code (think, the evil eval method).

So, for my case, I simply want to verify that the pickled data I stored is coming back to me unmodified. My solution: sign the pickled data. Using the same signing method as AWS, I present the following:

import hmac
import hashlib
import base64
from cPickle import dumps
 # The unsigned pickled data
string_to_sign = dumps({'foo':"bar",'spam':"eggs",'the answer':42})
 # The signature object
signature = hmac.HMAC(key="my application's super secret key",
    msg= string_to_sign, digestmod=hashlib.sha256)
 # The signed string: store this
signed_string = string_to_sign + base64.encodestring(signature.digest())

import hmac
import hashlib
import base64
from cPickle import loads
 # Break up the signed string into message and signature
signature = signed_string[-45:]
message = signed_string[:-45]
 # Calculate the signature of the message
msg_sig = hmac.HMAC(key="my application's super secret key",
    msg= message, digestmod=hashlib.sha256)
 # See that it matches the given signature
assert base64.encodestring(msg_sig.digest()) == signature

Py.Test, from the vaingloriously-named “py” module, is my unit testing framework of choice (I have written about it before). It provides a convenient way to collect tests and to write generative tests (which are super useful) for unit testing. After getting a few sets of unit tests rolled out for our API, we recognized that we would need some higher level tests – so called functional, or acceptance tests.

Functional Tests

Functional tests describe high-level tests that rely on the interaction of many components of the system, whereas a unit test will only test smaller, lower level components. For example, one (very high-level) functional test for an XML API would be to see that the resulting XML is well-formed. The well-formedness of an XML response from an API request is dependent on several components of the system. It requires proper request parsing, validation, error handling, template rendering, et al. A more typical test might be to see that the number of items returned by the API does not exceed a user-provided maximum, i.e., if the user requests http://api.example.com/?[request params]&max_count=10, no more than 10 results are shown.

Now, how to go about running these tests. The number of functional testing frameworks is too great to mention (here’s a bunch), but one that is well known and widely used is Selenium. It is written in Java and can do some pretty fancy stuff. However, one big drawback of Selenium is it’s weight. It’s heavy – it is Java after all, and requires a client server (whether you sacrifice your own cycles or a remote server). For the simple functional tests we were writing, it was completely overkill. After searching around for a Python functional testing framework (or at least something lighter than Selenium), it occurred to me that I could just use the test-collecting abilities of Py.Test plus some additional libraries. And that’s what we did.

Bottom Line

Mix together PyXML, Urllib2, and Py.Test and you have a pretty powerful (and portable) testing suite in Python. PyXML extends the built-in ‘xml’ module with some really nice packages including an XPath parser which I love.

Exempli Gratia

Consider an API that has a “users” noun, and just one verb “show”. We will allow one optional parameter order_by and one required parameter max_count. An valid URL would look like http://api.example.com/users/show?max_count=10&order_by=date.

We’ll start by creating the class that will contain the tests, and writing a function to get an XML doc given some url parameters.

import urllib2
from collections import defaultdict
from xml.dom import minidom
from xml import xpath
class TestUserNoun:
    def get_xml_doc(self,url_params):
        url = "http://api.example.com/users/show?"
        url += "max_count=%(max_count)s&order_by=%(order_by)s"
        url_p = urllib2.urlopen( url % defaultdict(str,url_params) )
        doc = minidom.parseString( url_p.read() )
        url_p.close()
        return doc

Now we can start writing some tests

class TestUserNoun:
    ...
    def test_user_count(self):
        # Test several values of max_count
        counts = (5,10,15,20)
        def count_users(n):
            # Test that the number of results returned is less than or equal to n
            doc = self.get_xml_doc({'max_count':n})
            user_count = len( xpath.Evaluate('/xpath/expr',doc.documentElement) )
            assert user_count <= n
        for c in counts:
            yield count_users,c
    def test_order_by_date(self):
        # See that each item is older than the previous one
        doc = self.get_xml_doc({'max_count':10,'order_by':"date"})
        items = xpath.Evaluate('/xpath/expr',doc.documentElement)
        # Get the date of the first item
        last_date = xpath.Evaluate('@date_attr',items[0])
        # Compare the date of each item to the previous one
        for item,i in zip(items[1:],range(len(items[1:]))):
            item_date = xpath.Evaluate('@date_attr',item)
            assert item_date <= last_date
            last_date = item_date

• Does the controller handle the requests properly? What about missing/extra parameters?
• Are errors handled properly?
• Is the resulting XML valid? This is implicitly done by parsing the XML document
• Does the resulting data correspond to the request parameters? This one will require the most tests to be written – don’t forget about generative tests!

A powerful test suite means a robust application. When you have a nice set of tests, you can push your code with confidence – and believe me, that is a very rewarding and relieving feeling. Writing this API has been an extremely rewarding experience, and probably the most educational thing I’ve done programming-wise since I wrote a cross-browser javascript event library like 5 years ago.

So go forth, programmer – embrace testing and empower yourself.

-David

Idea

Provide an interface for storing/retrieving serialized data on S3

Pretty simple idea, pretty simple implementation. Thanks to the S3 interface provided by Boto, things were a lot easier. I’m going to keep this open source under the MIT license. You can check out the code on GitHub repository – please feel free to fork, improve, submit, etc.

Overview

A quick walkthrough of the code will reveal truly how simple this is. The Client class provides basic CRUD methods for interfacing with S3: put, get, update, delete. The put and update methods store a timestamp as the “expires” header for the file to keep track of cache expiration. Also these two methods write a “type” header to the meta-data so CloudCached knows how to de-serialize the file.

class Client:
"Here's the class schema"
    def get(self, key)
    def put(self, key, value, time_to_expire=3600, replace=False)
    def update(self, key, value, time_to_expire=3600)
    def delete(self, key)

The put method checks the md5sum to make sure everything went through cleanly (maybe a bit costly, but worth it in my opinion). cPickle is used in favor of pickle for obvious reasons (it’s much faster).

Results

Some very early tests show that this might just be usable.

    CloudCached Benchmarks (10 runs)
    --------------------------------------------------------
    Test                 |  Average (s)     | Total (s)

    --------------------------------------------------------
    GET integer          |  0.0283360004425 | 0.283360004425
    GET string (32 byte) |  0.0315794944763 | 0.315794944763
    GET string (512KB)   |  0.1265994787220 | 1.265994787220
    PUT integer          |  0.0650457143784 | 0.650457143784
    PUT string (32 byte) |  0.0563205003738 | 0.563205003738
    PUT string (512KB)   |  0.1773290872570 | 1.773290872570
    --------------------------------------------------------

Advantages

• Highly distributed. S3 data is distributed across multiple availability zones and could therefor be utilized by an application running across multiple availability zones.
• No size limit. Unlike the physical limitations of a memcached machine (or cluster of memcached machines), S3 does not have limits on the number of files (caches) you can store. Also, with S3, you can write files from 1 byte to 5 GB (although I think a 5GB cache file would defeat the purpose).
• Parallel read access. If applicable to the application, cache reads can be largely parallelized which could potentially give linear speedup to the cache loading.
• No server necessary. Since the application is reading and writing directly to S3, there is no need to a “cache server”. This could lead to a great deal of savings for people running multiple memcached machines. Memcached servers typically have a large memory capacity which means a m1.xlarge or c1.xlarge EC2 instance (assuming it’s running in EC2).

Considerations

It’s going to be hard to beat the speed of memcached. As far as speed is concerned, I’m using built-in Python stuff including urllib, httplib, xml.sax, etc (all of which are used by Boto). It might be worthwhile to write a C implementation of the S3 communication methods (but maybe not). The most costly part of this code aside from network communication is probably the serialization, and since cPickle is used there is not really improvement to be made there.

It might be cool to couple the meta-data with SimpleDB.

I registered cloudcached.com in case this gains some momentum. I will post updates and benchmarks there as they arrive.

-David

Energy per spin plotted against magnetization

I leave the interpretation to you. The best part of this is that I can do these MPI runs without burning a hole in my lap (the MacBook gets rather warm). -David

The other day, I noticed my 8GB USB volume that I use for temporary incremental backups was quite full. Curious, since the folders I back up to that volume do not total but 200MB or so, and rsync was supposed to be doing incremental backups (link-dest ftw).

After a little searching around, I found someone who had a similar problem (and a solution). When you format a volume with OS X it will, by default, ignore file ownership (the linked article explores why this is perhaps). This proves to be a problem for rsync which considers file permissions and ownership as part of the file stat (as it should). Luckily the fix is easy – “Get Info” for the volume in question, then at the bottom unselect “Ignore ownership on this volume”

You will probably want to delete any backups that have been created (since they won’t have the correct file ownership). Source: Terminalapp.net

For my Master’s thesis, I’m going to be running a lot of MPI code, and naturally I need a place to run it. Let me first say that my university has an excellent high-performance computing center run by one of my committee chairs that is more than capable of serving my needs – but yet, I am unfulfilled. With our scheduling system, there is a “backfill” that is always available for running small jobs (like the ones I run), but for my thesis, I want to test the massive scalability of an algorithm (Replica Exchange). When I mean massive, I mean massive – think 1000 compute nodes or more.

Big ideas, people.

In order to satisfy my need for a massively parallel platform, I looked no further than Amazon EC2. As should be apparent from many of my previous posts, I have been doing a lot of work with Amazon’s cloud services – both school and work.

A few weeks ago, I started an MIT-licensed open source project on GitHub aptly named EC2MPI. Today I made a major step forward with this project which was the motivation for this post. I finally have everything configured properly and got my first no-hassle MPI cluster up and running.

The script I wrote (EC2MPI), is written in Python and presents an interactive prompt to the user. You select the architecture (i386 or x64), the number of instances, and I also have support for user-defined SSH keypairs (not AWS keypairs) for cluster security. The instances are spawned, and EC2MPI sets up the SSH keys, as well as MPI configuration. It is so freaking sweet.

I wanted to share some issues I’ve had so far while developing this and how I solved them.

Intra-EC2 communication – For this, I needed each instance to be able to talk to one another for point-to-point as well as collective communication. My solution for this was to allow the user to generate SSH keypairs which were stored in a private S3 bucket (owned by the user). My user-data script sent to the instances took care of downloading and installing the keys upon startup.

Shared storage among instances – In order to run MPI code, the nodes in the cluster need access to a shared storage volume which will contain binary files compiled by MPI. Since EC2 has no shared storage (for now), I had to find an alternate solution. The solution I settled on was to use s3fs: a fuse-based filesystem which allows you to mount an S3 bucket as a volume. Reading and writing to the shared volume is pretty slow (unless it’s cached), so for certain kinds of code this might not be ideal. However, I believe it is the best solution for now. I imagine one day Amazon will add a feature to the Elastic Block Storage volumes that allow them to act as shared volumes.

Starting up and tearing down clusters – I used Amazon SimpleDB to keep meta-data about the cluster: how many instances are in the cluster, internal/external IP addresses, etc. This is also how I define the master node and worker nodes. This will allow me to add features such as adding and removing instances from a cluster without having to tear the whole thing down. Also I did all startup config with a user-data script so the script does not have to log into each instance upon startup. This allows the clusters startup to scale well.

Check back soon for some benchmarks and more detailed write-ups as the project progresses. First, I need to get my maximum number of instances increased (right now I can do 20 max). Fast times ahead, friends.

-David

    #!/bin/bash
    echo "Choose Account:"
    read account
    base=grep $account ~/.ec2/README -i | awk '{print $1}'
    if [ ! -n "$base" ]; then
        echo "Sorry, that account does not exist"
        return
    fi
    declare -x EC2_CERT="~/.ec2/cert-$base.pem"
    declare -x EC2_PRIVATE_KEY="~/.ec2/pk-$base.pem"
    echo "EC2 environment updated"

Requires that you your private keys/certs in ~/.ec2, and they are named cert-{something}.pem and pk-{something}.pem. Also, you need a README file in ~/.ec2 that looks like

    something account1
    something-else account2

-David

If you’re a university researcher, student, or professor, visit http://aws.amazon.com/education for more information. One of my professors talked to me about giving a seminar on cloud computing in the fall. I believe these types of grants are issued for that sort of thing as well.

Totally putting this on my CV.

Set the “Content-encoding” header to “gzip”. Really, it’s that easy.

Kthxbye.

Well, since you came all this way, I’ll give a little more detail. First, make a file.

Now gzip it.

Upload it.

Find a utility that can modify file headers on S3: S3Hub (OS X), Cloudberry S3 Explorer (Windows), or any of the various 3rd party libraries.

Set the Content-type header to whatever the appropriate content type is: text/plain, text/css, text/javascript, image/jpeg, etc.

Set the Content-encoding to gzip.

Pat yourself on the back.

Here’s three versions of a text file I made and gzipped. Note that with appropriate headers, file extensions don’t mean squat.

1. http://mumrah-dot-net.s3.amazonaws.com/gziptest.txt.gz
2. http://mumrah-dot-net.s3.amazonaws.com/gziptest.txt
3. http://mumrah-dot-net.s3.amazonaws.com/gziptest

Go ahead and download one – you’ll see that the file is actually gzipped and your browser is doing the deflating on the fly. This is the same effect producted by mod_deflate in Apache.

-David