townx

Links for 2009-11-04 [del.icio.us]

Thu, 05 Nov 2009 00:00:00 PST

clarity at master - GitHub
For tailing/searching logs in /var/log via a web interface

Links for 2009-11-03 [del.icio.us]

Wed, 04 Nov 2009 00:00:00 PST

Browser Sandbox - Run any browser from the web

Back on Fedora

elliot — Tue, 03 Nov 2009 18:20:34 -0600

A few years ago, I used to work on RedHat Linux, then Fedora: my first Linux machine ran RedHat Linux. Then I got sick of things breaking every time I upgraded (on one traumatic occasion, my audio stopped working after a Fedora upgrade) and switched to Ubuntu.

But in my new job, I need Linux, but also need to encrypt my laptop's hard drive. I had a go at doing this with Ubuntu, with disastrous results. Didn't really know what I was doing (I did manage to encrypt the hard drive, but couldn't figure out how to mount it during boot). Fedora makes this simple, though. So I moved across, and have to say I'm pretty glad I did. It's very similar to Ubuntu in many ways, and I don't feel like I'm missing anything. My main annoyance was SELinux, which just wanted to stop me having any kind of fun with my computer; so I switched it off.

So, if you need to PGP encrypt your hard drive, but currently use Ubuntu, have a look at Fedora instead.

Playing mp3s on Moblin (2.1)

elliot — Tue, 03 Nov 2009 18:03:51 -0600

I have an EeePC with Moblin 2.1 (release candidate) installed on it (via the packages in http://repo.moblin.org/moblin/releases/2.1/ia32/os/). By default, you don't get an mp3 decoder with this distribution. It wasn't obvious (when I googled for it) how to install the right GStreamer plugins: several blog entries suggested a variety of pretty hacky methods and work-arounds.

Then I chanced on a blog entry which mentioned that Fluendo have a free legally-downloadable RPM of their own mp3 encoder; I remember someone at work mentioning it too. I downloaded and installed this onto my netbook with no difficulties. You have to register for an account and go through a faux shopping basket to get it, but it is fairly painless. Here's the product page URL:

http://www.fluendo.com/shop/product/fluendo-mp3-decoder/

Links for 2009-10-30 [del.icio.us]

Sat, 31 Oct 2009 00:00:00 PDT

Pyjamas Book
Write Python code and compile it to Javascript; also provides an AJAX widget set and a desktop widget set (so the same app. can run in either a browser or on a desktop)

Links for 2009-10-29 [del.icio.us]

Fri, 30 Oct 2009 00:00:00 PDT

Rails in a Nutshell
Rails 3.0 book available to read for free online

Links for 2009-10-28 [del.icio.us]

Thu, 29 Oct 2009 00:00:00 PDT

Play framework
Rails-like Java web app framework (via Simon Willison)
THE CARETAKER, Persistent Repetition Of Phrases - Boomkat music download
MOUNT KIMBIE, Sketch On Glass EP - Boomkat music download

Links for 2009-10-27 [del.icio.us]

Wed, 28 Oct 2009 00:00:00 PDT

Pivotal Tracker - Free Lightweight Agile Project Management & Team Collaboration
"Tracker is a free, award winning, agile project management tool that enables real time collaboration around a shared, prioritized backlog. Velocity tracking and emergent iterations Make planning decisions using accurate projections based on past performance. Story-based iterative planning Base your software project management on proven agile methods. Real-time collaboration See what your team is doing and react to change instantly."

Links for 2009-10-25 [del.icio.us]

Mon, 26 Oct 2009 00:00:00 PDT

Why I like Redis
Interesting-sounding "data-structure server", a bit like memcached with replication support, a kind of key/value store but with the facility to store more-structured data

My favourite blog

elliot — Tue, 15 Sep 2009 18:01:58 -0500

I was going to put "probably", but then decided that, no, this is my favourite:

http://ageofuncertainty.blogspot.com/

I love the way this chap writes. Mainly about second-hand books he's found and/or read, but often about their cover art, the ephemera he finds inside them, brilliant expositions about small towns he's visited, and other bits and pieces. There's a certain gentle Englishness, quiet pathos, and calm reflection about his work which strikes just the right balance for me.

xtranormal and making movies

elliot — Mon, 17 Aug 2009 17:38:10 -0500

Above is the first movie I've made with xtranormal, called Ex-super villain first date. Not sure how I've never come across this site before, but it is a fantastic piece of web engineering: an in-browser (Flash), animated movie creation application. Basic accounts are free, and give you a pretty good range of stuff you can do. The interface is very simple and intuitive. It seems best suited to the sort of stilted absurdist style I attempted above, which took me maybe an hour to put together. Much fun.

Memory leaks and JQuery and htmlunit and...

elliot — Wed, 12 Aug 2009 10:23:16 -0500

(Just a quick note about JQuery and rdfquery and memory leaks and htmlunit and such like for lost souls googling desperately for clues.)

I was having issues with some Webdriver tests, running under the htmlunit driver: they worked locally, but ran into Java "out of memory: heap space exhaustion" issues when running as part of a Hudson build on our Linux build server. The build also worked on another virtual machine with the Internet Explorer driver, so it seemed to be something specific to the htmlunit driver (maybe). After upgrading Java, upgrading Hudson (the continuous integration server), and tweaking the Java heap size, and still no joy, I found a few mentions of memory leaks in relation to htmlunit and JQuery (the Javascript library we're using), which I decided were worth investigating.

One comment in particular seemed plausible: that the Javascript on the pages under test was causing memory leaks in the Javascript engine in htmlunit (Rhino, I think?). It seemed possible that how we were using JQuery was causing Javascript to leak memory, but only in the context of the htmlunit driver. I trawled around the various bug trackers for htmlunit and webdriver, and got a few ideas (though couldn't pin the issue down enough to raise a bug myself).

One suggested fix (in the general context of JQuery, rather than specifically with regard to htmlunit) was to ensure that any event handlers bound to DOM elements via JQuery should explicitly be removed when the page unloads. This was easy to implement: I just added a final event binding to each page which looked like this (inside a <script> tag, obviously):

$(document).unload(function() {$('*').unbind(); });

I was also using the short-hand JQuery event binding syntax in my code, i.e. things like this:

$('#save').click(function() {
   // ...
});

I changed these to the long-hand form (as this issue only started appearing when I started using the short-hand version):

$('#save').bind('click', function() {
   // ...
});

One last thing I did was ensure that I called the quit() method on any Webdriver driver instances after I'd finished with them; and closeAllWindows() on any htmlunit WebClient instances when finished with them.

Unfortunately, I didn't do this very scientifically, and made all these changes at once. But the end result was that the build started running again. So if you're having out of memory errors with Webdriver/htmlunit/Hudson/JQuery/rdfquery, you at least have somewhere to start from :)

Never Toss a Johnny

elliot — Thu, 30 Jul 2009 15:14:06 -0500

I've completed track 2 of my mathis album, Never Toss a Johnny. At this rate, the album should be finished in 5 years' time.

You can listen to it on Last.fm here: http://www.last.fm/music/Mathis/_/Never+Toss+A+Johnny?autostart

Mind your ears, though - it's a bit peaky. I used this oscillator thing towards the end, and while a nice effect, it does veer towards damaging your ears at times. And there's some loud clanging in the middle which might wake a baby or dozing pet.

(For background, this track uses only samples of music by Johnny Mathis, plus lots of effects. You might hear a fragment of Fly Me to the Moon if you're lucky.)

If you want a download, I've put some higher-quality mp3s onto S3 (including some of my other tracks), as follows:

Never Toss a Johnny by mathis
Neighbour Forgotten Like a Silent Ray by mathis
Close To You (Unfinished) by Spill Twins - a cover of the Carpenter's song, written from memory, i.e. I didn't look at the sheet music while writing it; I also didn't finish it
Nipple Soup by Spill Twins
SplashFlap by Spill Twins
Bread Sounds (Batch and Tin) by Spill Twins
Breeding in Your Sponge by Spill Twins
Ant Mansion by Spill Twins

Prolegomena 1 (Logo poem/generative art, of sorts)

elliot — Wed, 22 Jul 2009 08:29:16 -0500

make "sky 1
make "into 2

to understand
  load "me
  clean home see home see obscure
  repeat :sky + :into [ see ]
end

to see
  television setscrunch :into golden :sky
  repeat often [ right pan forward sky ]
end

to golden :offering
  output :into * :offering
end

to television
  make "me happy
end

to happy
  op "erate "the "switches
end

to often
  make "trouble :sky + :into + :sky
  output power (:into + :sky) :trouble
end

to pan
  output :sky
end

to obscure
  hideturtle
end

This is a Logo program which is also (kind of) a piece of poetry and image generator (it creates the image shown above). I did it a while back and found it an interesting exercise. I thought you might also find it interesting. Have you done anything similar? Anyone?

Example of how to put RDFa into an HTML list

elliot — Tue, 30 Jun 2009 04:52:35 -0500

I'm not going to try to explain RDF and/or RDFa here, but thought any poor suckers looking for RDFa examples might benefit from me posting what I finally worked out, with help from my colleague Rob. Namely, how to annotate an HTML ordered list (<ol>) with RDFa attributes; and how to put RDFa attributes onto form elements.

Here's the HTML page with RDFa embedded in it. What I'm representing here is a sequence of collections, and the individual collections within it:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML+RDFa 1.0//EN" "http://www.w3.org/MarkUp/DTD/xhtml-rdfa-1.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>Collections</title>
  </head>
  <body>
    <h1>Collections</h1>
    <form method="post" action="http://receptacular.org/collections">
      <ol xmlns="http://www.w3.org/1999/xhtml" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:xsd="http://www.w3.org/2001/XMLSchema#" xmlns:rec="http://receptacular.org/schema#" typeof="rdf:Seq" about="http://receptacular.org/collections">
        <li rel="rdf:_1" resource="http://receptacular.org/collections/1">
          <span style="display:none;" rel="rdf:type" resource="http://receptacular.org/schema#Collection"></span>
          <div class="collection-label" property="rdfs:label">Vague Collection</div>
          <input type="checkbox" id="collections-1-hidden" property="rec:hidden" datatype="xsd:boolean" content="false"/>
          <label for="collections-1-hidden">hidden</label>
          <input type="checkbox" id="collections-1-defaultSearch" property="rec:defaultSearch" datatype="xsd:boolean" content="false"/>
          <label for="collections-1-defaultSearch">use for searches</label>
        </li>
        <li rel="rdf:_2" resource="http://receptacular.org/collections/2">
          <span style="display:none;" rel="rdf:type" resource="http://receptacular.org/schema#Collection"></span>
          <div class="collection-label" property="rdfs:label">Archive Collection</div>
          <input type="checkbox" id="collections-2-hidden" property="rec:hidden" datatype="xsd:boolean" content="false"/>
          <label for="collections-2-hidden">hidden</label>
          <input type="checkbox" id="collections-2-defaultSearch" property="rec:defaultSearch" datatype="xsd:boolean" content="false"/>
          <label for="collections-2-defaultSearch">use for searches</label>
        </li>
        <li rel="rdf:_3" resource="http://receptacular.org/collections/3">
          <span style="display:none;" rel="rdf:type" resource="http://receptacular.org/schema#Collection"></span>
          <div class="collection-label" property="rdfs:label">Main Collection</div>
          <input type="checkbox" id="collections-3-hidden" property="rec:hidden" datatype="xsd:boolean" content="true" checked="checked"/>
          <label for="collections-3-hidden">hidden</label>
          <input type="checkbox" id="collections-3-defaultSearch" property="rec:defaultSearch" datatype="xsd:boolean" content="true" checked="checked"/>
          <label for="collections-3-defaultSearch">use for searches</label>
        </li>
      </ol>
      <p>
        <input type="button" value="Save" id="save-collections"/>
      </p>
    </form>
  </body>
</html>

Available online here: http://receptacular.org/collections

Things of note:

The doctype declaration. This is the W3C sanctioned doctype for XHTML+RDFa pages. By the way, the W3C validator will correctly validate this page, but standard XHTML validators don't (e.g. like "this one"http://nutrun.com/weblog/xhtmlvalidator-validate-xhtml-in-java/:). That's another story...
Namespace declarations on the <ol> element wrapping the list items. This is what causes standard XHTML validation approaches to fail.
The <ol> element is defined with typeof="rdf:Seq" and about="http://receptacular.org/collections". This sets it up as the RDF Seq resource.
The <li> elements inside the <ol> are Seq items, within the wrapping Seq resource. Each is each defined as a resource using the resource attribute. They are linked back to the enclosing <ol> element using the rel attribute on each. Note that the value for the rel attribute is an rdf Seq number, which orders the items within the enclosing Seq resource.
The RDF type of each Seq item is set using a hidden <span> element. Note that these elements have no text in them, but have opening and closing tags. If you just use a self-closing start tag for this element, the HTML doesn't display property.
The <span> elements use the rel attribute to mark their RDF type relationship to the outer list item; and the resource attribute to specify the location of the resource representing their type.
Each Seq item has an enclosed <div> which represents its rdfs:label property. Note that the property RDFa attribute is used to specify which property of the enclosing resource is being defined. Also note that the value of the property is inlined between the start and end tags of the <div>.
The two checkboxes define two more properties for each Seq resource: rec:hidden and rec:defaultSearch. (The semantics of the properties aren't discussed here, as I'm concentrating on syntax.) Each is defined on a standard XHTML <input> element: the relationship to the enclosing Seq item is defined with the property attribute; the value of the property is defined using the content attribute; and the data type of the literal value is defined via the datatype attribute. Any of the standard XML datatypes could be used here, or other types from other schemas.
When working with form elements which represent property values for RDF resources, you may need to change the content attribute in response to UI changes. (In the application from which this example was extracted, we use JQuery to respond to changes in the check box which set the content attribute.)

To see the RDF which can be extracted from this page, you can use the W3C's RDFa Distiller. Here's the resulting RDF:

<?xml version="1.0" encoding="utf-8"?>
<rdf:RDF
  xmlns:dist="http://www.w3.org/2007/08/pyRdfa/distiller#"
  xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
  xmlns:rec="http://receptacular.org/schema#"
  xmlns:xhv="http://www.w3.org/1999/xhtml/vocab#"
  xmlns:xml="http://www.w3.org/XML/1998/namespace"
  xmlns:xsd="http://www.w3.org/2001/XMLSchema#"
>
  <rdf:Seq rdf:about="http://receptacular.org/collections">
    <rdf:_1>
      <rec:Collection rdf:about="http://receptacular.org/collections/1">
        <rec:hidden rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</rec:hidden>
        <rec:defaultSearch rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</rec:defaultSearch>
        <rdfs:label>Vague Collection</rdfs:label>
      </rec:Collection>
    </rdf:_1>
    <rdf:_2>
      <rec:Collection rdf:about="http://receptacular.org/collections/2">
        <rec:hidden rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</rec:hidden>
        <rec:defaultSearch rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</rec:defaultSearch>
        <rdfs:label>Archive Collection</rdfs:label>
      </rec:Collection>
    </rdf:_2>
    <rdf:_3>
      <rec:Collection rdf:about="http://receptacular.org/collections/3">
        <rec:hidden rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">true</rec:hidden>
        <rec:defaultSearch rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">true</rec:defaultSearch>
        <rdfs:label>Main Collection</rdfs:label>
      </rec:Collection>
    </rdf:_3>
  </rdf:Seq>
</rdf:RDF>

Saving changes to an RDFa-enabled form like this is another challenge, for which we used rdfquery, and RDF library for JQuery. (I recommend you use the latest svn HEAD version of this library, as older versions have a bug where they ignore RDFa elements nested inside elements without RDFa attributes.) Maybe I'll get round to that another time.

Script to copy random mp3s from a directory to an mp3 player

elliot — Tue, 30 Jun 2009 04:24:41 -0500

This is a Ruby script which randomly copies mp3 files from one directory to an mp3 player. I wrote it so I could easily fill up my mp3 player from the 9000 odd mp3s I have on a different external drive.

To run it, you'll need the sys-filesystem gem (see http://rubyforge.org/projects/sysutils):

$ gem install sys-filesystem

Next, edit these variables in the script (near the top):
* source_dir to the directory containing the mp3s you want to select from
* dest_dir to the path for the directory on your mp3 player you want to copy to

Be a bit careful, as this will attempt to fill the dest_dir you specify with mp3 files from source_dir. You might end up filling the wrong disk up.

Then just run it with ruby from the command line:

$ ruby mp3s_random.rb

Note that it won't delete anything from the destination drive, and will attempt to fill all the space available. Also note that it doesn't keep trying mp3s until it finds one which will fit the last remaining space: once it tries to copy a file which won't fit, it stops. You can always run it again to see whether the next run finds a file small enough to fit.

I've only tested it on Linux, but, who knows, it might work on Windows too. (No operating-system specific commands are used in the script, as it uses Ruby for all file operations.)

The code is below, but I've attached it as well.

require 'rubygems'
require 'sys/filesystem'
require 'ftools'

source_dir = '/media/disk/music'
dest_dir = '/media/disk-1/music'

files = Dir[File.join(source_dir, '/**/*.mp3')]
num_files = files.size

stat = Sys::Filesystem.stat(dest_dir)

disk_free_space_kb = (stat.blocks_free * stat.fragment_size).to_kb

files_selected = []

while disk_free_space_kb > 0 and num_files > 0 do
  # choose an mp3
  file_num_to_copy = rand(num_files - 1) + 1
  file_path = files.delete_at(file_num_to_copy)

  num_files = num_files - 1

  # work out how big file is
  file_size_kb = File.stat(file_path).size.to_kb

  # subtract from free space
  if (disk_free_space_kb - file_size_kb) > 0
    files_selected << file_path
    disk_free_space_kb = disk_free_space_kb - file_size_kb
  else
    break
  end
end

files_selected.each do |f|
  copy_to_path = File.join(dest_dir, File.basename(f))
  puts "Copying #{f} to device"
  File.copy(f, copy_to_path)
end

How I worked out that curl is doing bad things with MARC

elliot — Wed, 10 Jun 2009 04:21:31 -0500

I work on a system at Talis which posts MARC records from customer library databases into a MARC to RDF transformer. The resulting RDF generated from the MARC is sent into the Talis Platform, where it's used to power Prism.

Over the last day or so I've been working on a bug which has prevented some records going correctly through this process. Along the way, I noticed another bug occurring somewhere between the post from the customer site into our MARC to RDF transformer. It looked as if line break characters in the original MARC record were being lost somewhere in the process. Consequently, when the MARC was pushed into the transformer, the record got spat out as invalid, as the length specified in the MARC leader didn't correspond to the length of the record (now it had lost its line break characters). (By the way, working directly with byte streams is the only way to work with MARC, for precisely this reason.)

I had a sudden insight on the way home, triggered by remembering issues I'd had with curl (the command line HTTP client) working on another personal project. On that project, I'd been trying to post RDF triples in ntriple format into my application using curl. However, the application only seemed to recognise the first RDF triple in the posted file. I couldn't understand why.

Then, when I echoed the body of the HTTP request, as received by my app from curl, I realised the issue: curl was sending the body of the request WITHOUT LINE BREAKS. As line break characters act as the delimiter between triples in RDF ntriple format, my app was only seeing a single RDF ntriple. When I tried an alternative tool to send the posts (the extremely useful Poster add-on for Firefox), the ntriples were received correctly.

Once I remembered this, I decided to do some debugging of the kind of requests curl would send if it were posting MARC records. My hypothesis was that curl was stripping line break characters from the MARC record (which is bad, as they are valid characters in MARC), and hence causing the record to be shorter than the leader said it should be.

First step was to put together something to echo and/or save HTTP request bodies. Rack is ideal for this sort of thing, so I used this little Rack web server program:

require 'rubygems'
require 'rack'

def save_body(body)
  File.open('last_raw_request', 'w') {|f| f.write(body)}
  body
end

Rack::Handler::WEBrick.run(lambda {|e| [200, {}, save_body(e['rack.input'].read)]}, :Port=>7777)

This saves the raw request body to a file called "last_raw_request".

I first posted a MARC file with line breaks in it (attached) using Poster (with Content-Type set to application/marc21) through Firefox. The MARC file came through intact and still valid.

I then posted a MARC file with line breaks in it using curl:

curl -d @marcfile.mrc -H "Content-Type:application/marc21" http://localhost:7777/

Which produced an invalid MARC file with line breaks missing.

The solution is to use the --data-binary switch when using curl to send binary data, which we're not doing when sending MARC from the customer site. Mostly this doesn't matter, but it does when the MARC record contains line break characters.

Namely:

curl --data-binary @marcfile.mrc -H "Content-Type:application/marc21" http://localhost:7777/