The Bloggroll

Add subtitles to a DVD

Sun, 05 Dec 2010 17:29:46 +0200

Many people want to add subtitles to a DVD in their own language, and often DVD disks don’t come with subtitles in a language you want. For instance, if you’re staying in a foreign country and want English subtitles added to your rental movie. Or in my case, the library at the local Alliance Française has a lot of interesting movies completely in French, without English subtitles, and my french isn’t yet good enough to fully understand everything that is being said.

What is interesting is that there are sites on the internet that offer downloadable subtitle files for many movies. Simply type the name of your movie, plus the word “subtitle” into google and provided the movie is popular or well known enough you should be able to find a site offering the subtitle file for download. Mostly it seems that these subtitles were extracted from a DVD owned by someone else with the right languages. In other cases, it is purely fans of the movie who loved it so much and wanted to translate the words of the movie into a language that they know so that others can also enjoy the movie. I’m not so sure about the legality of the subtitles obtained in the first case, but certainly an argument may be made that since you own or have legally rented a copy of the movie it does seem to fall within “fair usage” to watch this movie with subtitles in a language you require.

So, you may be lucky. You may find the (fan-written) subtitle file that exactly matched your movie. By this I mean that the person who created your file had exactly the same version of the DVD, so that the lead in at the beginning of the movie is exactly the same length and in this case the subtitles remain perfectly in synch with the video.

However, don’t expect to watch the movie in your living room just yet. Most stand-alone DVD players that I’ve used do not have a facility to specify an external subtitles file, but almost all of the software based players on your computer do have this facility. So provided you got lucky, and got the correct subtitles file that matched your DVD, and are also content to watch movies in front of your computer that will be sufficient and you don’t need to read further.

Mostly though, the subtitles file doesn’t have the right timings for your DVD, or you might really really want to watch it on your TV in the lounge rather than with bowls of popcorn balanced on your lap in front of the computer. The guide that follows is for people who are running Linux, are comfortable with the command line, and who wish to hardcode ¹ subtitles into a XVID file for watching on a (living-room) player that can play DIVX movies. If this is not exactly what you want there still may be something for you in this blog post. I’ll outline my six-step method:

Rip (extract) contents of the DVD disk.
Attempt to play DVD file with subtitle files on desktop.
Create a XVID file without subtitles.
Transform the subtitle file so that subtitles are synchronized with the video.
Create a XVID file with subtitles hardcoded in.
Break up this file into smaller segments for players that cannot handle large files.

I myself don’t follow the above recipe every time – depending on the circumstances some of these steps may not be needed. And you may very well want a different outcome – it may be enough for you just to align the subtitles with the video, or you may want more such as recreating a DVD with subtitles. You may even despise XVID files with harcoded subtitles in them!

If you are running Debian/Ubuntu you’ll need a lot of packages to be installed – at least: mplayer mencoder ogmtools libdvdcss dvdbackup gaupol ffmpeg gstreamer0.10-x plus dependancies and probably gnome-codec-install.

The principal tools involved are the wonderful mplayer/mencoder command line tools :- they give fine control, are scriptable, and are compatible with each other in terms of command line flags.

For step 1, extract the DVD to your hard disk:

dvdbackup -i /dev/dvd -M -o name_of_movie

This will create a folder called name_of_movie in the current directory. If your DVD-device is not /dev/dvd put the correct device name there.

In step 2, you’ll need to work out which of the titles on the DVD disk contains the movie, start by typing:

mplayer -v -dvd-device name_of_movie/VIDEO_TS/ dvd://1

Keep on increasing the number at the end – from 1 to 2 to 3 etc until you find the actual movie and not just the adverts and bonus features. In the examples that follow I refer to title 1, you will need to replace that with your correct title number.

The next thing to do is test the downloaded subtitle file:

mplayer -v -dvd-device name_of_movie/VIDEO_TS/ dvd://1 -sub downloaded_sub.srt

The subtitle file doesn’t have to end in .srt – mplayer supports multiple formats. Make a careful note if the subtitles are in synch, namely do the spoken words match the subtitles. Press the right arrow key on your keyboard to advance the video. Ensure that subtitles remain in synch until the end of the movie.

For step 3, create an initial XVID file without subtitles. Yes, I know this can be lossy. And it really is not a required step if your subtitles are already in synch, if they are you can move straight on to step 5. However, if you do need to adjust the subtitle file to match the video, then in order to do so it may be necessary to work with the entire movie as a single file, instead of multiple individual VOB files.

So, here is a simple two-pass recipe for encoding a movie to XVID without subtitles:

mencoder -idx -dvd-device name_of_movie/VIDEO_TS/ dvd://1 -ovc xvid -oac pcm -xvidencopts pass=1:bitrate=1200:aspect=16/9 -o /dev/null

mencoder -idx -dvd-device name_of_movie/VIDEO_TS/ dvd://1 -ovc xvid -oac pcm -xvidencopts pass=2:bitrate=1200:aspect=16/9 -o movie.avi

Note the bitrate in the above call (1200) can be adjusted as required (higher numbers mean higher quality but also larger files). You will obviously need to make sure this call has the correct path for your VIDEO_TS folder, and also for the title track from the DVD (replace the dvd://1 with the appropriate title). Experienced mencoder users will possibly also want to add extra mencoder flags. Transcoding to XVID can take some time, depending on the speed of your computer.

Next up is step 4 – adjusting the subtitle file if that is needed. To do this, I prefer using the gaupol application under Linux. So, open gaupol and within gaupol select your subtitle file, and at the bottom of the gaupol screen select your .avi video file that you produced in the previous step. The trick to adjusting is to note exactly the time at which the first useful spoken words occur for which you have a matching subtitle, and the same for the last spoken words (unfortunately, knowing the last spoken words may spoil the end of the movie for you!). So, if you click the green play icon in gaupol it will play the video file with a clock at the top of the screen which you can use for noting when the spoken words occur. Based on the above, in an example I worked out that someone said subtitle #2 (you should be able to recognise the words based on context, even if you don’t speak the language) at 00:02:54.000 and that the one of the last spoken texts occurred for subtitle #613 at 01:29:17.000. The important thing is to get the timing of the first subtitle as accurate as possible, for the last subtitle it doesn’t matter if you’re a second or two out. To do the actual synchronization choose the “Transform Positions” tool from the subtitle menu (see screenshot below), specify the times you observed and the positions of the other subtitles will be adjusted proportionally.

To test the subtitle file is now correct, play the middle of the movie and see if the subtitles are fully in sync there as well. Repeat, if necessary with other time points, and if necessary adjusting only part of the file (“Transform selected subtitles”) rather than the entire file (“Transform current project”).

Now, for step 5 we are going to hardcode the subtitles into a new XVID file which we produce. We can choose to either start with the source coming from the original DVD-rip folder produced by step one, or we can choose to work with the XVID file produced in step three as the source. Obviously, working with the XVID file will be lossy, as one can expect some picture degradation using the XVID file as a source file. However, if this file were to be used, you would know for sure that your subtitles will sync up with the file. Think back to the DVD extract step with dvdbackup – did it report any errors at all during the extract from the DVD? For instance, if there was a scratch on the DVD disk dvdbackup would still be able to work around it by block filling in the file produced, but I have observed such a “repaired error” might cause problems with the timing and sync of subtitles in the step to follow if the original DVD extract is used as the source. Generally, because of this I prefer to work with the XVID file as the source, even though it may be lossy.

Before starting, remove any file in the current folder called “divx2pass.log”, this file will have been produced if you did a two-pass encoding as above and will interfere with the two-pass encode we are about to do.

To hardcode in the subtitles, using the divx file produced in step three as the source, here are the commands that will perform the two-pass encode for you:

mencoder -idx movie.avi -ovc xvid -sub subtitles_adjusted.srt -subpos 96 -oac pcm -xvidencopts pass=1:bitrate=1200:aspect=16/9 -o /dev/null

mencoder -idx movie.avi -ovc xvid -sub subtitles_adjusted.srt -subpos 96 -oac pcm -xvidencopts pass=2:bitrate=1200:aspect=16/9 -o hardcoded.avi

Now, if you wanted to work with the original DVD-rip folder as the source, rather than perform the lossy re-encoding of the DIVX file, then replace “movie.avi” in the above with “-dvd-device name_of_movie/VIDEO_TS/ dvd://1” adjusted where appropriate.

Also, note in the above that “subtitles_adjusted.srt” refers to your subtitle file that you produced with gaupol, and also note that I prefer specifying that my subtitles appear at position 96 (you can leave this out if it doesn’t bother you).

And also note that there is another way to hardcode in subtitles but this way can result in enormous files.

Finally, in step 6 I break up the resultant DIVX file (which would be called “harcoded.avi” if you followed step 5 exactly). Why do I do this? For two possible reasons:

If you are playing off a VFAT formatted memory stick files cannot be larger than 2GB in size.
My player can play DIVX files, but once the file exceeds 1GB in size the playback simply stops.

Now, there are two possible workarounds. You could have simply lowered the bitrate so that the final XVID file was less than 1GB in size. Or you could chop up a big file as I do below. Of course, if your player can play big DIVX files and you’re not working with a VFAT memory stick then there is no need to perform this step.

In this example I am working with a 3.1G file. Firstly, to determine how long the video is, I issue:

ffmpeg -i hardcoded.avi

In my case, it tells me that my video has

Duration: 02:05:16.88.

So, I’d like to split that into four parts, so that each part will come to less than 1GB in size (roughly) – so after some quick mental arithmetic I think I’d like each piece to be 32 minutes long.

Based on the above example, these are the four mencoder commands to splice the video into four 32 minute chunks (the last chunk will be less than 32 minutes):

mencoder -ovc copy -oac copy -endpos 0:32:00 -o hardcoded1.avi hardcoded.avi
mencoder -ovc copy -oac copy -ss 0:32:00 -endpos 0:32:00 -o hardcoded2.avi hardcoded.avi
mencoder -ovc copy -oac copy -ss 1:04:00 -endpos 0:32:00 -o hardcoded3.avi hardcoded.avi
mencoder -ovc copy -oac copy -ss 1:36:00  -o hardcoded4.avi hardcoded.avi

Granted, this is an extreme example (normally you don’t let the XVID file get so big!) but it does show how to splice up a video using the -ss and -endpos flags. Unlike the transcoding to XVID steps, this step executes very quickly as all it does is copy the video and audio streams.

And voilà, copy the split files to your memory stick, and take it to the lounge to enjoy with your popcorn, knowing that you’ve managed to render a previously unwatchable video into something you’ll enjoy, and somehow all this extra work makes the video more enjoyable too.

¹ Hardcoding means the actual video file is altered so that the subtitle is in the video stream. It’s a little nasty, and I agree softcoding (keeping the subtitles separate from the video stream) is better. The only problem is that my player refuses to accept files where the subtitles have been muxed in. If you’re curious about softcoding, I can recommend AVI-Mux GUI which does run under wine.

Tunnel through ISA proxy

Wed, 03 Nov 2010 16:30:33 +0200

Geeks often need to access their *nix computers from work. Doesn’t everyone want to do that? True geeks control their computers strictly using the command-line, of course, and the tool that is used to control a remote command-line session is ssh.

What one usually does is use a tool like corkscrew to send ssh traffic through an HTTP proxy.

At one place of employment, a known trick of using corkscrew to tunnel out using the work proxy failed, with this message:

Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied. )

I tried all combinations of DOMAIN\USERNAME:PASSWORD in my corkscrew auth file but nothing worked.

If you see this message have no fear! What you need is a utility that can negotiate NTLM authorization with the proxy.

There are several open source tools that can do NTLM, of these I chose cntlm. Often ntlmaps is recommended as a utility to negotiate the authorization. The cntlm man page indicates that cntlm is far more efficient than ntlmaps, both in terms of memory and CPU usage.

One oddity about cntlm relative to other software that you may have worked with is that configuration is a two-step procedure: firstly you configure the software with a default config file, like the following (the settings that need to be configured for now in the first-step are in the first paragraph: username, domain, password, proxy, proxy-port):

#
# Cntlm Authentication Proxy Configuration
#
# NOTE: all values are parsed literally, do NOT escape spaces,
# do not quote. Use 0600 perms if you use plaintext password.
#


Username	__username__
Domain		__domain__
Password	__password__		# Use hashes instead (-H)
#Workstation	netbios_hostname	# Should be auto-guessed


Proxy		__PROXY__:__PROXY_PORT__


#
# This is the port number where Cntlm will listen
#
Listen		3128


#
# If you wish to use the SOCKS5 proxy feature as well, uncomment
# the following option, SOCKS5. It can be used several times
# to have SOCKS5 on more than one port or on different network
# interfaces (specify explicit source address for that).
#
# WARNING: The service accepts all requests, unless you use
# SOCKS5User and make authentication mandatory. SOCKS5User
# can be used repeatedly for a whole bunch of individual accounts.
#
#SOCKS5Proxy	8010
#SOCKS5User	dave:password


#
# Use -M first to detect the best NTLM settings for your proxy.
# Default is to use the only secure hash, NTLMv2, but it is not
# as available as the older stuff.
#
# This example is the most universal setup known to man, but it
# uses the weakest hash ever. I won't have it's usage on my
# conscience. :) Really, try -M first.
#
#

Auth		LM
#Flags		0x06820000
#
# Enable to allow access from other computers
#
#Gateway	yes


#
# Useful in Gateway mode to allow/restrict certain IPs
#
#Allow		127.0.0.1
#Deny		0/0


#
# GFI WebMonitor-handling plugin parameters, disabled by default
#
#ISAScannerSize	1024
#ISAScannerAgent	Wget/
#ISAScannerAgent	APT-HTTP/
#ISAScannerAgent	Yum/


#
# Headers which should be replaced if present in the request
#
#Header		User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)


#
# Tunnels mapping local port to a machine behind the proxy
# 
Tunnel		11443:__OUTSIDE_HOST.COM__:443

Then, run
cntlm -v -M http://google.com (or any other external site)

Cntlm will use this to assess the type of auth your proxy can handle. In my case I got back something like the following output:

Config profile  1/11... OK (HTTP code: 301)
----------------------------[ Profile  0 ]------
Auth            NTLMv2
PassNTLMv2      AE1234567890123234567890123456C4
------------------------------------------------

For the second configuration step, these two lines need to be pasted back into your configuration file replacing the line that said “Auth LM” (and you must do this for your own situation, you can’t reuse my lines).

Then, startup the cntlm daemon:


cntlm

Let’s test if it works. Note that in the above config file I have a tunnel defined (the last line of the config file). Now, in order to ssh to port 443 of host __outside_host.com__ which is outside the proxy, one can do so using:


ssh -p 11443 localhost

The above assumes that outside_host.com has an sshd listening on port 443.

Cntlm also works fine under windows. If you don’t have “administrator” authorisation under windows, you can still run the cntlm executable, but need to specify which cntlm.ini file to use, in other words, something like:


cntlm.exe -c cntlm.ini

Of course, if you’re running on windows you won’t have an ssh command line client, but putty can be used nicely for this purpose. Bear in mind that putty needs to be configured (for the default cntlm configuration above) to use an HTTP proxy on host “localhost” listening on port 3128.

Split and encrypt files for google docs

Fri, 29 Oct 2010 15:25:12 +0200

Since January 2010, Google docs has allowed you to store any type of file, even arbitrary binary files. However, there are a couple of gotchas: one cannot upload files greater than 1GB in size, and you may want to encrypt your files so that not just anyone can read them (for instance server backup files).

The two bash scripts below provide a solution for the above. I call them the ‘mince’ scripts ‘cos they slice and dice your files and hopefully you’ll get hamburgers back at the end of the day. These scripts depend on you having a fairly new version of bash on your unix-like system, the ‘split’ utility and gnupg (GPG) which is used for the encryption/decryption. If you’re unsure of GPG, a good getting started guide can be found here.

It must be said that google docs is (in my opinion) currently not the best way to store your files in the cloud. In fact, I wrote another blog post describing the “google storage” options in greater depth.

The encrypt&split script is mince.sh and it takes two parameters, the first one a directory or archive, the second the email address for an already imported public key:

#!/bin/bash
# gpg encrypt archive and split into chunks
# $1 specifies base directory or compressed archive to encrypt.
# $2 is the recipients public key, eg. 'myfriend@his.isp.net'
set -e


CHUNK_SIZE=1000000000 #1000000000==1GB (not 1GiB!)
SCRATCH_DIR=~/scratch_space
TAR_REGEX='\.tar'


usage() {
    echo "ERROR: "
    echo " $*"
    echo "USAGE: "
    echo " mince.sh [DIRECTORY|ARCHIVE] PUBLIC_KEY_NAME"
    echo "EXAMPLE: "
    echo " ./mince.sh directory myfriend@her.isp.net"
    echo "FURTHER COMMENTS: "
    echo " if an ARCHIVE is supplied instead of a directory, it must have a name like file.tar or file.tar.gz or file.tar.bz2 "
}


#check parameters entered are valid
[ $# -ne 2 ] && usage "Two parameters are required." && exit 1
if [ ! -d "$1" ] && [[ ! $1 =~ $TAR_REGEX ]]; then
    usage "$1 is not a directory or tar/tar.gz/tar.bz2 archive."
    exit 1
fi


#if 1st parameter is a directory, then tar it up in the scratch space
if [ -d "$1" ]; then
    absolute="`cd $1; pwd` "
    mkdir -p $SCRATCH_DIR
    cd $SCRATCH_DIR
    nameonly=${absolute##*/}
    nameonly=${nameonly/ /} #remove trailing spaces
    tar -cf $nameonly.tar $absolute
    arch="${SCRATCH_DIR}/${nameonly}.tar"
    created=true
    echo "Created temporary archive $arch"
else
    arch="`readlink -f $1`"
    created=false
    echo "Working with existing archive $arch"
fi


#call for GPG encryption and compression of the archive
target="${arch##*/}.gpg"
name=${target%\.gpg}
mkdir -p $SCRATCH_DIR
cd $SCRATCH_DIR
echo "Commencing GPG encryption, please be patient"
gpg --bzip2-compress-level 6 --compress-algo bzip2 --output $target --encrypt --recipient $2 $arch


#split .gpg file into chunks of size CHUNK_SIZE
outdir="${SCRATCH_DIR}/output"
mkdir -p "$outdir"
mkdir -p "$outdir/$name"
cd $outdir/$name && rm -f $name*
echo "Splitting files"
split -b $CHUNK_SIZE "${SCRATCH_DIR}/$target"
for x in *
do
  mv $x "../${name}__$x"
done


#clean up - remove .gpg and temporary archive and temporary directory
cd $SCRATCH_DIR
rmdir "$outdir/$name"
rm $target
if [ $created == true ]; then
    echo "Removing temporary archive $arch"
    rm $arch
fi

echo "All file splits produced placed in $outdir"

Download link for mince.sh

The bash script to reconstitute the file is called unmince.sh and takes one parameter – the name of the first file downloaded from google docs:

#!/bin/bash
# reassemble an archive from chunks of a file that have been gpg-encrypted
# $1 specifies the first file produced from the mincing process, eg, file__xaa
set -e


SCRATCH_DIR=~/scratch_space
REGEX='__xaa'


usage() {
    echo "ERROR: "
    echo " $*"
    echo "USAGE: "
    echo " unmince.sh file$REGEX"
    echo "WHERE: "
    echo " file$REGEX is the first file produced by the mince script"
}


#check parameters
[ $# -ne 1 ] && usage "Only one parameter is required" && exit 1
if [ -d "$1" ] || [[ ! $1 =~ '_xaa' ]]; then
    usage "$1 cannot be a directory and must end in _xaa."
    exit 1
fi


#combine all chunks of the file
sourcepath="`readlink -f $1`"
pathonly="`dirname $sourcepath`"


just=${sourcepath##*/}
basenam=${just%$REGEX}
indir="${SCRATCH_DIR}/reconstituted"
mkdir -p $indir
cd $indir
[ -e $indir/combined.gpg ] && rm $indir/combined.gpg
for x in $pathonly/$basenam*
do
    cat $x >> combined.gpg
done


#decrypt the .gpg file
echo "Commencing GPG decryption, please be patient"
gpg --output $basenam --decrypt combined.gpg


#tidy up - remove the gpg file
rm combined.gpg


echo "The reconstituted archive $indir/$basenam was created"

Download link for unmince.sh

Since I might still tinker and improved these scripts, to get the newest version of these files take a look at my github repo at http://github.com/eyesonly/kenwood (Named after the Kenwood Chef, a famous mincer!)

"Google paid storage" and "google storage" are different things (for now)

Fri, 15 Oct 2010 15:29:11 +0200

So, at a friend’s prompting (and that it struck me as a bargain) I paid $20 over to “Google paid storage” to give me 80GB storage for one year. This may be cheaper than Amazon S3 storage where the cost calculation is more complicated (see below ¹) and is significantly cheaper than dropbox which charges $9.99 per month for 50GB. However, if you read further you’ll see that this price is too good to be true.

What I paid for was essentially this, an extension of the normal storage space that Google provides for free for gmail, gdocs, picasa and so forth: “Google offers a way to purchase more storage space when you run out of free storage space in Gmail, Google Docs, Picasa Web Albums, Blogger (for photos), and Google Buzz (for photos).”

So, in my google docs account I can now see that I’ve got 80GB available, but interestingly, if I want to access real “Google Storage” (ie. using an Amazon-S3-like API) it is something else, called Google storage for developers. Unfortunately, Google have named their products in a way that has confused me and is possibly confusing for others with these similar names.

Sadly, “Google Storage for Developers”, hereafter called by me “Real Google Storage”, is also only currently available to a limited number of U.S. developers only.

It kind of sucks that I can’t (currently) assign some of my 80GB over to Real Google Storage, although I assume Google will make some sort of plan for this in future.

In the mean time, I must make use of google docs, but it has some limitations. The way I see it, there are two big hurdles with google docs: firstly, the need to chunk files into 1GiB or less, and secondly for non-premier accounts (even if you paid for google storage) access is only through a web interface.

With respect to the first problem, I’ve written a script that encrypts/decrypts and then chunks files for storage in google docs, this is described in another blog post of mine here. With these scripts one is able to securely store your own private files on google docs and not worry about others reading the contents; also files greater than the 1GiB limit are split into multiple parts.

Secondly, with respect to access being through the web front-end, I investigated alternative ways of accessing my gdocs, for example as one would have to do from a server without a graphical web browser (for my primary use case I have a strictly command line only server overseas that I’d like to back up to my google storage).

There are two FUSE filesystem mounters for Linux (that I know of) which allow one to attach your google docs as a regular filesystem, there is a python one – google-docs-fs and a java one – gdocsfs. They work as advertised, except both do not appear to have support for content of an arbitrary file type, as the google-docs_fs manual states: “Google-docs-fs does not support the new feature of Google Docs which allows any file type to be stored. I am looking to add this in future releases.” The ability to upload files of any type is one of my primary needs with online files storage, I’d like to be able to store my server backups and files for my own personal transfer on google docs, and not just spreadsheet and document type files.

Well, since I’m a programmer I thought I’d write a simple utility to upload arbitrary files, and I discovered gdocs4ruby, but alas this seemed to only support an older version ² of the API that google provides for google docs.

Luckily, and unsurprisingly, python has official support for the gdocs V3 API. So, I quickly wrote a python script using the linked-to guide, and also quickly found out that I could not simply upload an arbitray file.

I obviously didn’t read version 3 of the google docs API closely enough as it quite clearly states that upload of files with an arbitrary content-type is only for Google Apps Premier domains.

I’m not the only one that has a problem with this: there is a forum thread here with many other frustrated folk who have paid for google storage and would like to use an API call to upload files of arbitrary types. On the one hand, perhaps I shouldn’t expect something for the price I paid, and I can understand the google do not want to turn google docs into another Rapidshare filled with copyrighted materials. On the other hand, Google’s vagueness about what one is buying at the time of buying it without spelling out the limitations, and not providing a contactable link or address for obtaining a refund for what they’ve sold you is also unfair.

So, I probably won’t end up using my google storage for much and can’t really recommend it to others who want to use it for the same things that I do, rather look at Dropbox or even Ubuntu One. Being forced to upload files through a web browser that supports javascript means I probably won’t be able to use it for uploading my server backup files to google docs as my overseas server is strictly command-line only. And uploading files from home using the web frontend for google docs is still painful and flaky – the web frontend consistently gives me “server error” messages when I attempt to upload my files just like others have experienced.

¹ Regarding S3 pricing, it is composed of both an in/out traffic charge and a storage charge; based on the examples given on the page, assume in a given year that I store 50GB in my account for the whole year, and that I transfer in 200GB and out 200GB during that year, then the charge for this will come to:
$0.150 per GB (US standard, not California)! * 50GB = $7.50 per month
16.667GB transfer-in per month * $0.10 per GB = $1.66 per month
16.667GB transfer-out per month * $0.15 per GB = $2.50 per month
So, approximately $11.66 per month which works out to be lot more than the $20 that google currently charge for so-called “storage” of 80GB for one year.

² I may be wrong about gdocs4ruby supporting only an older API for google docs as I didn’t tarry too long in Rubyland once I saw that I could not upload files of arbitrary type.

Emacs-like keybindings for the new ABAP frontend editor

Thu, 12 Aug 2010 10:15:44 +0200

In the new SAP ABAP editor, it’s sort-of possible to remap the keybindings you use. If you’re a regular Emacs user, you often find yourself pressing the wrong key combination accidentally – for instance C-a to go to the beginning of a line, only to find that the new ABAP editor by default respects the regular windows CUA keybindings – so C-a does a “Select All” as it does in many windows programs.

I’ve redefined many of the new editor keybindings that I use to be more Emacs-like. If you’re interested, here is a PDF document listing all of my defined keybindings for the “emacs” schema. This is a stylesheet transformation of the file “keymap.xml” that is stored in your SAP work directory (so, in my case this is C:\Documents and Settings\Jonathan User\SapWorkDir\ab4_data\keymap.xml). Here is my keymap.xml file, I would say simply replace your own keymap.xml file with mine BUT I’ve had sufficient grief doing it even when upgrading my SAP GUI from one release to another, so doubt this will work for you. If you do want to try that, please backup your entire SapWorkDir first (as a point of reference, I’ve had this blog post in draft form since December 2008 due to not being sure how to share my keybindings with others).

What is probably more sensible if you want to try a similar Emacs keybinding scheme is to take a look at the PDF containing my keybindings and to manually create your own Emacs keybinding scheme. To do so, start by clicking on the little yellow icon on the far right bottom of your editor screen (indicated by the arrow):

Why do I say that its only “sort-of” possible to remap the keybindings? Well, there are limitations : (1) Our nice custom keybindings can’t be used when in display only mode, like when we’re on a non-development system (2) We can’t do complicated key sequences, eg. C-x C-w can’t be bound to “File.Saveas”. (3) Meta key bindings are difficult as pressing the ALT key on its own makes the menu bar at the top of the screen active, so the ALT key must always be combined with other modifiers like control (4) The windows key cannot be bound. In fact, we seem to be stuck with Shift, Control and Alt as modifiers. (5) Some things like kill-line (C-k) have no equivalent command in the front-end editor. (5) Block mode in the two editors is different in concept.

There are some windows CUA keybindings that you may be particularly fond of that get lost if you rebind them – for instance C-v does a “Clipboard Paste”. I’ve also bound that to C-y in the keybindings file above to be more like Emacs, but you may not like that, so I guess whatever keybindings you end up using will become a matter of personal taste, which is as for all things Emacs. Still, some sensible substitutions for cut-copy-paste are also defined by default in the new editor – for instance Shift-Insert performs a paste as it does on many terminal emulators.

By the way, as far as I know the new editor mode only is available for the windows SAP GUI. ¹ The option to select the new editor does not appear in all the versions of the Java GUI that I’ve tried. It would be a shame if the new editor was never ported to the Java GUI, and my Linux street cred has suffered enormously as a consequence.

I have considered developing a proper “abap-mode” for the stand-alone Emacs editor as well, but it does seem it would have very little benefit for the effort involved – the benefit would apply if you opened a saved file with ABAP code in it in Emacs, or if you pasted into an Emacs window outside of your SAP GUI. If it were feasible to embed Emacs into the SAP GUI as the default editor then an abap-mode would be a real consideration. However, I don’t see how it’ll be feasible due to the tight integration that the new editor enjoys. For instance, one would have to allow for drill-down into other sections of code, allow for integration with the SE80 code browser and so forth. So I personally see a truly embedded Emacs as a wonderful thing (that will never happen!).

I also have a wish-list for commands that I’d love the new ABAP editor to support. Top of this list would be an Emacs kill-ring. Luckily, there is something similar – pressing “Edit.PasteExtended” (By default bound to Control-Shift-V) pops up a list of the clipboard history, so all is not lost however, and if you didn’t know about this feature it’s great.

¹ The new front-end editor is available for SAP GUI for Windows 6.40 Patch level 10 and onwards, also your server must be able to support it – the first official server version that supports it is SAP NetWeaver2004s, although since then the editor has been backported, you can also run with it on older versions going back to 6.20 (but your system must be at one of the newer patch levels).

Kinesis Advantage Ergonomic Keyboard

Mon, 14 Jun 2010 11:59:28 +0200

Never trust your hands to a generic, sticky old Dell keyboard that you just find on your desk at work. In my case, it took about a month of using that keyboard to develop a case of tendinitis. To be fair this is the second time I’d experienced such a problem; the mouse was partially responsible, and Emacs chording was also contributing it’s share.

Since this was the second time I’d experienced pains in my hands from typing I’d already done my research and knew that I’d want to look at buying a “Kinesis”: http://www.kinesis-ergo.com/ keyboard. The problem was primarily that Kinesis make two very different styles of keyboards – the freestyle which is most similar to a conventional keyboard – with a similar layout except that it is split into two independent halves and the more radical Advantage. The Advantage has two recessed key ‘wells’, and radically has thumb keypads forcing you to make use of your more powerful digits. It was this latter feature that made me order the advantage even though this is an expensive keyboard. Personally I’ve never seen any true ergonomic keyboards (other than Microsoft Natural boards and the like) in real life – they just aren’t sold in South Africa and it costs a lot to have them shipped over here. So I asked around on various forums and newsgroups before committing to the capital outlay. One of the things that I’d never considered when making the choice was the type of keyswitches in the keyboard. In the past I owned an IBM buckling spring keyboard, so I do appreciate the feel of a good keyboard, although quite frankly I found the gentle softness and quietness of cheap membrane PC keyboards to be a relief after the IBM! It turns out that the kinds of keyswitch most highly prized on the geekhack forums are the Cherry (Cherry “Brown” is liked the most by some) and Topre switches. So, reason number two to buy a Kinesis Advantage – it has Cherry Browns. If you’re interested in ordering the same from South Africa bear in mind that it’ll cost you about twice the direct exchange rate once you’ve factored in UPS shipping (obligatory) and customs duties. So anyone who lives in Cape Town and wants to try mine out before buying, I fully understand!

It has been an interesting journey and things I’ve long considered to be true have had to be reviewed. One of these things is about learning to type correctly. My wife, Shira, took the time to learn to type correctly – she bought a typing instructor (Deluxe!) CD-ROM and did all the exercises and games, and the first thing she does is position her fingers correctly on the home row. I’ve been hacking computers for years and have been able to fully touch type without looking down at my fingers since around about the time I took up programming full time. And of course I’ve picked up the same bad typing habits you have, and have never considered it a worthwhile use of time to learn typing per se.

My first experience of the Kinesis is that it felt like being taken straight back to being a complete keyboarding beginner – I was only able to type at a few words per minute at first and the feeling of disorientation was strong. The keyboard ships with a little booklet with typing exercises in it to help you pick up the habit of blind typing again, and spending a couple of evenings following the exercises in the book is sufficient to develop some basic keyboarding confidence again. The main Kinesis manual is clear that during the adaption phase one shouldn’t switch between the Kinesis and conventional keyboards, although after you have adapted you can switch back and forth easily (conventional keyboards do feel cramped after one has become accustomed to the Kinesis, and the staggered arrangement of keys on conventional keyboards mean that typing blind one starts to hit the ‘B’ key accidentally quite a lot again, but this is not so much of a problem). The booklet of typing exercises that is shipped with the keyboard is worthwhile and useful – I gained an appreciation of the logic behind key placement and design and also learned useful things about typing correctly, such as using the opposite hand to press shift when capitalizing a letter.

Some things I’ve noticed:

Using your pinkies. I never learned to use my ‘pinkie’ fingers in typing (possibly because laptop keyboards are compressed and you don’t need to), but I learned that in proper touch typing you will use your weakest digit quite a lot, especially for pressing ‘shift’ and the letter ‘P’ amongst others. Also, it is good typing practice to use the opposite hand for the modifier key, so you don’t scrunch up a single hand and injure it in this way (think of the compression that your left hand experiences if you press ‘Control’ and ‘C’ with this hand on a conventional keyboard).

Split keyboards. If you’re forced to type correctly and use the correct hands and fingers then even a non-ergonomic keyboard is fine. So if you really and truly never cross over your hands when typing you no longer need this feature, but I’m going to need to use a split keyboard with a gap between the left and right group of keys for some time to come. Even after using the Kinesis for a couple of weeks I find that I still try and type ‘T’, ‘G’ and ‘B’ with my right hand occasionally, swapping over is a difficult habit to drop. Here’s a pic of the split keyboards in Webwit’s collection:

(Picture shown with permission. Note that he has arranged the keyboards with his favourites in the front row – and the Kinesis advantage is the black and blue keyboard in the center of the second row from the front, so it is not one of his top picks although little can compete with the datahand in the front center. This is an impressive collection – about the only board that I’ve noticed missing is the smartboard).

Posture and balance – I’ve noticed that I walk up stairs differently after using the Kinesis for a few hours. This is partly because it has forced me to sit upright, although I think using all five fingers for typing is also a factor – the experience of typing on the Kinesis is more like that of a pianist using the whole hand; a greater muscular sensitivity and suppleness is developed by typing on this keyboard – it isn’t necessary to have to bash the keys as the keys register before bottoming out) and this does have a muscular repercussion on a greater level.

Non-staggered key arrangement. Why none of the Kinesis reviews don’t mention this is a mystery for me – this is one of the best selling points of this keyboard. Keys are in vertical rows – for example ‘E’ is directly above ‘D’ – this really does make it easier to type since all you need to remember to do is to move your finger directly upward one key.

Size. Ever so slightly smaller keys on the Kinesis would be nice. Sometimes it does feel like it was designed for someone with slightly bigger hands than mine, and slightly smaller keys and keywells would improve my experience of this board considerably.

Cannot raise the front of the keyboard. Other ergonomic keyboards offer the ability to tilt the front or the back of the keyboard (and I have found that raising the front of such keyboards does help me) but unfortunately the Kinesis does not offer this facility and it is difficult to maintain keyboard stability by propping it up from underneath due to the curved design of the underside.

For alphanumeric entry the Kinesis has become quite comfortable for me and I can hit a decent WPM, but when it comes to hitting ctrl- and alt- and the arrow keys I’ve been struggling. I also mispress ‘enter’ and ‘space’ quite frequently (they are on the same thumb with enter requiring a slighly greater stretch). A large part of the problem is the thumb keypads are elevated above where my hands are while typing. If you look at other ergonomic keyboard designs such as the Maltron (the similarity to the Kinesis is not accidental!) you will notice that the thumb keypads are recessed. Since the Kinesis is fully programmable in firmware you can remap keys however you desire. I have found that placing ‘Alt’ on the bottom of the thumb pads (where ‘end’ and page down’ are normally) has helped me to avoid doing weird contortions with my thumbs, and I’ve also taken care to have an ‘Alt’ key available for both my left and right hands so that I don’t have to scrunch up a single hand.

Certainly, I need to give myself more than a couple of months usage on this keyboard before I can say more about whether I like it or hate it – it has become much easier to use after a couple of weeks, but I do wonder if I’ll ever be able to use this keyboard completely on autopilot where there is not one part of my brain thinking where the alt key lies or how to find the arrow keys. On the other hand, the excellent keyswitches and the non-staggered split design are features that I will always look for on other keyboards in future.

Keyboard buying could easily become an expensive and distracting hobby. Sites like Elite keyboards offer an interesting range of mechanical keyboards, and hardened collectors order custom made keycaps. I’ll almost certainly be ordering a keyboard with Cherry blue keyswitches in the next while, but shipping to South Africa is a major cost and I’m not even sure I’ll like the feel of the keyboard without having access to other enthusiasts in Cape Town who have them on hand to try out. So, if you’re in Cape Town (or Joburg) and have a keyboard with Cherry Blues then please add a comment below.

Powerball Odds

Wed, 10 Feb 2010 12:20:00 +0200

This week’s powerball currently has a rollover jackpot of 90 million Rands. A highly attractive target. But before going off to “tata your chance”, let’s look at the statistical probability that you’ll win the South African powerball game.

To win the jackpot prize in powerball, you’ll need to select 5 numbers between 1 and 45 and also the matching powerball number which is between 1 and 20. The powerball number can be the same as one of the group of 5 numbers selected previously.

So, by my calculation, the number of ways 5 numbers can be randomly selected from a set of 45 is:
45! / ( 5! * (45-5)! ) = 1221759

And since, for each of the 1221759 combinations there are 20 different ways to pick the powerball number, the overall chance of getting the jackpot is one in ( 1221759 * 20 ) = 24435180, so in English it is roughly one in 24.4 million.

Out of curiosity, is it easier to win the regular South African lotto? Here the rules are simpler, you’ll need to select 6 numbers between 1 and 49. So, using a similar calculation as before:
49! / ( 6! * (49-6)! ) = 13983816, or in English nearly one in 14 million.

Tax on the stupid or not? Even knowing these odds though, I’ll probably be buying a powerball ticket for the next draw and pretend I’ve got a decent chance ☺

Crapple "release" the iPoo

Wed, 03 Feb 2010 14:59:58 +0200

CUPOOTINO, California. Steven Jobless, CEO of Crapple Corp, announced his company’s long awaited tablet form-factor machine last Wednesday. Whilst sequestered in the backwoods of Tennessee last year, Mr Jobless spent a lot of time waiting for medical test results and considering the meaning of life. Whereupon he had what he described as a “number 1 experience”. It hit him – the perfect Captive Market. The new product, marketed as the iPoo is of the ideal form factor for use whilst positioned on the lavatory. Unlike the early tablet products produced by their competitors, the iPoo is not a fully fledged compooter, but follows the Jeff Raskin school of thought on what constitutes a suitable “Intuitive Interface” for the average user.

Marketing of the new product is intended to involve heavy use of anti-social media – each geek on the planet is forced into deciding whether they would buy the device or not. Since most of them think it’ll be shit, it’ll count as a ringing endorsement for the non-tech savvy public. It seems that the geek public is waiting for the next shitteration of the device.

Taking full advantage of the DRM the product provides, it comes in two models. The slightly more expensive option, “Comes with Porn” (seemingly aimed at competitor Noklue) unlocks DRM for porn on the device, and also uses a new technology known only as “3D-multitouch” (details not yet provided at time of going to press).

We spent some time after the media announcement assessing the mood amongst the company faithful, and by and large that mood was one of “Relief”. Reaction amongst the Linux community was less positive, with comments ranging from “We invented Brown” (Ubuntu’s Mark Shuttleworth), “We invented shit-casting” (Communitization manager, Jono Bacon) to “It’s Crap” (Fabian Scherschel, from the Linuxoutlaws).

Dynamically hide elements with javascript - SAP CRM 2007

Fri, 18 Dec 2009 16:58:06 +0200

In a recent SAP CRM 2007 implementation, we had the requirement to hide a view (“partial”) in response to a change in a dropdown value.

This is easily done using javascript code on the client side, and what’s more such a solution to the problem is easy to implement and is not really restricted to SAP CRM 2007 or even SAP web development in general.

This is what we wished to achieve – hide block ‘A’ on the following screen if the value in the category dropdown ‘B’ is anything other than “corporate brands”:

This can be done using the ABAP CRM framework, but there are several potential concerns:

Components not previously enhanced may require enhancement
Components are designed to be independent, and making one component rely on another (which we want to do here) involves creating some sort of shared attribute
In order to suppress a view it will be necessary to re-render all views on the current frame and will involve an unnecessary round trip.

It was therefore decided to solve this problem using javascript code.

As SAP CRM 2007 is supported by Firefox (version 2, although version 3 will work with rendering glitches), it is possible to use the Firefox ‘Firebug’ extension to inspect a web page and determine the relationship between elements on screen and appropriate blocks of HTML.

From Firebug analysis (or if you have patience to do this manually) it was determined that this block of HTML code represents the Dropdown B:

<td class="th-ip-td1" style="overflow: hidden; width: 100%;">
<fieldset class="th-if-wrapper-onfocus" style="border: 0pt none ; margin-right: 3px;">
<input id="C14_W59_V60_ClassificationDdlb1" class="th-if th-if-icon" readonly="readonly" 
value="Corporate Brands" onkeydown="thtmlb_ddlbInputKeyDown(this,event);"
style="width: 100%;" size="16" onblur="thtmlb_inputBlur(this);"
onfocus="thtmlb_inputFocus(this);thtmlbSaveKeyboardFocus('C14_W59_V60_ClassificationDdlb1');"
onclick="thtmlb_inputClick(this,7);"/>
</fieldset>

The element we’re interested in detecting a change of value for is the input tag with id “C14_W59_V60_ClassificationDdlb1”

This block of HTML code represents the viewset area A that we will be hiding (ignoring a lot of the inner HTML):

<td class="th-gr-td" valign="top" rowspan="1" colspan="1">
<div id="C11_W43_V44_V48" excevt=""
intevt="c:C11_W43_V44_V48:C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do;" 
automode="true" tgt="" dhe="true" 
style="display: inline;">
<!-- Begin C11_W43_V44_V48 -->
<div id="C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do" class="th-ajax-area">
<!-- Begin C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do -->
<table ....

The element we’re interested in hiding is the div with id=“C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do”

In it’s simplest case, the following javascript, if inserted into the view somewhere after the thtmlb:gridCell tag that declares dropdown element B, does what is required – it will hide block A if the value of dropdown B changes to anything other than “Corporate Brands”:

<script for="ClassificationDdlb1" event=onchange type="text/javascript">
var dd = document.getElementById("C14_W59_V60_ClassificationDdlb1");
var hidee = document.getElementById("C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do");
if(dd.value != "Corporate Brands"){
  hidee.style.display = 'none';
}
else {
  hidee.style.display = 'inline';
}
</script>

Of course, we’re going to require something a little more complicated! The problem is that the id for the elements under consideration is not fixed – the C14_W59_V60 will be different for different roles and is affected by configuration changes. One way to solve this problem is to write a javascript function that will go through all elements and check for a regular expression match based on the contents of id. However the javascript function getElementById only accepts an exact match on id.

A custom function fuzzyElementSearch was created to find the element of interest (this function is contained within the file jonathan.js which will be listed afterwards), this is what the call to this function looks like:

<script src="/sap/bc/bsp/sap/crmcmp_ic_frame/scripts/common/jonathan.js" type="text/javascript">
</script>
<script for="ClassificationDdlb1" event=onchange type="text/javascript">

var dd       = fuzzyElementSearch("th-ip-td1","td",null,"input","ClassificationDdlb1", 0)[0];
var dd2      = dd[0];
var hide     = fuzzyElementSearch("th-gr-td","td", null, "div","productdetailviewset", 1)[0];
if ((hide) && (dd2) && (hide.length > 0)) {
  var hide2    = hide[0];

  if ((dd2.value) != "Corporate Brands") {
    hide2.style.display = 'none';
  }
  else {
    hide2.style.display = 'inline';
  }
}

</script>

The file jonathan.js contains the following javascript:-

function fuzzyElementSearch(classname, tagname, root, tag2, subbie, level) {

    // Call function from Rhino book that returns
    // an array of DOM elements that are members of the specified class,
    // have the specified tagname, and are descendants of the specified root.
    var elements = getElements(classname, tagname, root);

    // Find children (of tag2) for each of these elements
    var subfound = [];

    for(var i = 0; i < elements.length; i++) {
        var subchild  = elements[i];
        var grandkids = subchild.getElementsByTagName(tag2);
        if (grandkids) subfound.push(grandkids);
    }

    // search "level" deep for each of the above matches
    // for an element where the id contains the substring
    var pattern = new RegExp(subbie, "g");
    var matches = [];
    for(var i = 0; i < subfound.length; i++) {
        var matchee  = subfound[i];
        if (matchee.length > level) {
          var matchin = matchee[level];
          var matchid = matchin.id;
          var reslt = pattern.test(matchid);
          if(reslt == true) {
             matches.push(matchee);
          }
        }
    }

    // Note that we always return an array, even if it is empty
    return matches;


 // *** The following attribution applies to the rest of the code in this file
 // This code is from the book JavaScript: The Definitive Guide, 5th Edition,
 // by David Flanagan. Copyright 2006 O'Reilly Media, Inc. (ISBN #0596101996)
 // *** 

 /* Rest of code == EXAMPLE 15-4 From David Flanagan, "Javascript"
 * getElements(classname, tagname, root):
 * Return an array of DOM elements that are members of the specified class,
 * have the specified tagname, and are descendants of the specified root.
 *
 * If no classname is specified, elements are returned regardless of class.
 * If no tagname is specified, elements are returned regardless of tagname.
 * If no root is specified, the document object is used.  If the specified
 * root is a string, it is an element id, and the root
 * element is looked up using getElementsById()
 */
    function getElements(classname, tagname, root) {
        // If no root was specified, use the entire document
        // If a string was specified, look it up
        if (!root) root = document;
        else if (typeof root == "string") root = document.getElementById(root);

        // if no tagname was specified, use all tags
        if (!tagname) tagname = "*";

        // Find all descendants of the specified root with the specified tagname
        var all = root.getElementsByTagName(tagname);

        // If no classname was specified, we return all tags
        if (!classname) return all;

        // Otherwise, we filter the element by classname
        var elements = [];  // Start with an emtpy array
        for(var i = 0; i < all.length; i++) {
            var element = all[i];
            if (isMember(element, classname)) // isMember() is defined below
                elements.push(element);       // Add class members to our array
        }

        // Note that we always return an array, even if it is empty
        return elements;

        // Determine whether the specified element is a member of the specified
        // class.  This function is optimized for the common case in which the
        // className property contains only a single classname.  But it also
        // handles the case in which it is a list of whitespace-separated classes.
        function isMember(element, classname) {
            var classes = element.className;  // Get the list of classes
            if (!classes) return false;             // No classes defined
            if (classes == classname) return true;  // Exact match

            // We didn't match exactly, so if there is no whitespace, then
            // this element is not a member of the class
            var whitespace = /\s+/;
            if (!whitespace.test(classes)) return false;

            // If we get here, the element is a member of more than one class and
            // we've got to check them individually.
            var c = classes.split(whitespace);  // Split with whitespace delimiter
            for(var i = 0; i < c.length; i++) { // Loop through classes
                if (c[i] == classname) return true;  // and check for matches
            }

            return false;  // None of the classes matched
        }
    }
}

The function fuzzyElementSearch accepts five parameters, and is used to find the element based on it’s relation to an an outer element. In other words, we find the element we’re interested in based on the context of the surrounding HTML code.

So, if we consider the case where we are looking for the element that represents viewset A that we want to hide:

<td class="th-gr-td" valign="top" rowspan="1" colspan="1">
<div id="C11_W43_V44_V48" excevt=""
intevt="c:C11_W43_V44_V48:C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do;" 
automode="true" tgt="" dhe="true" style="display: inline;">
<!-- Begin C11_W43_V44_V48 -->
<div id="C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do" class="th-ajax-area">
<!-- Begin C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do -->
<table ....

The outer element in this case is <td class="th-gr-td" valign="top" rowspan="1" colspan="1">
The inner element (which is what we want in the end) will be <div id="C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do" class="th-ajax-area">

The following five parameters supplied to the fuzzyElementSearch are what the function uses to robustly identify the element we want:-

classname = the CSS class of the outer element, or th-gr-td in this case.
tagname = the tag type of the outer element, td in this case
root = the root element from which we start searching. We’ll search the entire document so can leave this as null
tag2 = the tag type of the inner element, or div in this case
subbie = substring (actually a regular expression string) that will match the id of the inner element
level = number of levels down from outer tag to inner tag. In the above case, we are interested in the 2nd div tag nested inside the td tag. Since javascript counting starts at 0 the level we are interested in is level 1.

Putting it all together, the following call to the fuzzyElementSearch gives us the inner element we want every time (it will be the first item in the array returned):

var hide     = fuzzyElementSearch("th-gr-td","td", null, "div","productdetailviewset", 1)[0];

How does fuzzyElementSearch work? It makes use of a function getElements that is from David Flanagan’s book “Javascript” (ISBN 0-596-10199-6, also known as the Rhino book) – this function allows us to search for elements based on class and/or tag name. With the results of this function similar code is used to find all the children of these elements that have the correct tag and regular expression match. I heartily endorse David Flanagan’s book, and if you are interested in learning javascript then consider visiting the book’s web site.

Using mencoder to fix a broken video file index

Tue, 24 Nov 2009 09:32:06 +0200

An oldie, but a goodie, and there is always someone out there who doesn’t know about this. Often, one encounters video files that do not have an index or have a broken index- this is almost certainly the case if your media player or PVR refuses to allow you to fast forward within the file. Easily fixed under Linux using the following mencoder incantation:

mencoder -idx input.avi -ovc copy -oac copy -o output.avi

As a bonus, this apparently also repairs broken interleaving.

Some more great mplayer/mencoder hints, including tricks such as appending multiple AVI files are here.

Linode and Panix VPS benchmark comparison

Mon, 23 Nov 2009 11:49:01 +0200

In May 2009 I moved my VPS (Virtual Private Server) from being hosted by Linode in Dallas to Panix in New York City. Price wise, the offerings are identical, $21 monthly consisted of a Xen virtual server with 360MB of RAM, 16GB storage and 200GB of transfer, plus one additional public IP address. However, not all virtual private servers are created equal, and as I discovered benchmark measurements showed performance differences between the two offerings. Please bear in mind that the benchmark results that follow are not up to a proper standard of testing (the one host was running a 32 bit operating system whilst the other was 64 bit), testing was done in May (things may have changed subsequently), and you may luck out and be assigned to an underutilized host and get better than expected performance. These benchmark tests are just an interesting read, and may not be as conclusive as I would like, but they allowed me to make up my mind to move over to Panix.

One of the greatest limitations on any virtual host is the reduced amount of RAM available. This affects things like the number of web application instances you can keep running on your web server at the same time. For instance, the Phusion Passenger Users Guide recommends that the parameter “PassengerMaxPoolSize” which is the maximum number of Ruby on Rails or Rack application instances that may be simultaneously active be set to a value of 2 for a VPS with only 256MB of RAM. Having more RAM makes a big difference – the same guide recommends that if your system has 2GB of RAM the parameter be set to a value of 30.

However, another consideration in the case of virtual private servers is the issue of contention for disk access. If your VPS is sharing the same host with approximately 40 others and you’re all trying to read the same physical disk simultaneously it makes sense this will also be a bottleneck. In the case of a Panix VPS it is explicitly stated that storage is achieved using SAN disk arrays over fibre channel. From what I can tell about Linode, disks are not SAN but are directly attached RAID 10.

In the benchmark results that follow, ‘old’ is the Linode running 32-bit Debian stable (Lenny) vs 64-bit on ‘mail’ – the Panix host. Bonnie++ is a tool for disk and filesystem performance measurement, and showed superior benchmarks for Panix (at the cost of higher CPU utilization):

Panix:

Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
mail             1G 41869  75 47323  15 22774   4 51539  72 53464   3 410.8   0
                     ------Sequential Create------ --------Random Create--------
                     -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  16 +++++ +++ +++++ +++ +++++ +++ 31720  72 +++++ +++ +++++ +++
mail,1G,41869,75,47323,15,22774,4,51539,72,53464,3,410.8,0,16,+++++,+++,+++++,+++,+++++,+++,31720,72,+++++,+++,+++++,+++

Linode:

Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
old              1G 25095  51 21076   4  8116   0 22256  41 36907   0  74.1   0
                     ------Sequential Create------ --------Random Create--------
                     -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  16  3171  98 +++++ +++ +++++ +++  3285  98 +++++ +++ 12042  98
old,1G,25095,51,21076,4,8116,0,22256,41,36907,0,74.1,0,16,3171,98,+++++,+++,+++++,+++,3285,98,+++++,+++,12042,98

hdparm -t /dev/xvda doesn’t show much difference between the two services, but is too unscientific to come to any conclusion. An odd hdparm conclusion is that ‘secondary’ disks, eg. /xvdb are slower, but it could just be file composition.

Nbench test results showed that the offerings were largely equal in terms of CPU speed:

Panix:

BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)

TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          872.48  :      22.38  :       7.35
STRING SORT         :          131.52  :      58.77  :       9.10
BITFIELD            :      4.0716e+08  :      69.84  :      14.59
FP EMULATION        :           137.8  :      66.12  :      15.26
FOURIER             :           26793  :      30.47  :      17.11
ASSIGNMENT          :          30.251  :     115.11  :      29.86
IDEA                :          6029.6  :      92.22  :      27.38
HUFFMAN             :          1976.1  :      54.80  :      17.50
NEURAL NET          :          39.809  :      63.95  :      26.90
LU DECOMPOSITION    :          1396.4  :      72.34  :      52.24
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX       : 62.027
FLOATING-POINT INDEX: 52.042
Baseline (MSDOS*)   : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
CPU                 : 4 CPU GenuineIntel Intel(R) Xeon(R) CPU           E5410  @ 2.33GHz 2333MHz
L2 Cache            : 6144 KB
OS                  : Linux 2.6.28-xen3-U-64
C compiler          : gcc version 4.3.2 (Debian 4.3.2-1.1)
libc                : libc-2.7.so
MEMORY INDEX        : 15.823
INTEGER INDEX       : 15.224
FLOATING-POINT INDEX: 28.865
Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38
* Trademarks are property of their respective holder.

Linode:

BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)

TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          818.88  :      21.00  :       6.90
STRING SORT         :           117.4  :      52.46  :       8.12
BITFIELD            :      3.6408e+08  :      62.45  :      13.04
FP EMULATION        :          123.95  :      59.48  :      13.72
FOURIER             :           23156  :      26.33  :      14.79
ASSIGNMENT          :          29.876  :     113.68  :      29.49
IDEA                :          5413.8  :      82.80  :      24.58
HUFFMAN             :          1786.4  :      49.54  :      15.82
NEURAL NET          :          35.209  :      56.56  :      23.79
LU DECOMPOSITION    :          1297.9  :      67.24  :      48.55
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX       : 56.803
FLOATING-POINT INDEX: 46.438
Baseline (MSDOS*)   : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
CPU                 : 4 CPU GenuineIntel Intel(R) Xeon(R) CPU           L5335  @ 2.00GHz 2000MHz
L2 Cache            : 4096 KB
OS                  : Linux 2.6.18.8-linode16
C compiler          : gcc version 4.3.2 (Debian 4.3.2-1.1)
libc                : libc-2.7.so
MEMORY INDEX        : 14.617
INTEGER INDEX       : 13.852
FLOATING-POINT INDEX: 25.756
Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38
* Trademarks are property of their respective holder.

Performance differences for the Nbench benchmarks may be largely attributed to the difference in processor speed between 2.33 GhZ (Panix) and 2.00 GhZ (Linode). I do know that Linode do continuously and regularly upgrade their hardware, so bear in mind that the above was what I saw on a single instance in May and that things may have moved on since.

There are some minor ping latency differences between Linode and Panix, but in the case of Linode you can choose which data center you want your VPS to be hosted in. Often ping differences are more an accident of geography and network topology; I have been led to believe though that it makes sense that if you’re in South Africa, or Europe to try and use a data center on the upper East Coast to reduce the number of network hops. A friend in the UK had an average ping time to the Linode data center in Newark of 90 ms which seemed to be 40-50 ms faster than the Linode data center in Dallas that I used. Conversely, in my limited testing from Cape Town, I recorded an average ping time to Linode Dallas of 386-400 ms but to Panix in New York it was 432-433 ms (proving nothing really other than that South African ADSL is pretty awful!).

If performance is not your sole reason for choosing a VPS provider then Linode has got a lot going for it with the Linode control panel which is by far the best way to control your VPS. With Linode, I was able to shrink disk images, expand disk images and clone disk images, all of which is useful for making backups. The Panix interface is a lot simpler, and only options to create or destroy images are provided. But, keeping to the topic of backups, Linode do not backup your VPS for you (RAID10 aside) whereas Panix offers a free rsync backup. The Linode backup service is currently in Beta – it looks like it’ll cost you about $5 per month for the Linode plan I was on, but it is worth it.

If you like BSD it is important to note that Panix provide disk images for FreeBSD and NetBSD guests, whereas Linode only offers Linux guests. BSDs are possible with Linode, but seem to be more experimental at this moment in time.

The Time Traveler's Wife

Sat, 12 Sep 2009 22:19:35 +0200

I’ve been quite taken by the Time Traveler’s Wife (The novel by Audrey Niffenegger). Which is why I’m writing this post – the film adaptation of the book has left me with a hollowness inside that the book did not have, even though I so enjoyed the film. So herewith my first blog film review. DISCLAIMER: PLOT AND FILM SPOILERS TO FOLLOW, please do stop reading if you ever intend to read the book or see the film. My recommendation is to read the book first.

According to Wikipedia, Niffenegger wrote the book as a “metaphor” for her own failed relationships. Out of which came what aspires to be a heady love story mixed with science-fiction. Not since Marge Piercy’s Body of Glass has such a blend of the realistic and fantastical appealed to me in the same way. When reading the book, I kept on thinking to myself “I wish I could have thought of this story myself” and “I wish that I could write like this!”.

Niffenegger’s Henry suffers from an involuntary genetic condition which causes him to spontaneously time travel to places and people in his life. Although Henry has no control of destination (or much else in his existence), the destination is not completely random and is often as a result of Henry’s unconscious yearnings.

It’s an ambitious theme to follow – and such a romantic one at that – when Clare meets Henry for the first time as an adult at the age of 22 she’s known him on and off since she was 6 and has an unquestioning certainty of their love. Henry at the age of 28 however has never met her before. It should be completely unfair to Clare, who after all never did have an opportunity up until then to meet Henry as an equal but only as a mysterious adult. It is unfair to Henry too, but it will be an older Henry who later on will go back in time and who will eventually meet up with (and finally seduce) the younger Clare. The book downplays the seductive element of it – Henry is given a free pass here as he lives in a deterministic universe where he is unable to change the flow of time or to alter outcomes. Or at least that would have to have been the defense in a court of law “your honour, I couldn’t help myself”. I guess one should lift a little scepticism here and just believe that it is all for the good, financial and otherwise, which is what I did and I thoroughly enjoyed reading the book. In many ways it strives to be the ultimate love story – in this universe it is as simple as the fact that Henry and Clare will love each other for who they are intrinsically, regardless of their lack of shared personal history or shared personal time line, and it is as if their love has been pre-written in the book of time.

The film starts by covering the scene in the book where Henry’s mother dies in a car crash. It’s a good starting point for the story-translated-to-script. But other choices made to translate the book into the film, although essential, feel like an amputation of the book. Characters are eliminated en mass (including all the black characters in the book, which I guess is not too much of a bad thing as they’re all servants except for the token Lesbian – hey we’re all equal opportunity discriminatory). One of the things that I was looking forward to from reading the book is seeing what the real world Newberry Library looks like (Henry works there as a librarian, and it is lovingly described throughout the book). But although I got to see the inside of a library (and maybe the outside too, I may have missed it!) I’m not sure if it was the actual library itself, and I never got to see “the cage” in the library – a meshed-in stairwell where Henry gets trapped due to his time travelling. The cage is such a strong visual metaphor, and I couldn’t help but feel that the writer of the screenplay thought it might be too strong an image for his audience. And therein lies the rub. The film has been cleansed and sanitized. It almost feels like the book was rushed too quickly into a movie, or perhaps, and this is my own bias showing, a director of the European school ¹may have been sensitive to which cuts were kind to the story, and which were unnecessarily kind to an audience deemed too sensitive to see the amputation of Henry’s feet.

I loved Rachel McAdams in “The Notebook” but in this movie she fell short of my own expectations of Clare. She was great, in a beautiful almost Angelina Jolie like way, and her character carried loads of emotional depth and a sort of unforgettable “shimmer” on screen, similar to her portrayal of Allie in The Notebook. She just was different to what I was expecting of a Clare, perhaps due to my own misreading of the character. On the other hand, there are some scenes in the movie that for me personally had a far higher emotional impact when seen on celluloid than when read in the book. And I’m thinking here of the scene where Henry meets his daughter for the first time and learns of his own death. What a tear jerker, I don’t know if it’s my own personal childhood losses that are being triggered here, or if I was being taken in by the universal chick flick. It could also be that the character of the 10 year old Alba so strongly reminds me, and I’m sure many others, of our need to protect our children from the uncertainties of life.

Perhaps the unkindest cut of all was the changing of the ending. In The Titanic James Cameron fished together a mystical final scene where the protagonists are reunited after death. Hollywood is quite capable of appending its own version of a satisfying ending to a story of disaster and tragedy. Niffenegger’s book too has its similar anchoring final chapter where an old Clare living alone is visited one last time by her “late” husband Henry as a much younger man. Inexplicably this too was removed from the movie, and replaced with a scene where Henry and Clare get to say good bye to each other at an earlier stage. Satisfying, sure, after all there is no long waiting suspenseful period, but it is somehow without the intensity and poignancy of the original.

What are your thoughts on the transition from book to film? Anything that you need to process about this film or book? Please comment

¹ The director Robert Schwentke is German, but is also Hollywood trained

Why begone :-(

Wed, 26 Aug 2009 15:10:20 +0200

I’ve been quite shocked and saddened by the online disappearance of why the lucky stiff last week. Heck, the blogging engine for this site is written to make use of his micro web framework – camping.

_why wrote (writes!! we await the second coming) some of the more interesting ruby code out there. And he also wasn’t afraid to write things that were quite out there – on the borderland between programming and art. One of his creations was called blimlimb (the code is still around, but I fear some of the blimlimb scripts and repository history is lost). Imagine this, you’re sitting in a peaceful IRC channel and all of a sudden a whole troupe of bot actors appear in channel. They start performing a play for you, complete with actions and appropriate dramatic pauses. And the weird part is that the bot puppetmaster can also add a bit of improvisational scripting to spook those in channel further.

Unsurprisingly (knowing the delicate nature of IRC in-tractions) blimlimb was poorly received! I don’t think it was a case of “the implementation lacketh much” but rather that the blimlimb works themselves were directed towards the wrong audience.

I wrote a blimlimb script too, but have always been too scared to present it. It was destined to have been run in #clug (the Cape Town Linux Users group channel on the Atrum network) and featured pseudonymous appearances by some of the denizens of that darkened pit. The intention was to advertise the appearance of a new version of irbie, my IRC bot, that was now able to do two new things – it could now parse python code and had an intelligence engine added so that irbie could interact as if it were an ELISA robot. My apologies to Mary Shelley and all those mentioned (and not mentioned), herewith is presented the lost blimlimb script of #clug:

[a] aizonli
(a) Cough. Announcement time. Will take about 5 mins.
[v] viatta
v: All names are changed to protect the *innocent*
[w] woozy
[m] MrKkkkkkkkkken
[y] MythRandy
[s] humblesteed
w: http://www.zimbabwesituation.com/
m: innocent? did someone mention my name?
a: Hmmm... everyone seems to be too much in character, lets go with the Frakenstein theme now
v=VictorFrankenstein
a=TheMonster
m: Oh all right, I'll be Igor...
m=Igor
m: Yess master
a: Well there once was a poor misunderstood bot, called...
m: Monster?
a: No, Irbie
[f] LordoftheFlies
(f) /kill irbie
a: too much in _character_ again :-(
(f) goes back to the bovines
-f
a: Who needed a makeover.
w: or a personality?
[i] influence
[c] cocochanel
[h] hodgepodge
[j] jermy
c: fp
j: Spinasie: choose Erlang or Ruby\ to go with my chunky bacon
(a) Cue lightning. Loud scary noises.
y: As a child, I thought I could live without pain,without sorrow
(w) straps Monster to the lab-bench
m: Looks like irbie, although noone really knows what he looks like
[p] pipecleaner
(p) threatens irbie with the leadpipe
p=irby
(p) starts speaking parseltongue
a: Or for the humans among you irbie now evals python
a: (Smoke and mirrors, of course)
a: and just like the python shell anything starting with >>> gets evaluated
v: Of course, the monster still can speak Japanese?
a: Yes, that's a double chevron >> , as before
w: What about personality?
a: Well, hopefully the lightning worked. How about addressing irbie directly?
v: Message delivered, can we go back to being ourselves?
a: kbye
-h
s: k...
-i
c: ...bye. Darn too late again.
-j
-a
-v
-w
-m
-y
-s

Of course, _why’s withdrawal from the world means that irbie is currently broken. It used to use _why’s “TryRuby” website to parse ruby code, so you won’t see it in channel at the moment.

Max DB log is full - SAP NW 2004S TestDrive for Linux

Fri, 17 Jul 2009 13:36:26 +0200

A short while back I downloaded and installed the SAP NetWeaver 2004S TestDrive SAP system (running on my Atom! machine at home). This provided me with my own (90-day licensed) basic SAP R/3 system which, according to the documentation, includes both ABAP and Java stacks running on SAP’s own database Max DB.

One of the issues that this system is prone to is that the Max DB log gets full (there is no default log cycling). It is quite a well known issue and is even described on the SAP on Linux: Test Drives – Tips and Tricks FAQ . In my case, I encountered this issue after running an SGEN job to generate all the basis components on the system. I could log into the SAP system, but the GUI froze before anything useful came up. Trying to run stopsap, it hung forever at the checking database step (where it runs R3trans d).

Looking in knldiag, I knew I had a full log problem because of this message:

2009-07-17 10:38:54    21 WNG 20001 KernelDB  99 percent of log area occupied, 126717 pages used

What one should do in this case is use one of the database management tools (either dbmgui or dbmcli) to clear the log area. However, no matter which “database owner” username and password I tried, I could not gain access to administer the database, and others have experienced the same problem too . In other words, I was trying something like this:

dbmcli -u some_username,some_password -d N4S util_execute set log auto overwrite on

but was receiving this error message:

Error! Connection failed to node (local) for database N4S: ERR_USRFAIL: User authorization failed

Luckily that same SDN page described another means by which I could authenticate in order to perform database management. Philip Kisloff mentioned that it is possible to connect using

dbmcli -U c

instead of

dbmcli -u some_username,some_password

(For the record, his method works for me when logged on as the n4sadm user, which did have a .XUSER.62 file in ~/
This .XUSER62 file was created by the TestDrive install script).

The log was cleared by issuing the following as n4sadm (originally obtained from this forum thread ):

dbmcli -U c -d N4S db_admin
dbmcli -U c -d N4S util_connect
dbmcli -U c -d N4S util_execute clear log
dbmcli -U c -d N4S db_online

These steps should be performed on a stopped SAP system. If you can’t stop the SAP system since stopsap will stall with a full log – then the first step of the above will take the database offline and will allow you to run stopsap to completion. BE AWARE that you’re deleting all log information – in my case I wasn’t too worried as I could just reinstall the testdrive system, but you must be warned that this could be risky.

To prevent future overflow of the log the following can be used:

dbmcli -U c -d N4S util_execute set log auto overwrite on

This is also not without risk. As SAP put it on this page

Please note: With this setting, the database will overwrite its logs circularly, so the database will be unable to recover, if the DATA volume crashed. When developing a lot with the TestDrive, doing a backup or increasing the LOG volume would be a better solution.

Evaluate python from ruby

Wed, 01 Jul 2009 15:02:34 +0200

The newest version of my IRC bot, irbie is able to evaluate python code in channel and respond with the output.

It does this in only 33 lines of RUBY code.

#!/usr/bin/env ruby
require 'rubygems'
require 'mechanize'

class Sacrilege


  def initialize
    @code_stack = Array.new
    @agent = WWW::Mechanize.new
    @agent.user_agent_alias = 'Linux Mozilla'

    page = @agent.get('http://shell.appspot.com/')
    @form = page.forms[0]
  end

  def eval(msg)
    @code_stack.push(msg) if ( msg =~ /:$/ || !(@code_stack.empty?))
    if msg == nil && !(@code_stack.empty?)
      set(@code_stack.join("\n") + "\n")
      @code_stack.clear
      page = @agent.submit(@form)
    elsif @code_stack.empty?
      set(msg)
      page = @agent.submit(@form)
    end

    return page.body.split("\n").slice(0, 15) if page
    []
  end

  def set(val)
    @form.fields.find{|f| f.name == 'statement' }.value = val
  end


end

Of course, as with everything irbie does it’s smoke and mirrors. I just call it efficient use of webservices.

If you haven’t used mechanize before, it’s also a good advert for mechanize, which, by the way, has been ported to many other languages. Python included.

Targeted AIDS advertising

Sat, 27 Jun 2009 13:58:18 +0200

(Sign seen at the Bellville cemetery) No camping, no cars, no HIV…

This is the most directly targeted HIV/AIDS campaign that I have seen thus far, but I’m afraid it may be a little too late.

Seek within long podcasts on your car stereo

Sat, 27 Jun 2009 13:44:06 +0200

Seek and ye shall find…

My car stereo (a JVC that accepts USB thumb drives) is really slow at seeking to a specific position in long audio files [To be fair regarding long seek times, the JVC does seek faster off a CDR disk. However, it’s not much faster and then you’re left with a pile of disks that have been played only once.]. So, if you loose your place in a long podcast (e.g. the kids want to listen to something) you can spend up to 15 minutes holding down the fast forward button just to find your place again. It also jumbles up the play sequence for all the audio files within a folder. Both problems can be easily solved on the computer or when you put the audio files onto the USB drive.

My solution to the first problem is to split the audio file into multiple files, each file being 5 minutes long. mp3splt is a free utility that splits mp3 or ogg files without decoding the file into another format first, so there is no lossy decryption and encryption step. It is also pretty neat in that mp3splt tries to split on silent areas of the file (in between words), and the newer versions can preserve the ID3 tags within the file.

Here is the incantation that I’ve worked out to call mp3splt:

mp3splt -g %[@o,@N=1] -t 5.00 -a -o @n_@t YOUR_AUDIO_FILENAME

A brief explanation of the various command line switches:

switch	explanation
-g %[@o,@N=1]	set custom tags on the split files where @o = set original tags; @N=1 auto increment track number
-t 5.00	split file into five minute chunks
-a	use auto-adjust silence detection
-o @n_@t	output filename format @n = track number; @t = title

Note that the ‘-g’ clause of the above was not available from the Debian Lenny repository version of mp3splt (that version is very old).

Optional extra: On BSD/Unix/Linux systems instead of remembering that whole string, I would recommend putting the above incantation line into a file with executable permissions in the /bin folder called say /bin/mp3s. Replace the YOUR_AUDIO_FILE part with $1. So you could then easily split an audio file called SOME_FILE simply by typing: mp3s SOME_FILE and it’s not necessary to remember the complex switches as above.

The solution to the second problem (scrambled audio files within a folder) where files seem to be playing in some random order is really trivial. It seems that windows users have never encountered the problem because when they copy an entire folder to the USB drive, the file modification information is in the same sequence as the filenames. For instance, if a windows user were to copy a folder with three tracks in it

track01.ogg
track02.ogg
track03.ogg

to the USB disk the top file would be the oldest one and the bottom file would be the youngest one. If you did the same thing on a Linux workstation, say by copying a folder with

cp -r somefolder /mnt/usb/

The bottom file from the list might not necessarily be the youngest one (in other words the file modification time is not in the same sequence as the alphabetical sequence). The solution is really trivial, copy your files like this:

cp somefolder/* /mnt/usb/somefolder/

The asterisk (*) gets expanded by the shell to a file list in the same order as you see when you list your files sorted by filename.

Gitosis on Debian Lenny

Thu, 04 Jun 2009 15:56:19 +0200

No, not halitosis !

There are many wonderful public git repositories out there – like github, of course, and gitorious (pity they didn’t seem to be able to find a sexual pun in that name!). But they don’t offer you a free private repository where you can store the code that you’re not particularly proud of or don’t wish to release into the public domain, etc. However, if you have a public facing internet host, or just about any ssh accessible box you can easily setup gitosis to create a private repository that is world accessible.

When I first looked at setting up my own private git repository, I looked at running git as a daemon, and in fact there is an Ubuntu/Debian package called git-daemon-run for that purpose. But it’s overkill for a private repo only going to be used by yourself, or yourself and your development team. Also, it seems from my limited research that git-daemon is most useful for providing read-only access in much the same way that github provides public clone URLs. If you do decide you need git-daemon, this article looks like it may be helpful, especially when it comes to integrating git-daemon with gitosis.

Gitosis steps neatly around the need to have yet another permanently running daemon (so there is one less permanently available server process that may be compromised by an attacker). What it does is create its own gitosis user (also called ‘gitosis’ on Debian) with access to the git repository being through ssh private/public key transfers as that user. It is not possible to ssh to a shell prompt as that user, it is all automatically controlled. And another neat gitosis feature is that configuration is all done through git itself. One clones the gitosis configuration repository from the server, alters the configuration files on one’s own workstation, and then simply git pushes the configuration back to the server. Really neat!

I found a really good tutorial covering gitosis at scie.nti.st. However, it really isn’t Debian Lenny specific so that’s why I’m going to add a few extra notes here. From my usage on Lenny, the Debian supplied gitosis package is stable. So, my recommendation is to rather use the Debian supplied version, and skip the git-clone/compiling/dependancy parts of the scie.nti.st tutorial. The next big difference by doing things the Debian-way is that the user created for gitosis is called ‘gitosis’ and not ‘git’. So, after installing the gitosis package, the next thing you’ll need to do is become root and then su to the gitosis user which would have been automatically created for you. Then, as gitois issue a:

gitosis-init < /path_to/id_rsa.pub

where id_rsa.pub is the public SSH key from your workstation user that you’ve previously copied to the server. Make sure that the path to the public key file is accessible by the gitosis user.

And then, because you’ve initialized the gitosis configurarion with your public key, from your regular workstation, all you’ll need to do is issue a:

git clone gitosis@YOUR_SERVER_HOSTNAME:gitosis-admin.git

and git will pull all the configuration files for gitosis to your local workstation! Obviously choose which workstation you’re going to be using wisely – for instance don’t choose your workstation at work, thinking you can complete the rest of the configuration after hours – until you’ve added other administrators to the gitosis-admin repository you’ll need access to the initial workstation and user account.

The rest of the scie.nti.st tutorial is straight forward, because after that you’ll need to create new repositories and add users to those repositories.

One gotcha: for the users that I was adding to the repositories, I initially transferred the public keys of all of these users to the server, and not to the initial workstation. You don’t add new users on the server (DO NOT append to the public keys in the gitosis ~/.ssh/known_keys file on the server). In fact you’re NOT meant to configure gitosis further on the server. You do it on the first workstation that has got the cloned gitosis-admin repository by ‘git add’ing the users public keys and configuration there and then ‘git push’ing to the server. It’s really simple, just slightly different to the way most of us are used to working.

Oh and another gotcha!: I use a non-standard port for ssh to my server. So, instead of using port 22 to ssh into that host I use port 443. I couldn’t find an obvious way to tell git to use a different port, but it is no problem really as you can tell ssh to use port 443 by putting the following in ~/.ssh/config:

host  YOUR_SERVER_HOSTNAME
        Port 443

All in all though, gitosis is a really neat idea, and it works well for the creation of private git repositories.

Welcome to my new blog

Sun, 31 May 2009 14:38:53 +0200

Welcome to my new blog – it’ll have all the sort of interesting ‘tech’ GNU/Linux/OSS/Who-abbreviated-this posts that I’d be most interested in reading about, the odd technical snippet that I’ve found to be indispensable, and maybe an article or two about plants will creep in here as well. I also give you my word that I will try to only post items of quality here. Although as they say a man’s word is never done….

Expect puns, and other plays. On words. And ~~satire~~ parody.

I know it’s an eccentric, indefensible position to have, but reading other people’s blogs has made me (1) dislike wordpress and (2) gain the belief that if you’re writing a technical computer blog and can program, then you should write your own blogging engine. Or at least change wordpress enough so that it doesn’t feel like wordpress. This blog uses it’s own custom ruby blogging program written with Camping which I’ll describe ad nauseum in a later post.

I’ve got a lot of pent up “creative” computer related writing sitting in various text files on my hard drives and now in wiki’s all over the world. So, as it looks to me now, there will be a whole lot of new technical posts on this blog in the coming months.

Enjoy!