Please skip or stop reading now if you’re easily offended.

Our Fawkner, who art in Hampton
Hawthorn be thy name.
Thy King's Park come; thy Williamstown,
In Eltham as it is in Hampton.
Give us this day our daily Burwood
And Footscray us our Kensington
As we Footscray those who Toorak against us.
And lead us not into Templestowe
But Donvale us from Yarraville
For thine is the King's Park,
and the Prahran, and the Glen Waverly,
For Malvern and Malvern.
Armadale.

It’s relatively easy to roll own your own URL shortening service once you know the magic behind it. I’m going to explain the basic idea so you can start rolling out your own.

UPDATE: check out django-shorturls, a Django app recently created by Simon Willison and Jacob Kaplan-Moss, it uses a similar base 62 approach.

How it works

The basic idea of how these services work is by storing URLs in a database, each with its own unique ID, which is normally an integer. The ID is then magically converted into a series of characters which can be used to make up a complete URL that people can pass around to people.

When someone opens the shortened URL with the code, the server will convert the code back into the unique ID so that it can retrieve the URL in the database. Once the URL is retrieved then the server will send a HTTP redirect response to the browser.

The magic

It’s not magic, really.

The conversion function from an integer to the short code must produce as short code as possible while still being practical: the characters in the code must be valid for URLs, no “funny” characters (“?”, “/”, “&”, “%”, “+” are probably a bad idea).

One of the easiest functions to use for this purpose is to encode the integer (decimal, base 10 number) into base 36.

Normally we use base 10 to count. In base 10 numbers we only have 10 possibilities (0 to 9) for each digit before we need add another digit in front of it and roll over to the starting symbol (“10″ is two digits, because we have no more possibile symbol to use after “9″).

In base 16 (hexadecimal) we add 6 more symbols (“ABCDEF”) so that we can count to “F” (or 15) before we need to add a digit in front and roll over to the starting symbol (“10″, or 16 in hexadecimal). If you’re having trouble following, imagine having 16 fingers and 16 toes to count with.

Same idea for base 36, but we add the whole 26 Roman alphabets, “A” to “Z”. So “Z” is 35 in base 36, and “10″ is 36.

How short is short?

With a 3-digit base 10 numbers, we can only have 1000 possible combinations (0 to 999). If I remember my high school math correctly, the formula is N^D where N is the number of possibilities for each digit (or the base number, 10 in this case), and D is the number of digits (3 in this case), so 10³ = 1000.

The goal is to have as many combinations as possible with as little number of digits as possible. Since the number of digits will grow over time (as we store more URLs) then the only way we can increase the number of possible combinations is by increasing the base number.

That’s why converting from base 10 to base 36 works.

So why stop at 36, you ask. As a matter of fact, we can make use of both uppercase and lowercase letters of the alphabets so that we can use base 62. If you’re willing to have non-alphanumeric characters in your short codes, you can even push it further with a few more characters. In fact, this is what essentially base 64 encoding does. Be careful though, the commonly found base 64 encoding normally works on strings instead of numbers, converting 6 bits of data at a time, and it can add paddings, the output is 33.3% longer than the input. Quite different from what we want.

I would probably use base 62 because it’s slightly more pragmatic and it has less chance of screwing up, but if you insist on using base 64 (for numbers, not strings), then I suggest you use “-” and “=” as the additional symbols. I’d avoid underscores (“_”) because they can be hard to see in links, which are normally shown underlined.

The following table shows the number of possibilities for some of the bases:

Say, you add 1 million new URLs every month on average. With base 62, it would take a little over a year to use up all the possible 4 digits, and after that it would take another 75 years to use up all the possible 5 digits, and after that it would take another 4,658 years to use up all the possible 6 digits, and so on.

It would probably take even longer to use up the digits if you don’t add URLs that are already in the database. Conversely, you might use up the digits faster if you want to do per-user tracking, for example.

Sample code

Here’s a simple Python code that you can use to convert an integer to a base N “number” (actually string), and vice versa.

# Code adapted from: http://en.wikipedia.org/wiki/Base_36#Python_Conversion_Code
# This is base 62:
ALPHABETS = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"

def base_n_decode(s, alphabets=ALPHABETS):
    n = len(alphabets)
    rv = pos = 0
    charlist = list(s)
    charlist.reverse()
    for char in charlist:
        rv += alphabets.find(char) * n**pos
        pos += 1
    return rv

def base_n_encode(num, alphabets=ALPHABETS):
    n = len(alphabets)
    rv = ""
    while num != 0:
        rv = alphabets[num % n] + rv
        num /= n
    return rv

Can’t we just use MD5?

No. We can’t use hash functions like MD5, because hash functions are one-way, irreversible, meaning we can’t get the original input just from knowing the hash code. Also, hash functions can have collisions, meaning from one hash code there is a remote possibility that it “maps” back to two or more URLs. Plus, the output of hash functions are normally long-ish fixed-length strings.

So, what are you waiting for?

Now that the magic is revealed, you can create your own URL shortening service. Be creative and innovative so you can compete with all the other URL shortening services out there.

As a bonus, you can also use this technique to shorten the permalinks or absolute URLs for your objects in your Django (or Rails, etc.) websites. So instead of http://example.com/post/12345 it would be http://example.com/post/3D7. Note that this has nothing to do with security whatsoever. The coded ID can still be reversed with some effort. And you shouldn’t rely on obfuscation to protect anything anyway.

Bad usability, especially obvious ones, is one of my biggest pet peeves in web development. So when a website that I — and probably many others — frequently visit has a terrible usability, I have a mixed feeling of anger and frustration that I can only hold for so long. In the spirit of positive thinking, I’d like to channel out that anger and frustration through this (hopefully) constructive criticism.

A Closer Look at the Problem

Let’s take a look at IndosatM2‘s member site as an example.

First, I’d like to ask a question. What’s the main reason people go to their own ISP’s website? I bet checking their usage is one of them. It’s probably and arguably the number one reason, if not the only reason. It is for me at least, since my ISP doesn’t provide a Dashboard widget and no one has yet to write such widget to work with IndosatM2.

The most commonly used function should be as easy to reach and as quickly displayed as possible. It wouldn’t make sense to force the user to jump through many hoops just to get to the most frequently wanted information. So let’s count how many steps are required to check the usage for an IM2 subscriber.

Step 1: Logging In

The login form is not on the main indosatm2.com website, requiring users to click a link that goes to a separate login page. (Of course, one can bookmark the page for quicker future access.)

Things to note:

• CAPTCHA. On a login form? What’s the point?
• The size and the location of the login form compared to the text on the side.
• The order of the points in the text. The “Usage History” point is last.

How to improve:

• Remove the CAPTCHA. To prevent or deter brute force login attacks, use delayed response on failed login attempts within a short period of time. Block the IP address (either temporarily or permanently) after 50 repeated failed logins within 5 minutes, or whatever is reasonable.
• Make the login form bigger and place it in the center of the page. It is a login page after all, so the login form should be the center of attention. Move the text below the login form. Show the Usage History as the first item.

Step 2: Closing Useless Popup

Things to note:

• Needs to click on OK to make it go away.
• It serves no useful purpose.

How to improve:

• Remove it altogether. We would know if the login is successful or not. Only tell us when something is not working, not when something is working. It’s like a little child yelling proudly everytime they managed to accomplish the smallest things: it’s cute at first but it gets annoying after 3 times. Yes, Mr(s). Programmer, I know you have mad Javascript skills, but who are you trying to impress? Let me guess, you got the inspiration from Windows’ popup balloons?

Step 3: The Home Page

Things to note:

• The size of the image.
• How the image serves no useful purpose.
• Needs to scroll down to see the actual contents.
• How “Last Invoice” and “All Invoice” (sic) functions are highlighted in the home page.

How to improve:

• There’s already a menu on the right side. Make it hierarchical so that the user can get straight to the function they want rather than requiring them to see an extra page that only contain the submenus.
• The “Last Invoice” and “All Invoice” (sic) buttons can go away. They are no longer necessary. Instead, put the contents of step 4 and 5 directly into this page. Read more below.
• Remove the obnoxious image. At least, don’t make it so big, or move it somewhere else, don’t make it the first thing I see when I log in.
• This step can be eliminated.

Step 4: The Usage Page

Things to note:

• It already knows what month and year it is now (they are shown as the default).
• Needs to click “OK” to get the information for that month.
• Hmm. What does “Cancel” do?
• The obnoxious image that serves no useful purpose.
• Needs to scroll down to see the actual contents.

How to improve:

• Show the month selection box straight in the home page, instead of in a separate page.
• Make last month as the default value. See “How to improve” in step 5 for the reasoning.
• This step can be eliminated.

Step 5: Finally

Things to note:

• Again, the obnoxious image and the scrolling down.
• The inconsistent use of English and Indonesian.
• There’s no easy way to see data for other months, must go back to step 4.

How to improve:

• Show the results for the current month straight away in the home page. This is what most people want to see. And since you already know what month it is now, why don’t you just show me right away?
• The system can detect what day of the month it is now. If it’s still early in the month the system can also show the total usage for last month in this page.
• The month selection box in step 4 is always shown at the top (and optionally at the bottom) of the page with the default value set to last month.

Conclusion

The steps can be reduced down from 5 to only 2 without sacrificing any functionalities. The most useful and frequently used features are highlighted and shown as reasonable and sensible defaults. Logging in should be easy (username and password, that’s it). Then the home page, containing the current month’s usage (probably the most commonly requested information), is shown right away, without any annoying popup. If the user needs to see data for other months, then the month selection box should already be accessible from the same page, showing the next commonly wanted information (last month’s usage).

Other Examples?

There is another public website that I use quite often and I find that it has plenty of rooms for improvement in terms of usability: KlikBCA. But one can write a whole book on that website alone discussing what’s wrong and how to improve it.

Have you seen other public websites that can be improved in its usability?

DON’T SIT SO CLOSE TO THE MONITOR!

Compiling psycopg2 using MacPorts as well as from the tarball would result in something like this when loaded from a web application that runs on Apache:

ImproperlyConfigured: Error loading psycopg2 module: dlopen(/Library/Python/2.5/site-packages/psycopg2/_psycopg.so, 2): no suitable image found.  Did find: /Library/Python/2.5/site-packages/psycopg2/_psycopg.so: no matching architecture in universal wrapper

So all I needed to do was force psycopg2 extension to be built for 64-bit as well as 32-bit. After extracting the latest psycopg2 source, do this instead:

LDFLAGS="-arch ppc -arch i386 -arch x86_64" CFLAGS="-arch ppc -arch i386 -arch x86_64" python setup.py build

Then sudo python setup.py install. After that, check the .so file, it should look something like this:

$ file /Library/Python/2.5/site-packages/psycopg2/_psycopg.so 
/Library/Python/2.5/site-packages/psycopg2/_psycopg.so: Mach-O universal binary with 3 architectures
/Library/Python/2.5/site-packages/psycopg2/_psycopg.so (for architecture i386): Mach-O bundle i386
/Library/Python/2.5/site-packages/psycopg2/_psycopg.so (for architecture ppc7400):      Mach-O bundle ppc
/Library/Python/2.5/site-packages/psycopg2/_psycopg.so (for architecture x86_64):       Mach-O 64-bit bundle x86_64

Don’t forget to restart Apache, and that should be it.

Update: It happened to PIL too, it was giving this error: The _imaging C module is not installed, so I had to do the same thing as above: LDFLAGS=... CFLAGS=... python setup.py build followed by sudo python setup.py install.

If you’re like me and most people, you press “Cancel”, reinsert your card, punch in your PIN again, and navigate to the screen that you want. Try repeating that while finding a menu item and a few people line up behind you and you keep reinserting the card back as soon as the machine pops it out.

I saw there were at least two buttons on the keypad that were unassigned and could have been programmed as a generic back button. Even if implementing a generic “back” button is too much effort, the “back” function could have been easily assigned to one of the side screen buttons.

And if you want to pay for a ticket, do you go to “Pembayaran” or “Pembelian” first?

klikbca is not any better. They seem to enjoy torturing their user. OK, rather than just complaining, I’ll straight away give one possible improvement to their balance checking interface. Merge the “Informasi Saldo” and “Mutasi” as one menu item called “Informasi Saldo”. Then, show the balance at the top and all the transactions for the past 31 days below it by default while still allowing the user to filter the transactions by date. Don’t ask the user to manually count (today – 31 days), just show it. If the range I put is more than 31 days, then just show what you have up to 31 days, don’t ask me to correct the starting date if you can easily fix it for me. If the info is too much and I don’t want to see that much info, then I’ll filter it later. If it’s too much info to show in one page, split it up into multiple pages. This is common sense. Why force the user to jump extra hoops every time?

The default should be sensible, and the default should be chosen because most people use it that way, NOT because it’s easier to program it that way. In order to understand the user’s pains, a designer must see from the user’s perspective and use the things they’re designing themselves. My girlfriend pointed out to me that most of these usability idiocy is because the people who designed/created it don’t use it themselves very often, and I agree with her.

All these big companies providing services to the public must hire at least a usability expert rather than wasting their money on paying half-assed so-called “designers” (that is if they even hire designers, they probably just let the programmers do the design). I mean just look at their websites. If any of you is an Indosat M2 subscriber, you would know how terrible their website is when you check your internet usage, it looks like it’s designed and written by a high-school kid who just learned PHP. Who the hell in this Web 2.0 age uses pop ups to notify the user that the login was successful?! Yes, successful, not unsuccessful.

So I did these:

• I clarified the general topic for Planet Terasi in the About Planet Terasi section,
• I also put clarification for inclusion (and rejection) in the contact page,
• I removed some of the non-functional feeds,
• I removed some of the feeds due to complaints I received from readers,
• I changed Budi Rahardjo’s feed to only include the tech section (Pak Budi: I know you asked me to remove your feeds from Planet Terasi, but I hope you don’t mind me keeping this one. If you do mind, please let me know, I will gladly remove it.)
• I went through 188 inclusion requests sent to me via email (collected for over a year now) over a slow internet connection and reviewed all 188 blogs (the working ones anyway) to be included in Planet Terasi,
• I decided to add 20 new blogs to Planet Terasi (can you name them all?),
• I bugged my network administrator to open SSH access so I can update Planet Terasi,
• I wrote this post.

Most of the blogs from the 188 requests that I did not add are due to obvious splog (spam blog), off-topic, or unreachable when I reviewed them. Some of them are apparently very popular, very well written, and very well designed. I’m hoping I could post them here later so that you can still check them out if you like. I have added some of them to my personal aggregator.

I’m actually amazed and happy at the same time that people care enough about Planet Terasi to make a petition. But it was not necessary. All you need to do is fill out the contact form and talk to me. I appreciate all your input. And I hope I can keep Planet Terasi useful and fun to all its readers. Thanks for all your support.

“Thanks before.”

“Same same.”

“Don’t follow mix.”

“Your children fruit is stupid very much.”

“My body is not delicious.”

I don’t know what the correct English translation is, so I will just assume that I can translate the words one by one, it should make sense, right? After all, each word can only have exactly one direct translation, right? It’s impossible that a word in one language can have more than one meaning in another language. And screw the context. We don’t need no context. And what did you say about culture? It has nothing to do with translating.

Since we are very good with assumptions, we always assume everything we don’t know because we can’t be bothered to look for what’s right. Assuming is easier. Mathematicians do it often, why can’t we? If I just assume the problem doesn’t exist then it’s not there anymore. Problem solved. Wait, what problem?

OK, the next person who says “thanks before” has to send me cookies.