type object 'AnswerInline' has no attribute 'date_hierarchy'

After a bit of digging, I found that I was registering the AnswerInline with the admin, which was not necessary – it was already in the inlines attribute of QuestionAdmin.

Here is the code:

from django.contrib import admin
 
from questions.models import Question, Answer
 
class AnswerInline(admin.TabularInline):
    model = Answer
 
class QuestionAdmin(admin.ModelAdmin):
    model = Question
    inlines = (AnswerInline,)
 
admin.site.register(Question, QuestionAdmin)
admin.site.register(Answer, AnswerInline) # This line is not necessary and causes the problem

From time to time one needs to fetch a random row from a table in a db.

Solution

Take one

The “obvious” solution is to order by RAND() and get the first:

SELECT * FROM users ORDER BY RAND() LIMIT 1; /* Don't do that */

It does the job, problem solved!

Well, no, not exactly! While it works, the performance will be awful for any table of significant size.

Take two

There is another, simple, but efficient, method to get a random row:

1. Fetch the number of rows using COUNT().

   SELECT COUNT(*) FROM users;

2. Get a random positive number, that is less than (but not equal) to that count.
3. Select the row using OFFSET (on MySQL – there are ways to do the same on all major RDBMS)

   SELECT * FROM users LIMIT 1 OFFSET :rand

   where :rand is the number you computed.

While this uses two queries – both of them are very efficient.

Here is the concrete code on how to do that in Django (using a custom manager):

class UsersManager(models.Manager):
    def random(self):
        count = self.aggregate(count=Count('id'))['count']
        random_index = randint(0, count - 1)
        return self.all()[random_index]
 
class User(models.Model):
    objects = UsersManager()
    # ... [fields] ...

Usage is as simple as

User.objects.random()

Just for the record – the Django way to shoot yourself in the foot do ORDER BY RAND() is

User.objects.order_by('?')  # If you do that in production - you may as well forget about sleep!

Proof

Here is the evidence that doing ORDER BY RAND() is very slow.

For the test we will have a simple Django model and a few of scripts – one that populates the database with bulk data, one that fetches a user using ORDER BY RAND() and one that fetches it using a second query. We populate the db with one million records and get a random one.

Here are the results (done using Python 2.6.4 and MySQL 5.1/SQLite 3.6.16 on Ubuntu 9.10):

Using the naive method by doing ORDER BY RAND():

$ time python naive.py
#       MySQL        SQLite
real	0m4.519s     0m2.061s # seconds?!?! 
user	0m0.152s     0m1.952s
sys	0m0.028s     0m0.056s

(executing the query directly results in times close to those).

Using the smart method with two queries:

$ time python smart.py 
#       MySQL        SQLite
real	0m0.367s     0m0.339s # much better
user	0m0.156s     0m0.284s
sys	0m0.032s     0m0.052s

The more records you add to the table, the slower the naive method will get, while the smart one will run at about the same speed.

If you need to get a random record out of a filtered set (using WHERE) is’s basically the same. You just need to add the WHERE clause to both queries.

Details

Below is the code that is used to make the benchmark.

This is the models.py file of a simple Django app.

from random import randint
from django.db import models
from django.db.models import Count
 
class UserManager(models.Manager):
    def random(self):
        count = self.aggregate(ids=Count('id'))['ids']
        random_index = randint(0, count - 1)
        return self.all()[random_index]
 
    def random_naive(self):
        return self.all().order_by('?')[0]
 
class User(models.Model):
    username = models.CharField(max_length=128)
    password = models.CharField(max_length=128)
 
    objects = UserManager()

The bluk.py script that will fill the db with junk.

import os
import settings
 
os.environ['DJANGO_SETTINGS_MODULE'] = os.path.join(
    os.path.dirname(__file__), 'settings')
 
from users.models import User
if __name__ == '__main__':
    for i in range(0, 1000000):
        u = User(username="user{0}".format(i), password="pass{0}".format(i))
        u.save()

This is naive.py – the script that uses the slow method.

import os
import settings
 
os.environ['DJANGO_SETTINGS_MODULE'] = os.path.join(
    os.path.dirname(__file__), 'settings')
 
from users.models import User
 
if __name__ == '__main__':
    u = User.objects.random_naive()
    print(u.username)

And smart.py that uses an additional query.

import os
import settings
 
os.environ['DJANGO_SETTINGS_MODULE'] = os.path.join(
    os.path.dirname(__file__), 'settings')
 
from users.models import User
 
if __name__ == '__main__':
    u = User.objects.random()
    print(u.username)

Also, if you can, grab a copy of his book The Art of the Start. It’s the best 200 pages of entrepreneur advices you can get.

1. Go to System → Administration → Printing and delete the pdf printer.
2. Create the ~/PDF folder if it does not exists
3. Run sudo aptitude reinstall cups-pdf to reinstall it.
4. Add a pdf printer again using the wizard.
5. Profit. :)

The fix¹ is quite simple, actually – just paste the following code in ~/.fonts.conf. This settings seem to work, unlike the many I tried before that.

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
 <match target="font">
  <edit name="rgba" mode="assign">
    <const>rgb</const>
  </edit>
  <edit name="antialias" mode="assign">
   <bool>true</bool>
  </edit>
  <edit name="hintstyle" mode="assign">
   <const>hintfull</const>
  </edit>
   <edit name="hinting" mode="assign">
    <bool>true</bool>
   </edit>
  </match>
</fontconfig>

The forts are anti-aliased, as expected, and look fine in FF, Gnome and Qt-based applications.

____________________
¹ http://swiss.ubuntuforums.org/showthread.php?t=1316216

On the SD card there must be a directory structure as follows:

sdcard
   ⌊media
      ⌊audio
         ⌊notifications

Every tone dropped in the notifications folder is visible from Settings → Sound & display → Notification ringtone.

According to the android spec only h.263, h.264 and MPEG-4 SP are supported. It seems that Xvid or whatever the most videos on the net are encoded with is unsupported (I tested it). Also, the optimal size of the video is 480×352. What I wanted was the ability to watch clips and movies on the phone so I set on a mission to convert a simple .avi file to the needed .mp4 h.264 format.

The first try was ffmpeg… FAIL! The ffmpeg shipped with Ubuntu doesn’t seem to be a proper one, some codecs are missing and the overall complexity of the ffmpeg turned me away. I started looking for something else…

…and I found HandBrake. Nice little application that comes even with a GUI. Superb… Or so I though. I tried using the GUI, but it kept crashing after a few minutes of progress. Then I toyed around with the command line version. After a bit of digging around the Web and the help pages of the app I managed to come up with a script to do all of the work. Here it is:

HandBrakeCLI \
    --input "$1" --output "$2" --format mp4 --encoder x264 \
    --x264opts level=30:bframes=0:cabac=0:ref=1:vbv-maxrate=1500:vbv-bufsize=2000:analyse=all:me=umh:no-fast-pskip=1 \
    --vb 320 --rate 23.976 --aencoder faac \
    --mixdown stereo --ab 96 --arate 44100 \
    --maxHeight 352 --maxWidth 480 --two-pass

As you can see I’m using the verbose version of the different options, because I believe that no matter how large the program is – 3 lines or 3M lines – it should be readable.

Usage of the script is simple as:

./encode input.avi output.mp4

What it does is that it will convert the video to mp4, encoded to h.264 with a few options that will make the android device play it smoothly, and it will resize the video for optimal viewing, preserving the aspect ratio.

Oh, and it’s multithreaded as well, so it will take advantage of all the cores/CPUs available.

If you can think of an improvement – please let me know in the comments. Hope it will help someone.

What is Null?

Here is an explanation from wikipedia.

Null is a special pointer value (or other kind of object reference) used to signify that a pointer intentionally does not point to (or refer to) an object.

From an object oriented language point of view Null is a a value, whose type is a subclass of every other type in the system. It’s always at the bottom of the hierarchy. Because of this and the Liskov substitution principle Null can be used everywhere normally other type would be used. Have a method, that is marked to return string? It can return Null.

What is the problem with Null

The problem with it is that every method can return Null. And you, as a consumer of that method, need to check every time if the return value is Null. If you don’t – you (usually) get a Null pointer exception and your program crashes.

Now, exceptions are a very good tool – but particularly the Null pointer exception is a bad thing. Why? Because if it bubbles more than one level up it’s a sign of a leaky abstraction. Let me back up this claim:

First, what does a Null pointer exception mean? Let’s have a look at the documentation of it in Java:

Thrown when an application attempts to use null in a case where an object is required. These include:

• Calling the instance method of a null object.
• Accessing or modifying the field of a null object.
• Taking the length of null as if it were an array.
• Accessing or modifying the slots of null as if it were an array.
• Throwing null as if it were a Throwable value.

Basically it says that an object is expected, but no such is provided. Well, this is an implementation detail. In your business logic I hardly believe that you will be talking about references and such. We usually talk about real world object like people, accounts, etc. Any method that throws a Null pointer exception unveils parts of its implementation.

How many times has happened to you to forget to fill a field in some form and get a Null pointer exception.
I forgot to write my email – what does that Null thing has to do with it?!?

A method should never throw Null pointer exceptions (I’ve seen people doing it) – an Illegal argument exception is a much better one. Most of the time the underlying platform is the one throwing Null pointer exceptions.

It’s just a convention, but Null pointer exceptions should be extinguished when sighted.

So…

How can we handle Null

Take 1

The most straightforward and wide spread solution is to tell everyone to check the values for Null references. But that is a blacklist approach and, for this kind of problem, it doesn’t work. People forget stuff – and sooner or later someone will forget to do it and your 24/7 program will crash at 3AM on Sunday. Unit testing can help a lot here, but it requires work and you still cannot be 100% sure.

Can we do better?

Take 2

I would like to point out that the following method makes more sense in a statically typed language, where you have a compiler, checking your types for validity.

Most of the time we return Null as a value from a method that we want to return nothing, but some return value is expected and we throw Null at it.

Let’s have a look at the following Java code:

public String tld(String domain) {
    int dot = domain.lastIndexOf('.');
    if (dot == -1 || dot == domain.length()) {
        return null;
    } else {
        return domain.substring(dot + 1);
    }
}

The method above returns the top level domain part, or null if none found.
The “right” user code of this method is like this:

String d = tld("example.org");
if (d != null) {
    // Save to database
} else {
    // Show the user a message
}

But, as I pointed out earlier – there will be time that you will forget to make the check and the user will see a nasty exception, leaving him clueless about what she did wrong.

What we can do instead is signify our intentions that we might not return a value.
In Scala, we have the Option class – heavily inspired by Haskell‘s Maybe monad.
The Option class has two (final) subclasses²: Some and None. You return Some when you have something to return and None when you don’t.
Let’s look at the tld method implemented in Scala using the Option class:

def tld(domain: String): Option[String] = {
  val dot = domain lastIndexOf '.'
  if (dot == -1 || dot == domain.length) None
  else Some(domain.substring(dot + 1))
}

And the usage if it:

val com: String = tld("example.com").getOrElse("unknown")

That line above will return either the tld part the domain or "unknown" if none is found. Concise, isn’t it?
If we omit the getOrElse part the compiler with complain the we are not giving it a String value, but None instead.

// Does not compile!
val com: String = tld("example.com")

That way we are able to catch the error early (say 17:00 o’clock on Friday which is much better than 3:00 in the morning on Sunday).

What we’ve done? We told the type system that we might return a value, but not necessary. Specifying the return type of tld to be Option[String] instead of just String does just that.

So… Can we do that in Java?

Let’s give it a try.
We’ll have a JOption interface with JSome and JNone implementing classes:

public interface JOption<T> {
    public boolean isEmpty();
    public T getOrElse(T other);
}

The JNone class will mark the lack of value.

public class JNone<T> implements JOption<T> {
    public JNone() {}
 
    public boolean isEmpty() {
        return true;
    }
 
    public T getOrElse(T other) {
        return other;
    }
}

and the JSome class – the presence of it.

public class JSome<T> implements JOption<T> {
    private T value;
 
    public JSome(T value) {
        this.value = value;
    }
 
    public boolean isEmpty() {
        return false;
    }
 
    public T getOrElse(T other) {
        return value;
    }
}

And now our revised tld method, using the new JOption machinery:

public JOption<String> tld(String domain) {
    int dot = domain.lastIndexOf('.');
    if (dot == -1 || dot == domain.length()) {
        return new JNone<String>();
    } else {
        return new JSome<String>(domain.substring(dot + 1));
    }
}

And using it becomes:

String com = tld("example").getOrElse("unknown");

The tld method isn’t longer than our original Null-returning version, but now the compiler will help us, because if we write:

// Does not even compile
String com = tld("example");

we’ll get a compiler error.

Final thoughts

The proposed method probably isn’t the best and can be improved³, but it illustrates the concept.

Also, while this solution might help, the Null value still exists and people canwill (ab)use it.
Scala has a Null value as well – for compatibility with Java.

On the bright side – Haskell, Erlang and other functional languages don’t have anything like Null in them and use some form of the described approach.

But on the dark side we have Javascript – there you have undefined, null and NaN that adds even more codding horror. F# is in similar state.

Please, share your thoughts bellow.
__________________
¹ On 25 August 2009 InfoQ published the video.
² To be precise, None is an object (a singleton in Scala).
³ We can also add other methods to JOption like get() that will throw an exception in the case of JNone and return the value for JSome, as well as a proper equals implementation.

Megacorp138 HQ

Time goes by, John is now 28 and he is a leader of a small team, working on a very interesting project, which they have just started. From scratch! Life could not be better…

Unfortunately, soon things start to become worse. The time passes by and the project is delayed many times. Eventually it goes into production, but the users are finding new and strange bugs every day. In time all of the developers get frustrated and quit. John is now The Man of The Project. He knows (almost) everything about it and is a master of The Language.

Now John has kids, a house, a couple of cars, and even a better paid job. The project he was in charge of is now cleared of bugs, he knows most of the pieces in it, but from time to time there are some surprises that he does not expect.

John is 39. A new start-up comes along claiming that they have better product than the one John has developed.
Aah, we can crush them! They have nothing better than us — that was the mood in the company.

Few years fly by. The start-up is still doing well. The mood in MegaCorp138 has slightly changed to: We are much bigger and stronger than them. They have some extra features we don’t have, but we’ll beat them. John is the Head of the programming department now.

After few more years and some mood changes, MegaCorp138 is near to bankruptcy. The owners of the company decide to sell it to that small start-up they thought they will crush. John gets laid off.

John is 51 now, looking for a new job in the programming world. But he is having a very hard time in finding a new job, and, most importantly – he has no clue why? All these people are asking for completely different skills. John has been programming in The Language. He knows every single part of it. Of course, while he was working in MegaCorp138, now and then, a young chap would come and told him:

xUnit Test Patterns

– You should learn at least one new language per year.
– Why should I learn a new language? I’m using The Language!

Another chap tried to convince him to write tests.

Tests? I know The Language and The Product from inside and out. I don’t need tests. I’ve tested it and – here – it works. It’s on my screen. I don’t have the time to write 50% more code. I’m way too busy.

Those were commonplace pieces of advice to John.

Here is John today, still looking for a job. At one of the interviews the interviewer tells him a story.

A man was walking through the forest when he noticed a woodcutter. The man went closer and saw that the saw of the woodcutter was very blunt.
– Hey, your saw is very blunt. Why don’t you stop and sharpen it?
– STOP and sharpen it? — the woodcutter responded rudely — Can’t you see that I have a lot of work to do?

What a stupid woodcutter!, thought John.

Are you That woodcutter? Or That John?

___________
Many thanks to Nas OOD for reviewing this post.