David Ziegler's Blog

Django-css is dead, long live Django Compressor

Thu, 14 Jul 2011 09:57:38 -0700

This announcement is long past due. When I originally forked django-css from django-compressor the Django landscape and state of django-compressor were very different. The main impetus for the fork was to be able to use CSS compilers and fix some outstanding bugs.

Since then, django-compressor has been taken over and revitalized by Jannis Leidel. Not only is there a lot more active development, but it has a pre-compiler feature which allows you to easily plug in your favorite CSS and JS compilers. Since the original goals of django-css are now being met and surpassed by its parent, I think the responsible thing to do is officially kill django-css and recommend that people migrate to django-compressor.

github: https://github.com/jezdez/django_compressor

documentation: http://django_compressor.readthedocs.org

pypi: http://pypi.python.org/pypi/django_compressor

Migrating from django-css to django-compressor

For the most part, settings and usage and very similar. You should probably read the excellent documentation before you do anything, but I just want to highlight the main difference in how django-css and django-compressor handle pre-compilation of files.

Let’s assume my django-css setup looks like this:

HTML:

{% load compress %}
{% compress css %}
<link rel="stylesheet" href="/media/css/one.css" type="text/css" charset="utf-8">
<link rel="stylesheet" href="/media/css/two.sass" type="text/css" charset="utf-8">
{% endcompress %}

Settings.py:

COMPILER_FORMATS = {
    '.sass': {
        'binary_path':'/path/to/sass',
        'arguments': '*.sass *.css'
    }
}

The equivalent django-compressor code would look like this:

HTML:

{% load compress %}
{% compress css %}
<link rel="stylesheet" href="/media/css/one.css" type="text/css" charset="utf-8">
<link rel="stylesheet" href="/media/css/two.sass" type="text/x-sass" charset="utf-8">
{% endcompress %}

Settings.py:

COMPRESS_PRECOMPILERS = (
    ('text/x-sass', 'sass {infile} {outfile}'),
)

That’s basically it, but for more info check out the COMPRESS_PRECOMPILER setting in the docs.

Graceful Degradation and Web Typography

Mon, 24 Jan 2011 10:27:36 -0800

This is a cross-post from the Codio engineering blog, but I feel it’s justified since I wrote it: Graceful Degradation For Non-Standard Web Fonts.

Codio Now in Beta

Tue, 14 Dec 2010 05:27:00 -0800

Codio is now in Beta! http://blog.codio.com/codio-goes-beta

Things have been going pretty well so far. Since Chris is in San Francisco and I’m in Pittsburgh for the time being, we’ve been working on our own schedules and coordinating online.

Pros of working in Pittsburgh: I don’t know anyone here, so there’s not much to distract me. All I do is code, work out, and sleep.

Cons: It’s cold as shit, and all I do is code, work out, and sleep.

I don’t want this to become a blog about Codio, so barring an IPO or acquisition I’ll try not to mention it here anymore. If you want to follow Codio’s progress check out our blog and twitter account.

django-urlcrypt now with RSA

Sat, 06 Nov 2010 00:11:47 -0700

Someone on Reddit pointed out that our url obfuscation method for django-urlcrypt was rather simplistic. We already kind of new this, and that if someone was able to figure out the obfuscation key then they’d be able to decode the urls to get someone’s hashed password. I don’t think that’s the end of the world because the password is still sha1 hashed, but it’s certainly not ideal.

So with 0.1.4 Chris added an option to use RSA to encrypt the url tokens, and we’re not including the hashed passwords as info that we’re encrypting. From the end user’s perspective the only difference is that if the user changes their password, the old links will still be valid.

If you want to use RSA, which is recommended, just add

URLCRYPT_PRIVATE_KEY_PATH = '/path/to/private_key'

in your settings.

django-urlcrypt

Thu, 28 Oct 2010 23:56:00 -0700

http://github.com/dziegler/django-urlcrypt

Chris and I just open sourced django-urlcrypt, a Django app for encrypting information in urls. The main use case for this is when you want to give a user a url that logs the user in, and redirects them to some url.

For example if I want to email users a link that logs them in and sends them to http://www.davidziegler.net/inbox/, I would send them something like http://www.davidziegler.net/r/TkNJBkNFAghDWkdFGPUAQEfcDUJfEBIREgEUFl1BQ

Some desired properties of this url are that

1. User X can’t construct a url that logs him in as User Y

2. If User X changes his password, it should invalide all of the old login ursl.

3. The url should be relatively obfuscated so it’s not totally obvious what we’re doing.

Usage

There’s more in depth examples in the README, but basically it works like this:

from django.core.urlresolvers import reverse
from urlcrypt import lib as urlcrypt

token = urlcrypt.generate_login_token(user, reverse('message_inbox'))
encoded_url = reverse('urlcrypt_redirect', args=(token,))

Or in a template,

{% load urlcrypt_tags %}

<a href="{% encoded_url user message_inbox %}"> 
    click me to log in as {{user.username}} and go to {% url message_inbox %} 
</a>

Basically it uses hmac to create a hash using your SECRET_KEY from settings.py, the user’s hashed password, id, and timestamp. For details, check out urlcrypt/lib.py in the source. It’d be relatively straight forward adapt to other frameworks but I just haven’t had the need to do so yet.

I guess we could have just used public key encryption too…but ~~pycrypto is a pain to install~~ (EDIT: oops, I was thinking of python-mcrypt. Maybe someday we’ll implement public key encryption). Of course, crypto is hard and it’s possible that there’s some glaring security hole in there that we missed, so please let us know if you find one or send us a patch on github.

update (11/6/10): django-urlcrypt now with RSA

Codio Alpha Launch

Sat, 23 Oct 2010 20:15:00 -0700

Hiring is one of those topics you constantly hear people gripe about. I’ve been on both sides of the fence and know how hard it is to find top talent and how frustrating it can be to get lost in a stack of resumes because you didn’t include the right buzz words and acronyms.

Codio is an attempt to improve the technical hiring process. Companies post jobs and specify what programming languages they’d like candidates to be tested on. When a candidate applies for a job, they’ll be scored on an adaptive set of tests in that language. All keyboard input is recorded so whoever reviews the submissions can playback the solution and see how the applicant arrived at their answer.

We’re hoping this allows companies to screen people who are obviously terrible coders without wasting time on a phone screen or in-person interview, and gives non-technical people a better metric to judge engineers, since 90% of resumes are useless.

This is only an alpha release so there are probably bugs, but we’re iterating quickly to clean things up and add new features. My co-founder Chris and I are so excited about Codio that we’re leaving Mingle to work on it full time starting in November.

If you’re a company or recruiter and you’d like to try it out, signup at http://www.codio.com using the code ‘daveblog’. Everything is free right now and when we solidify our pricing model, early adopters will get grandfathered in at a free or discounted rate.

Feel free to send any feedback or problems you have to david@codio.com.

Python Books

Mon, 04 Oct 2010 13:12:00 -0700

My dad recently asked me for a good book to learn Python, causing my heart to swell with pride. I’m pretty sure the last programming language he learned was Fortran, since that’s what they’ve been using for the past zillion years in his department at GE. He once told me that he’s gotten to the age where every new thing he learns causes something else to get deleted, but I think Python is worth giving up his memories of my crappy piano recitals.

Coincidentally I was asked not too long ago to review the books Python Multimedia and MySQL for Python. MySQL for Python hasn’t been published yet, but I’m looking forward to reading it because we use MySQL at Mingle, and to be honest, my MySQL knowledge is not the greatest. If it wasn’t for John and Glen the database server would probably be on fire actually.

When I’m done with Python Multimedia I’ll post a full review.

Full disclosure: I received a free electronic copy of Python Multimedia and the amazon links are affiliate links (have to get my laundry money somehow).

Life Update

Sat, 02 Oct 2010 19:30:17 -0700

It’s been a while since I’ve written here because my life has been pretty busy lately. Here’s the rundown:

I’m splitting my time between Palo Alto and Pittsburgh. Helen is doing a MS in product design at CMU so I decided to move to Pittsburgh with her. Her program is only a year long, and we plan on moving back to the bay area in May. I’m still working full time at Mingle, and they’ve been generous enough to fly me back to Palo Alto roughly every two weeks to work face to face with the rest of the team.

That means a lot of traveling, which is good and bad. Flying sucks, but living in two places at once is kind of fun. I’ve only been doing this for about a month, so we’ll see how long I can keep it up.

I want to publicly announce that I’m working on a project called cod.io, and that I hope to release it in private beta in the next two weeks. This is mainly just to give me an incentive to keep it on schedule, so if you want more info about it just wait until then or send me an email.

Minor updates:

I’ll be at Startup School on Oct. 16th
Djangocon 2010 was meh
My cat Lucifer died of liver failure

Some Common Django ORM Pitfalls

Sun, 25 Apr 2010 09:19:00 -0700

For the most part, I like the Django ORM because it makes it easy to write reusable code that reads and writes from the database. I’ve found that the ORM can be a double edged sword though, as it sometimes becomes too easy to read and write from the database. In hindsight, most of the following mistakes are pretty obvious once you understand how the ORM works, but I still see these all the time so I thought it’d be good to point them out. If you want a more basic guide to Django model and querying patterns, Better Django Models is a great article for that, so I won’t reiterate the points made in there.

For the following examples, I’ll be using these models:

class Book(models.Model):
    author = models.ForeignKey(User)
    
class Profile(models.Model):
    user = models.ForeignKey(User)

1. book.author does a database query

OK, this is pretty basic, but it has a bunch of implications, such as:

book.author.id != book.author_id

Well, the values returned will be the same, but book.author.id does an additional database query. There is pretty much never a good reason to do book.author.id unless you know for sure that you’re accessing an internally cached instance, either obtained from select_related or because you’ve already accessed book.author and created a cached instance, but even then, why chance it?

For the same reason,

this is bad

book = Book()
book.author = profile.user
book.save()

and this is good

book = Book()
book.author_id = profile.user_id
book.save()

2. Querysets are not lists

How many database queries is this?

books = Book.objects.all()
print books[0]
print books[1]

The answer is 2, one for each slice. It’s much easier to see that this is 2 separate queries once you realize that the above is essentially equivalent to

print Book.objects.all()[0]
print Book.objects.all()[1]

This result is a combination of Django’s querysets being lazy, meaning they won’t be evaluated until they’re accessed, and because a queryset’s internal cache doesn’t get populated unless you iterate through the queryset. If we do:

books = Book.objects.all()
for book in books:
    print book
print books[0]
print books[1]

This will result in one database query because by iterating through the queryset, the internal cache will get populated and books[0] and books[1] will simply access the internal cache (I don’t recommend iterating through the entire queryset if you only need the first two books, I’m just trying to make a point).

3. Use iterator() when you don’t need or want the internal queryset cache

As I just mentioned, iterating through the queryset will populate the internal cache. Sometimes though, the internal cache may not be desirable. For example if we have one million users:

users = User.objects.all()
for user in users:
    print user.username

this will load one million users into memory because users internal cache will be populated. The iterator() method will tell the queryset not to populate the internal cache, which can significantly reduce memory usage and increase performance.

users = User.objects.all()
for user in users.iterator()
    print user.username

Even for smaller querysets, it’s not a bad idea to use the iterator() method if you know you’re not going to reuse the queryset.

4. Be careful with model properties/methods that do database lookups

class Profile(models.Model):
    
    user = models.ForeignKey(User)
    
    @property
    def username(self):
        return self.user.username

There’s nothing necessarily wrong about this, but it’s dangerous to expose properties or methods that hide database lookups. Especially if you’re working with designers who may not know what your schema looks like, exposing properties like this makes it easy to do:

{% for profile in profiles %}
    <li>{{ profile.username }}</li>
{% endfor %}

whereas it’s much easier to see that

{% for profile in profiles %}
    <li>{{ profile.user.username }}</li>
{% endfor %}

will do N User lookups. If for some reason you find that you do need to create a property that does a database lookup, make it private.

class Profile(models.Model):
    
    user = models.ForeignKey(User)
    
    @property
    def _username(self):
        return self.user.username

Private methods can’t be used in templates, so it becomes much harder for a designer to shoot your site in the foot.

Hopefully this was helpful for someone. Feel free to comment, subscribe, or follow me on twitter.

Announcing django-cachebot

Fri, 05 Mar 2010 17:28:57 -0800

“There are only two hard things in Computer Science: cache invalidation and naming things.” —Phil Karlton

Over the past couple weeks I’ve been working on a Django app to do automated caching and invalidation. The basic usage follows like this:

Photo.objects.cache().filter(user=user, status=2)

Anything I would say here would mostly be a repeat of the documentation I wrote, so you should just check it out for yourself: http://github.com/dziegler/django-cachebot

Be sure read the caveats. We’re using this in production at mingle.com, but it’s an early stage project so it’s possible that there are edge cases that I’ve missed. That being said, since we’re using it in production, I will try to fix any bugs as soon as I can. If you’re familiar with Django internals or feeling adventurous, feel free to take a look at the source and send me some feedback on how it could be improved.

Also, as a followup to Phil Karlton’s second point, I was thinking of naming this Sir-Cache-a-Lot, but thought that would be too hard to import, so I went with django-cachebot.

Test Database Settings in Django

Thu, 04 Feb 2010 02:21:05 -0800

For early stage local development with Django, I typically use sqlite. It’s easy to setup, delete the database if I need to, etc. Later on though, I find that it’s a good idea to switch my local database to whatever I’m using in production (postgresql, mysql, etc), either because I want to make sure that my schema migrations work, or I might have some custom non-database agnostic SQL written for an app.

The problem is that if you’re not using sqlite, Django tests run incredibly slow. With sqlite, Django will use an in-memory database for testing that is an order of magnitude faster than mysql or postgresql.

For some reason there’s no way to specify your test database engine in settings.py, but if you do the following settings hack you can use the in-memory database as your test database.

Create a file called settings_test.py that contains:

from settings import *

DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = 'dev.db' 
DATABASE_USER = ''   
DATABASE_PASSWORD = ''         
DATABASE_HOST = ''         
DATABASE_PORT = ''

And when running tests just do:

python manage.py test --setting=settings_test

You can obviously include other variables in this file if you want create some test specific behavior. For example, if you have a signal that makes an API call to some remote server everytime you create a user, you might not want to do this during testing. The simple way to deal with this would be to create a variable called RUNNING_TESTS and default it to False in settings.py and set it to TRUE in settings_test.py.

Hopefully now you have one less excuse for writing unit tests, and the people who inherit your code won’t have to track you down and strangle you.

See Which Twitterers Don’t Follow You Back (updated)

Fri, 08 Jan 2010 01:16:39 -0800

It turns out that if you were following more than 100 people or had more than 100 followers, there was a bug in my script to check who on Twitter doesn’t follow you back. Since I’m not super popular and have less than 100 for both, it took me a while to figure this out.

The getFriends and getFollowers api methods in python-twitter are paginated to 100 results per call, so I needed to modify the script to loop over the paginated results. If I was doing anything more complicated I’d probably use tweepy because it’s a more robust api wrapper, but whatever.

Also, my follower/followee count is still less than 100, so feel free to let me know if this doesn’t work, or follow me an twitter so I can test it myself :)

Programming Gloves

Thu, 03 Dec 2009 20:20:14 -0800

Unfortunately a lot of the code I’m working on is on lock down at the moment, but I thought that since it’s winter, I’d share a little trick to keep your hands warm during those cold late night coding sessions.

If you’re like me, you’re unnecessarily frugal. This means wearing 3 layers of clothes, a blanket, and maybe even a snuggie before thinking about turning on the heat. Unfortunately, computer work requires finger mobility which generally leaves my hands exposed while I’m working. It used to get so cold that my hands would be in physical pain unless I stopped every 5 minutes to sit on them to warm them up.

Hobo gloves would probably work, except they come up a little too high on your fingers to type, and I don’t have a pair of gloves lying around to cut the fingers off (obviously, my unnecessary frugality prevents me from buying them). So if you’re like me and cheaper than a hobo, you can make some fancy fingerless gloves out of old socks.

Just cut 5 holes on the end, and you’re done. I recommend thick socks obviously to keep your hands warmer, and black socks are probably a little more stylish.

This is also a good use for all of those lame socks that go all the way up your calf (I don’t know why my mom keeps sending these to me?), because it’ll keep your forearms warm.

Try it, it works.

Halloween

Mon, 02 Nov 2009 00:26:54 -0800

This is me as Kim Jong Il.

Unfortunately, my dedication to the role meant shaving a receding hairline, which means I now have a shaved head. My alternative costume was to be Kim Jong ILL, North Korea’s finest gangster rapper.

A replacement for django-admin.py startproject

Wed, 30 Sep 2009 23:16:37 -0700

When I create new Django projects, I find myself doing a lot of the same things over and over. For instance, the file structure of each project is pretty much identical, and looks something like this:

deploy
- wsgi_handler.py
docs
env (my virtualenv folder)
src
- apps
  - profiles
  - photos
  - etc.
- localsettings.py
- manage.py
- scripts
- settings.py
- static
  - css
  - images
  - js
- templates
- urls.py

The things is, I’m lazy, and it’s tedious to create all those directories, create my wsgi_handler, uncomment the admin app, update my settings.py, urls.py, etc for every project I create. I basically end up copying/pasting/deleting from old projects.

So to automate some of that, I made a small reusable django app to serve as a replacement for django-admin.py startproject. So instead of

 django-admin.py startproject project_name

you would do

 create_project project_name

To install, either clone the git repository which you can find here:

http://github.com/dziegler/django-create-project

or install with pip using:

pip install -e git+git://github.com/dziegler/django-create-project.git#egg=django-create-project

I made this mainly for my benefit, so some of the settings are tuned to my preferences. For example, it changes TIME_ZONE in in settings.py from ‘America/Chicago’ to 'America/Los_Angeles’, and automatically installs django_extensions, debug_toolbar, and django-css because I use those in all of my projects. Since it’s on github, it’s fairly easy to fork and customize to fit your preferences. You can find it here:

http://github.com/dziegler/django-create-project

Procrastination

Mon, 21 Sep 2009 16:07:01 -0700

I just wasted an hour solving this stupid puzzle: http://www.techcrunch.com/2009/09/21/google-is-searching-for-beautiful-minds-but-so-far-no-m-i-t-students-have-broken-its-code/

I say stupid, because once you discover the answer you’re like, why did I just waste my time solving this? It’s not a cool puzzle that you might have to write a neat algorithm to decipher, it’s just one of those “ah-ha!” type puzzles, which to me, don’t really tell you that much about a person if you were looking to hire them.

If you call the number they admit that they’re not even Google! I don’t particularly want a job either at Google or in Massachusetts, but now I’m annoyed that they lied to me. I’m still curious though, so I left a message and I’ll post info if they call me back. Probably not a good way to hire people by lying to them off the bat though.

Click here for the answer

Email Fail

Sat, 29 Aug 2009 12:30:10 -0700

I just noticed that my mail client will occasionally auto-select my old work email address for the “from” field when composing new emails. Unfortunately, I no longer have access to emails sent to this address.

So if I wrote you an email and you responded to it, but I never replied back, this is probably the reason. Sigh.

Feeling Nostalgic

Wed, 19 Aug 2009 20:56:00 -0700

I had the urge to create a retro version of my homepage today. I cheated a little bit because I didn’t have the patience to use frames or tables.

http://www.davidziegler.net/retro.html

Sikarra

Tue, 11 Aug 2009 17:15:00 -0700

No updates in a while because I’ve been traveling around a lot. I did manage to build this in my free time though: http://www.sikarra.com

It’s basically an online wishlist that lets you bookmark stuff you’d like to buy. Nothing fancy or anything, it’s something that I mainly just wanted for myself. I use Amazon wishlist a lot, but I didn’t like how I could only bookmark Amazon products. I have some cool features in mind that I’d like to build into it, but it depends on how much free time I have or if other people actually start to use it.

When my birthday came around my parents had no idea what to get me, so I figured I’d make it easy to share with your friends and family. The link to my public list is just http://www.sikarra.com/david

Maybe I can market this to camwhores or something? Who knows.

SF Theater Buzz

Fri, 24 Jul 2009 12:12:39 -0700

It looks like it’s safe to mention this now. I recently finished building this site for a client: http://www.sftheaterbuzz.org

My friend Kim Nguyen did the design and artwork, and I think it looks pretty awesome. You should definitely hire her, as long as it doesn’t interfere with her working with me. The whole thing is written with Django and jQuery, and I used django-css with CleverCSS to write the CSS.

It’s a little sad because there are no reviews yet, but I don’t think the client has had time to promote it. I hope he gets around to it, because I’d hate to see such a beautiful site go to waste. Also, if you’re in the Bay Area, check out SF Theater Festival this Sunday. It’s free theater, and maybe afterwards you might feel like writing about what you saw…hint hint.