Ruby 1.8.7 and *&#!ing Curly Quotes.

We have a lot of user input in databases at work, and in several places, big paragraphs of text. Naturally people copy and paste word documents into these things all the time, which is fine in 99.9% of the cases. We’re working with a few new hoops at work, however, and in jumping through a few of them last week, these so-called “smart quotes” started causing me problems.

It turned out that in Ruby 1.8.7–but only in some cases–the interpreter was mangling these non-ascii characters when they were serialized and sent in a JSON response body. I explained the problem to a couple of other devs. One of them couldn’t reproduce it by logging into the development server and running the problematic example on the console, the other could. So there was some disparity in what we were seeing.

Gentoo Locale Settings

Eventually I started thinking about Unicode support on my local box. I use Gentoo Linux as a desktop, which I’m usually extremely happy with, but I’ve never figured out my troubles with unicode in the browser.  The classic Look of Disapproval looks like a couple of empty boxes on that machine. After poking into the locale settings a bit, I discovered my local settings were POSIX. That didn’t seem right.

Same thing when I logged into the dev box. Another developer who uses ubuntu locally saw that all his LC_* settings were en_US.UTF-8, and–surprise, surprise–he was unable to reproduce the issue on the console, whereas it presented to me. Finally onto something.

After some digging, I found that where several other Gentoo users have a 02locale file in /etc/env.d, and I didn’t. After adding that file and logging out and back in, my locale settings matched up with the other developer, and boom: the problem no longer presented on the console for me.

Apache and Phusion Passenger

It turns out there’s an annoying bug in Passenger where it doesn’t pick up environment variables from Apache in the way that it should.  For now, we’ve been able to work around the issue by adding a wrapper script to the ruby executable that Passenger uses, and set the necessary LANG variable in there. It’s not the greatest solution in the world but it’s clean, doesn’t require messing with the system-wide ruby binary, and should cause no speed issue.

If anyone else has run into this and found a cleaner solution, please drop me a comment or shoot me an email.

Stackoverflow

Love or hate them, Team Attwod-Spolsky has turned the stackexchange platform into *the* Question+Answer platform. Perhaps somewhat surprisingly, they’ve had virtually no major scaling challenges that I’ve heard of. The latest performance note from Attwood claimed that the site is “almost comically overprovisioned” with 10 webservers. Last I heard they were somewhere around the 200 million pageviews per month mark.

A disclaimer for that last link:  It’s clear that Digg made some serious engineering missteps with the latest revision of their platform, but they also have to deal with bursts of traffic that probably isn’t seen in the same magnitude on something like SOF. On the other hand, I could be way off the mark. Just my $0.02.

Reddit hits 1b pageviews

My favorite online free-time black hole, Reddit, announced yesterday that they’d surpassed the 1 billion pageviews per month mark. They’re peaking around 1250 requests per second on the app servers, or 2500+ on the cache nodes [link]. It’s nothing short of amazing that they keep things running relatively smooth given those bursts.

Beyond all the awkwardly funny or just plain compelling content on the site, I’ve been fascinated by  Reddit ever since I first happened upon it for a couple of reasons:

1. It was founded by two 22-year-old college grads and was picked for a round of funding from Y Combinator (you should be reading Hacker News)
2. They team running the site is tiny. Really tiny. Last I checked there were 6 people behind the site, and that includes development, hosting/ops, design, everything.
3. They essentially gobbled up most of Digg’s userbase. I left Digg quite a while back, thought I no longer remember exactly why other than it had lost it’s tech focus that piqued my interest in the beginning.
4. They survived the Dreaded Rewrite. Reddit was originally written in Lisp but was rewritten to Python in 2005 and continues to run on Pylons to this day.

I think what I find most fascinating about the site is that it’s completely run in the cloud. Reddit’s growth has come with it’s share of major engineering challenges, but so far they’ve been able to overcome.  As of october, they were running everything on 112 EC2 instances.

The Hacker News post is full of some other people sharing their numbers, and sites like HighScalability have more info on some bigger name sites. Read up and write ye some efficient code!

We’re just getting switched over to git from something horribly obscure (Surround SCM) at work and I’ve been picking up and cataloging random tips as I come across them. Anyways, there’s a spiffy alias for a really verbose version of `git log' that I liked:

[alias]
lg = log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit --date=relative
st = status

When I tried to run it, however, something had clearly gone wrong.

hornsebr ~/working/railsdev $ git lg
fatal: cannot exec 'git-lg': Permission denied

I didn’t even have a reasonable theory as to why I was seeing this error message. Some searching around suggested that I had a file somewhere in my $PATH that was named `git-lg', which didn’t take long to rule out.

As it turned out, /usr/games was set to 700, and owned by games.root. Correcting the problematic permissions solved the issue.

As you’d expect, the DB2 drivers for Python and Ruby are written in C and compiled to modules for their respective language targets. What you might not expect is that DB2 has a CHAR/VARCHAR type modifier that makes it difficult, if not impossible to work out of the box with nearly every mainstream framework. It’s called CHAR FOR BIT DATA. Here’s what IBM has to say about it:

A CHAR FOR BIT DATA type allows you to store byte strings of a specified length. It is useful for unstructured data where character strings are not appropriate.

So, these basically take a string that looks like '20100322014820681369000000' and turns it into something like ' \x10\x03"\x01H h\x13i\x00\x00\x00'. DB2 also includes a function called generate_unique() that, not surprisingly, generates IDs for these column types that are guaranteed unique across an entire database. Since the Python C API knows how to take a byte array and turn them into a Python string, everything should be good, right? Wrong.

What happens if you try to compress data for one of these columns, but it looks like '20000000000000000000000000' instead? You get this:

>>> from binascii import a2b_hex
>>> a2b_hex('20000000000000000000000000')
' \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

That’s a lot of null bytes, but it’ll store in the database just fine. Retrieving that from the database, however, has some unexpected results: Python only gives you one of the 26 characters you stored.  Turns out the driver does a strlen on the binary data sent from the database engine, which sees a null byte and treats it as a string terminator, truncating your data.

The good news is the fix was trivial once a convention was agreed upon. The group responsible for the Ruby driver also picked up the fix and will be releasing a comparable patch it in the near future. Big thanks to both groups for helping us get over a substantial hurdle.

Update (3/31/2010): The devs for the Python ibm_db project have released a new version that includes the fix mentioned above. I finally got around to testing the change, and it looks to have solved our problem. Hopefully the Ruby driver is corrected upstream as well shortly.

Who cares about how you do it. That’s just techniques.No matter how you devide[sic] it up into templates and scripts, if the site changes, you have to make those changes. How easy is that process?? That is the question.

Apologies for taking that somewhat out of context to prove a point.

If your software maintenance is painful or leaves you feeling apprehensive, you need to figure out why it’s that way and go after the core of the problem. It only gets worse the longer you ignore it.