Tyler Butler Feed

xkcd 2

Thu, 17 May 2012 18:23:00 UTC

I’ve just put up a small side project I whipped together over the last day or so called xkcd 2. It might not make much sense why I built this thing, so bear with me while I try to explain.

Why

I’ve been reading xkcd for years, but it wasn’t until fairly recently that I discovered most of the comics actually have a caption.¹ It’s tucked away in the title attribute on the img tag, and in most browsers you really only see it if you hover your mouse over the image for a few seconds.

Discovering this was a total accident. I read most xkcd comics via the RSS feed in Reeder. If you zoom-pinch on an image in Reeder, it opens up a view like this:

An xkcd.com comic in Reeder

That panel at the bottom displays the title attribute, but it only shows up if you tap once on the image. Anyway, after coming across this in Reeder, I started looking at all the captions every time a new comic would come out. It turns out that in many cases the caption is funnier than the comic itself. Case in point: comic 1049, the one featured in the picture above, is my most recent favorite. The caption perfectly mirrors my own experience with Ayn Rand, and in my opinion makes the comic that much more interesting as a whole.

Anyway, I find it frustrating that the comic captions are not a more visible part of the xkcd site itself, so I decided to put together a separate site that loads the comics from xkcd and simply displays them in a slightly different way – and includes the comic captions directly. The same comic shown above looks like this on xkcd2.com:

An xkcd comic as shown on xkcd2.com

How

Now that you know the why of xkcd2.com, let’s talk about the how.

xkcd 2 is an exceedingly simple Flask app. It’s ~100 lines of Python (not including empty lines of course). A majority of the time was spent messing with the HTML and CSS, not on the Python itself.

Each time a comic is requested, I use the excellent httplib2 library to retrieve the corresponding page directly from xkcd.com. httplib2 behaves more like a browser than a typical HTTP library; it handles caching the page content and everything for me, so if the same comic is requested many times httplib2 will return the page content from its local file cache instead of bouncing the request to xkcd.com every time.²

Once I get the response and page content from xkcd.com, I parse the page into a DOM object using html5lib. Then I use some rather inefficient list comprehensions to find the right elements in the DOM and extract the relevant information. There are probably more efficient means of doing this, but the page structure is simple enough that it’s not too bad performance-wise. I did find myself desperately wanting server-side jQuery…³

Once I have all the comic details from the page, I store it all in a pickle file so I can load it from the local disk rather than parsing the HTML on each request. Thus, if httplib2 reports that it used its cached version rather than retrieving a fresh page (via the response.fromcache property), I discard httplib2’s response entirely and simply load the comic metadata from the pickle file.

Once I have the metadata, I just pass it to the page template and render the page.

But… but…

I considered a couple of different high-level options before settling on the ‘request the page, parse the metadata’ approach. First, since the set of comics is finite (1056 as of this writing), I considered writing a simple scraper that would ‘preload’ all of the comics and just load from that cache. However, in the end this isn’t much different than what I’m doing. I still would have had to retrieve the page, parse out the relevant data, and store it. The current approach does this lazily, which makes a lot more sense since I doubt most people are looking through really old comics (though it is interesting to see how Munroe’s style has changed over the years). I still get the benefits without having this separate ‘build the cache’ step.

I also considered using the RSS feed rather than scraping HTML. This was appealing because the RSS will be less likely to change format. If xkcd.com undergoes a rewrite, my current HTML parsing code will probably break.⁴ However, the feed is a snapshot of the most recent comics, not all of them, and I really wanted to support all of them. How broken would it feel if someone sent you a link to a comic on xkcd2.com and it didn’t load because there was a new comic published that morning and the old comic was no longer in the RSS feed? Spoiler: very broken.

Anyway, hopefully some folks will find this useful and/or interesting. Code is up on github and the site is live at xkcd2.com.

Oh, you already knew that? Well good for you. ↩
There are no doubt some subtleties in the httplib2 cache behavior that I did not dive into. My hunch is that a request still gets made to xkcd.com – even just a HEAD request – because httplib2 needs to somehow determine whether its cache is still valid. I’m not an expert on its behavior. Regardless, using its cache is almost certainly better than naively proxying the request to xkcd.com – and waiting for/parsing the response – each time. ↩
This is probably actually possible using node.js, but it seemed like overkill for this project. ↩
I do use IDs and classes and avoid relying on actual page structure as much as possible, but sometimes it’s unavoidable. HTML scraping is always brittle no matter what precautions you take or how smart you try to be. ↩

Convergence

Mon, 30 Apr 2012 21:56:00 UTC

Love this quote from Tim Cook:

Anything can be forced to converge, but the problem is about trade-offs, and you end up with trade-offs that don’t please anyone. You can converge a toaster and refrigerator, but the end result won’t be pleasing to the user.

Lukas Mathis

Mon, 23 Apr 2012 21:27:00 UTC

I just linked to a piece that I got via Lukas Mathis, but I wanted to explicitly encourage you to read his stuff. In my opinion, ignorethecode.net should be required reading for anyone doing user experience design today. Very smart, very insightful, very well-reasoned. Solid work.

The Monochrome Trend

Mon, 23 Apr 2012 21:13:00 UTC

Keith Blount on color, or lack thereof, in user interfaces:

I learned something about the way my brain works that I hadn’t hitherto ever had to think about: my brain is an awful lot faster at processing colours than it is at processing shapes.

I’ve felt this way for awhile about modern UI design trend to “drain the color” from UI, to borrow Blount’s phrase. I really like color and when used tastefully and correctly I find it to be a wonderful usability aid.¹ It’s nice to hear I’m not the only one that feels this way.

In Mathis’ link to Blount, he pulls a quote from his own book, which in turn quotes Colin Ware (I’m dangerously close to Inception-level quote recursion at this point!):

In [his book] Information Visualization, Colin Ware notes that color is “preattentively processed,” meaning that we identify color before we give it conscious attention. In other words, when we look at a user interface, we can find and identify user interface elements with a specific color really quickly and easily.

On a related note, I have noticed I also have trouble with UI elements that are text-only with no color or icon hint. For example, the bookmarks bar in Safari, which eschews displaying favicons in favor of text alone, is maddening to me because I naturally look for an icon or color of some kind when looking through the links. There are probably differences in the psychology of this compared to color, but I wonder if there is experimental evidence that supports this notion that icons, color, or shape are easier to recognize quickly than words. This paper from Microsoft typography seems to indicate that we do indeed recognize words by their shape, though the prevailing scientific beliefs around the exact mechanics of that recognition have undergone some changes over the years. Interesting stuff.

The fact that my current site design arguably does not use color ‘tastefully’ or ‘correctly’ is not lost on me. I realize that the current ‘unicorn vomit’ color explosion is probably overkill and needs some adjustment. ↩

Mies

Wed, 28 Mar 2012 01:05:00 UTC

I rarely go to the Google homepage, but I wound up there today and was surprised to see a Doodle very reminiscent of Crown Hall. Sure enough, it’s an homage to Ludwig Mies van der Rohe – or simply Mies as he was always referred to on campus while I was there¹ – whose birthday was today in 1886. Very cool.

If Wikipedia is to be trusted, referring to him simply as Mies is widespread. I had always assumed this was an IIT-only thing given his importance to the campus and College of Architecture there. ↩