Adam Hooper's Engineering Tips

refinery: a fast RAW decoding library

Wed, 15 Dec 2010 15:57:25 -0500

Don’t you hate how long it takes for your computer to import RAW photos?

The de-facto program to convert RAW files to other formats, dcraw, by Dave Coffin, gives great-quality images … slowly.

Dcraw’s author refuses to turn dcraw into a library, saying: “Library code is ugly because it cannot use global variables. Libraries are more difficult to modify, build, install, and test than standalone programs.” Those are bold statements, and I disagree with all of them.

A couple of alternatives have come up. libraw is a library built by running a Perl script on dcraw code. It presents a couple of speed-ups and allows multi-threading, but the final output’s interface is icky because dcraw’s is. libopenraw is very well documented library and has a much nicer interface, but it’s slow and all it does for now is thumbnails.

That’s where my new project, refinery, fits in. It’s brand-new, so it doesn’t do much. Here’s what sets it apart:

Refinery is twice as fast as dcraw and can produce the exact same output. On a 2.4Ghz Intel Core 2 Duo, dcraw can take 36 seconds to process 10 photographs. Libraw takes 25 using threads. Refinery takes 18 using threads.
Refinery doesn’t handle metadata. Use Exiv2, which refinery depends on, for that. (As it turns out, most popular RAW-refining software that depends on dcraw or libraw depends on Exiv2 anyway.)
Refinery doesn’t extract thumbnails. Cameras store thumbnails as metadata, so you should use Exiv2 to extract thumbnails.
Refinery grants access to image data from any step in the image-processing pipeline. For instance, users can access the camera’s raw sensor data.
Refinery is unit-tested. Unit tests are small, fast and precise, so developers don’t need to process dozens of test RAW files every time they tweak the code.

At the time I write this, refinery can only read Nikon D5000 .NEF files and PPM files. It’s not difficult to support every camera dcraw supports: it just takes some time and understanding to adopt dcraw’s code. The intricacies of each camera are buried in uncommented if-statements within dcraw. Refinery uses an object-oriented approach to put each camera’s traits into understandable classes.

The one thing going against refinery is that I don’t want to maintain it. I prefer photojournalism: I just want to import my photographs more quickly.

I’m making refinery as open and accessible as possible, so others will take it up. Check out the code and submit issues at refinery’s GitHub repository and read more at the refinery project page.

SVG/JavaScript mapping

Tue, 27 Apr 2010 05:24:24 -0400

Mapping is all the rage on the Web these days, but it often involves complicated mapping servers, complicated features like zooming and tiling, and complicated coordinate systems.

Try answering this: how can I display a map of a country which shows a certain economic indicator (for instance, proportion of people with education) by administrative region?

Yes, you can fire up Mapnik, worry about OpenStreetMap layers and install server after server to preprocess and prepare the data. But I’m looking for a one-sentence answer.

Here’s mine: I display the map as SVG and color it with JavaScript.

Example

I’ll start with the fun stuff. Here’s an interactive map, entirely public-domain. Play all you want, and “View Source” to play even more. Below, I’ll describe how this came to be.

If you see this message, the map isn’t loaded and all of a sudden my post looks a lot less awesome. Your loss.

Here is the actual data used for the map. What I mean is, the JavaScript reads the data from this table and colours the map with it. Click to change the indicator shown on the map.

Region	% Adults without education	% Non-enrollment in primary school	% Individuals ill in 4 weeks before the survey	% Adults in agriculture	% Children(5-14) working	% Households with modern walls	% Households with modern roofs	% Households connected to electricity grid
Dodoma	31	58	34	67	67	23	33	6
Arusha	20	53	23	42	73	18	53	11
Kilimanjaro	12	81	23	56	64	39	85	18
Tanga	31	50	23	67	80	6	41	7
Morogoro	26	61	32	63	55	28	45	10
Pwani	39	56	34	62	57	6	33	6
Dar es Salaam	8	71	19	3	28	88	98	59
Lindi	44	44	20	72	40	4	16	5
Mtwara	28	59	28	69	46	11	28	5
Ruvuma	15	63	22	77	73	67	42	5
Iringa	16	76	25	67	60	28	48	6
Mbeya	16	69	24	55	53	32	53	9
Singida	27	61	29	60	52	7	21	5
Tabora	31	55	27	69	39	6	24	4
Rukwa	30	61	24	76	51	32	16	4
Kigoma	28	48	21	76	60	39	21	6
Shinyanga	40	46	32	68	69	6	24	3
Kagera	25	59	34	81	68	13	53	2
Mwanza	27	52	30	71	84	10	42	5
Mara	24	62	29	70	55	23	43	10

Source: 2001 Household Budget Survey

How it works

This guide is written for somebody familiar enough with HTML and JavaScript, and inspired enough by the above map, to want to play around for a few hours.

In broad strokes, you must:

Create an SVG map of the area in question, entirely from closed path elements, with class attributes indicating the Region and District.
Output all the data to map, as in the above table.
Write JavaScript to parse the data and update style attributes in the map.

I suggest you “View Source” on this page if you want to follow along.

1. Create an SVG map

This will take some hunting, and I’m the wrong person to ask for advice. I created this map of Tanzania by running the mi2svg tool on a public-domain MapInfo (.mif) file from maplibrary.org, which I found by starting at OpenStreetMap’s Tanzania wiki page. I tidied it up in Inkscape and removed Lake Victoria. It’s a bit disappointing because most lakes and national parks aren’t shown, and some of its administrative regions are missing or misspelled, but it’s better than nothing.

Insert the SVG image into an XHTML document using an iframe tag.

I’m afraid I’m no expert on either SVG or mapping. I don’t know any good sources for maps and I don’t even know whether SVG can be embedded in HTML documents in the same way. I stopped when I got something that works for me.

2. Output the data

There’s a big fad out there called unobtrusive JavaScript. Rather than go into it in detail, I’ll merely suggest that instead of outputting a JavaScript array programmatically, you output an actual table element and parse it out in JavaScript. You won’t regret it.

I can’t help you find the data, either. Census data is always nice, and your organization’s in-house data may be great.

3. Write the JavaScript

On document load, my JavaScript (which, incidentally, is public-domain—you are welcome to use it) does this:

Parses the data from the HTML table (into an Object mapping “keys” to numeric values)
Analyzes that data to find the maximum value (it assumes 0 is the semantic minimum) and does any necessary preprocessing so we get a quick function to map values to colors
Walks through every path in the SVG document, looks up our value (using the path‘s “key”, like the key in step 1), finds the color (using our analysis from step 2), and sets the fill color appropriately in the path’s style

The crux is the “key”: once SVG paths and HTML tds can be translated into the same String key, everything falls into place. If your SVG map and dataset agree, everything will go smoothly. In the case of this particular map, there are some issues which can be resolved by tweaking the SVG map, the dataset, and/or the JavaScript. (I tweaked all three.)

Adding the interactive legend is a simple JavaScript (and styling) exercise.

Pleasures

It’s interactive. Just play with the values above and watch it work. Isn’t it fun?

It’s simple.

Advantages of SVG come for free: for instance, maps scale very nicely to large sizes and high resolutions. To render a PNG version, just use “Print Screen”.

Every step is fast. The processing time scales linearly with the size of the dataset and the complexity of the SVG file. (Okay, the sort() I put in the analysis step doesn’t technically scale linearly, but it’s darned fast anyway.)

The data is open. You can just as easily set it up so that the data is a set of editable text-boxes or is one big CSV textarea. Imagine copy/pasting random spreadsheets into a page to produce maps. (The code, while simple, is not included here. Yet.)

The complex task of generating maps from data can be left in the hands of the slightly-computer literate. There’s nothing more satisfying to a programmer than programming himself out of a task forevermore.

Limitations

Great system, right? There are some flaws.

For one thing, Internet Explorer (up to version 8) doesn’t support SVG. You’ll have to find a workaround (complicated canvas magic? server-side SVG rendering?) which might remove interactivity. Or you can just ignore Internet Explorer or wait for people to take up Internet Explorer 9. (I used this particular map in an internal website at an office where nobody uses Internet Explorer.)

The SVG file is hard to get right: you’ll find problems in any map data, and if you’re trying to merge multiple maps (for instance, merging lakes and rivers with administrative regions) you might run into inconsistencies. I’m by no means an expert at generating an SVG map.

Your dataset and your SVG must share the exact same definitions. This can be finicky. In this example map of Tanzania, the SVG borders are at the district level while the dataset operates in terms of regions. The dataset isn’t perfect, either: it’s missing Manyara and the five regions of Zanzibar. Try not to think in terms of “correct” and “incorrect”: just get things to match up. Set VERBOSE=true in the JavaScript to display those inconsistencies.

The more time you spend, the more ideal your SVG and dataset can become.

Finally, if you’re trying to generate a map while keeping the data behind it private, this whole system will not work. If you “Print Screen” to make a PNG file, though, the PNG version won’t carry your data table.

Because each SVG file is unique and each dataset is, too, lots of the JavaScript code here isn’t portable to another project. So the last limitation of this mapping technique is its learning curve.

Still, it’s easier than setting up a map server.

Conclusions

I can’t think of another way to map data in such a quick and satisfying manner, either on web pages or in standalone applications. The concept and the code are simple and intuitive.

Internet Explorer may prevent this kind of map from adorning many websites, but intranets everywhere can use maps like this to bring data from complicated databases into simple graphics.

Legal

I declare this code to belong to the public domain in every way possible and in every country. Use it however you want without fear of reprisals from me (though if you seek to use these ideas or any others to plan raids and bombings, I urge you, on a more personal level, to reconsider your ambitions).

If you’d like me to share the code with you under a more restrictive license, send me the license file and I’ll probably agree.

Bodacity

Tue, 01 Sep 2009 15:07:42 -0400

I’ve just released some handy JavaScript toys as open-source: Bodacity JavaScript Utilities

Ruby's Object#respond_to?: String or Symbol?

Tue, 28 Jul 2009 15:27:41 -0400

When calling Object#respond_to?, use a Symbol when you can. Ruby’s internal method lookup uses “IDs,” which correspond to programmer-visible Symbols.

To get an ID from a Symbol, the Ruby interpreter uses a bit-shift, SYM2ID:

#define SYM2ID(x) RSHIFT((unsigned long)x,8)

To get an ID from a String, on the other hand, it does this:


static ID
str_to_id(str)
    VALUE str;
{
    VALUE sym = rb_str_intern(str);
    
    return SYM2ID(sym);
}

…and since that calls SYM2ID anyway, and we assume the shortest path between two points is a straight line, your code will be more efficient if it uses a Symbol.

Be wary of premature optimization, too: this tip is really to justify a convention, not to encourage you to rewrite all your code.

HTML Namespacing in Rails

Fri, 10 Jul 2009 17:53:00 -0400

CSS is a nightmare. In Rails, Sass makes it more bearable, but class naming is always frustrating: you must pick a convention before your project starts and follow it religiously, or your CSS rules will soon find themselves being applied way too liberally. Even with strict conventions, reality soon outpaces theory and your CSS directory becomes a mess.

In Rails, the truly annoying bit is the contrast with partials: CSS is a mess, but partials are organized so nicely. Why not organize your CSS to line up with your partials?

No more excuses. Here is a solution, fully functional and documented: HTML Namespacing.

HTML namespacing automatically annotates your HTML partials with special class attributes, and it lets you write Sass files with rules automatically scoped to those classes. Should you feel the need, you can even scope JavaScript files to those classes.

How? By using parallel directory structures. In your Rails framework, you can create a convention to tie the following files together automatically:

app/views/foos/_foo.html.haml: specifies the HTML
app/stylesheets/views/foos/_foo.sass: specifies the CSS
app/javascripts/views/foos/_foo.js: specifies the JavaScript

(I will probably be releasing a Rails plugin to ease JavaScript inclusion within the next month. And though HTML namespacing could apply to any web framework, my particular implementation is Rails-only.)

HTML namespacing is especially helpful when working in a team: nomenclature decisions are mostly handled automatically, and every developer knows where to put each CSS rule and line of JavaScript.

Give HTML namespacing a try. It is easy to integrate into existing projects, and it can save you headaches.

Full Sentences in Rails Validation Messages

Thu, 09 Jul 2009 11:35:21 -0400

This article assumes a rudimentary understanding of Rails Internationalization (or at least a working knowledge of Rails and a vague awareness that strings meant to be displayed to users can be stored outside of models, controllers, and views).

I will explain why Rails’s validation messages are incorrect, and I will provide a simple monkey-patch which can let you write less restricted validation messages. In the process, you should become more comfortable with translations and less comfortable with several Rails methods.

Rationale

There is a golden rule in i18n (“internationalization”): do not use string concatenation.

Why String Concatenation Breaks…

An example: I once worked on a website which lists organizations in both English and Swahili. In English, one could write, “found 16 companies”; in Swahili, this might translate to, “masharika 16 yamekutanwa” (if I studied my grammar correctly).

A programmer might be tempted to write:

s = I18n.t('Found') + ' ' + number + ' ' + I18n.t('company', :count => number)

with a translations file containing

en:
  company:
    one: company
    other: companies
  found: found

But in Swahili, the best this strategy could produce is, “Tumeyakutana 16 masharika,” which is completely out of order.

Instead, our intrepid programmer should write:

s = I18n.t('found n companies', :count => number)

with a translations file containing

en:
  found n companies:
    zero: found no companies
    one: found one company
    other: 'found {{count}} companies'

In English, the output is identical. At first glance, one might find the translation file repetitive (and clearer to read). But the real gain is that our program can now be translated to any language (depending on the I18n backend). We can now write a Swahili version of the string (excuse my grammar if I get these wrong):

sw:
  found n companies:
    zero: masharika sifuri yamekutanwa
    one: sharika moja limekutanwa
    other: 'masharika {{count}} yamekutanwa'

Problem solved! And we learned a valuable moral: do not use string concatenation to build sentences.

… Even in a Single Language

Another example: suppose an ActiveRecord validation produces an error. Rails will normally return a string such as “title: can’t be blank”. You may want to adjust that message to, “Please enter a title.”

ActiveRecord does not support this use case, despite the fact that ActiveRecord validations are advertised as i18n-aware. Why? Because ActiveRecord, at least in Rails 2.3, has a nasty tendency to build sentences through string concatenation: an absolute no-no.

I can help you fix those validation messages.

fix_active_record_validations_full_messages.rb

ActiveRecord::Errors has a method called full_messages which returns a list of all error messages on an object. In Rails 2.3, full_messages is the unlikely arbiter of error messages. Let us examine the original:

...
    def full_messages(options = {})
      full_messages = [] 

      @errors.each_key do |attr|
        @errors[attr].each do |message|
          next unless message

          if attr == "base"
            full_messages << message
          else 
            attr_name = @base.class.human_attribute_name(attr)
            full_messages << attr_name + I18n.t('activerecord.errors.format.separator', :default => ' ') + message 
          end  
        end  
      end  
      full_messages
    end

See the + signs next to I18n.t? Those are string concatenations: definitely off-limits. The funny thing is, by the time full_messages is called, all those messages have already been translated (in ActiveRecord::Errors#add).

Getting confused? That is not your fault. Suffice it to say, validations in Rails 2.3 are crufty, confusing, and ultimately incorrect.

The fix is simple: just remove the fancy logic from ActiveRecord::Errors#full_messages.

In your Rails project, put this in config/initializers/fix_active_record_validations_full_messages.rb:

# Ensures that when we pass a :message parameter to our validations, that
# message is a sentence (and not something to be prefixed by the column
# name). Rationale: ActiveSupport::Inflector is in over its head on this
# one.
#
# So instead of:
#   validates_presence_of :name, :message => 'should not be blank'
# Use:
#   validates_presence_of :name, :message => 'Name should not be blank'
#
# If, however, you just use:
#   validates_presence_of :name
# The behavior will remain unchanged.
if RAILS_GEM_VERSION =~ /^2\.3/
  ActiveRecord::Errors.class_eval do
    # Remove complicated logic
    def full_messages
      returning full_messages = [] do
        @errors.each_key do |attr|
          @errors[attr].each do |msg|
            full_messages << msg if msg 
          end 
        end 
      end 
    end 
  end 
end

config/locales/en.yml

The default Rails 2.3 error messages do not interpolate the attribute name: as we saw above, that is done in Rails 2.3’s flawed ActiveRecord::Errors#full_messages.

The sensible thing to do is correct the messages which, I think everyone should agree, are incorrect (because—you guessed it!—they are not full sentences).

Write this in config/locales/en.yml (or wherever your translations take place):

en:
  activerecord:
    errors:
      messages:
        # Default messages: complete sentences
        inclusion: "{{attribute}}: is not included in the list"
        exclusion: "{{attribute}}: is reserved"
        invalid: "{{attribute}}: is invalid"
        confirmation: "{{attribute}}: doesn't match confirmation"
        accepted: "{{attribute}}: must be accepted"
        empty: "{{attribute}}: can't be empty"
        blank: "{{attribute}}: can't be blank"
        too_long: "{{attribute}}: is too long (maximum is {{count}} characters)"
        too_short: "{{attribute}} is too short (minimum is {{count}} characters)"
        wrong_length: "{{attribute}}: is the wrong length (should be {{count}} characters)" 
        taken: "{{attribute}}: has already been taken"
        not_a_number: "{{attribute}}: is not a number"
        greater_than: "{{attribute}}: must be greater than {{count}}"
        greater_than_or_equal_to: "{{attribute}}: must be greater than or equal to {{count}}"
        equal_to: "{{attribute}}: must be equal to {{count}}"
        less_than: "{{attribute}}: must be less than {{count}}"
        less_than_or_equal_to: "{{attribute}}: must be less than or equal to {{count}}"     
        odd: "{{attribute}}: must be odd"
        even: "{{attribute}}: must be even"

These are the default Rails 2.3 validation messages with the attribute thrown in. With this en.yml and the above-mentioned monkey-patch to ActiveRecord::Errors#full_messages, Rails 2.3 will superficially behave identically to a non-monkey-patched version.

(Personally, I take this opportunity to make Rails’s default messages more human-friendly, by removing colons, adding periods, and rephrasing some sentences.)

Why

With our groundwork in place, Rails gets out of the way and lets us write what we want.

For instance, suppose we have an Article with a :body and a :title, both validated using validates_presence_of, with an extra validates_length_of on the :title. We can add the following in config/locales/en.yml, within the en: { active_record: { errors: ... } } section (alongside messages):

...
      models:
        article:
          attributes:
            body:
              blank: Please enter body text for your article.
            title:
              blank: Your article is desperately seeking a title.
              too_long: "Your article's title cannot exceed {{count}} characters. Give it a trim."

And there you have it: full sentences in validation messages.

The Last Word

I started this article under the guise of internationalization, but my actual code and use case are purely English. This is because you should not build sentences using string concatenation, no matter which language you speak (or, for that matter, in which language you program).

This article should help you avoid string concatenation in one part of Rails 2.3; there are many other errors in Rails 2.3 and perhaps even in your own code, and I hope you feel better-equipped to remedy those problems as well.

Appendix: Other Confusing Rails Methods

Here is a list of Rails methods which you should think before using:

ActiveSupport::Inflector#titleize (Never use this: it does not translate, and even in English it will fail you every time. Even the string used as an example in the method’s documentation, “Man From The Boondocks,” is titleized incorrectly by this method.)
ActiveSupport::Inflector#humanize (Never use this unless you are rewriting Rails’s internals. Maybe you want ActiveRecord::Base#human_attribute_name and ActiveRecord::Base#human_name?)
ActiveSupport::Inflector#ordinalize (If you mean to translate your application to non-English and you use ordinalize, you will need to override this method or define a replacement.)
ActiveSupport::CoreExtensions::String::Inflector#singularize and ActiveSupport::Inflector#singularize (Never singularize user-input strings, but feel free to singularize hard-coded strings such as class names and method names.)
ActiveSupport::CoreExtensions::String::Inflector#pluralize and ActiveSupport::Inflector#pluralize (Never pluralize user-input strings, but feel free to pluralize hard-coded strings such as class names and method names.)
I18n::Backend::Simple#pluralize (The Simple I18n backend is only correct for English and a few other languages (by happenstance)—pluralization rules are different in different languages. If you later translate to a language with more complex pluralization rules than “zero/one/many”, you will need to replace the I18n backend for a more complete pluralize method, but you will not need to change the files which called pluralize.)
ActionView::Helpers::TextHelper#pluralize (This is the helper you can call from your views. Never use it: it does not translate to other languages, and even in English you should be writing special strings instead of displaying “0” and “1”, as a matter of style.)

HTML Rendering Tests with Rails

Thu, 16 Apr 2009 15:06:22 -0400

The Problem

Picture the situation: 3pm on deploy day, after fully testing that your Rails application outputs the proper HTML, you run Capistrano and bring up the newest version of your beautiful website. An uneventful half-hour passes, after which you are told by your CEO that in Internet Explorer 7, a certain component of your web page is misaligned by 20px. While investigating further, you find your supposedly-vertical lists of checkboxes are being rendered in a diagonal line by Internet Explorer 6. Your deploy has transformed your once-perfect website into an unprofessional disaster.

You blow off dinner plans, fix bugs, validate all other pages on all major browsers, and finally, around 8pm, relax. At this point, the question is not so much, “why?” as it is, “how can we make sure this never ever happens again?”

Testing manually is simply too time-consuming (especially if you support many browsers). You want to go The Rails Way and automate your tests. How can you automatically render your HTML on various browsers and ensure the output looks right?

Assumptions

You are using Ruby on Rails.
You want to verify every pixel on every browser/platform in your tests.
You want your test to fail if any pixel on any browser/platform changes from what you verified.
Different browsers/platforms will very rarely produce exact, pixel-by-pixel matches, and the differences should be validated manually.
You do not mind if the HTML structure changes, so long as the rendered output does not change.
Tests should be easy to write and edit.

Overview

We need a test framework which can render any Rails partial or template using any browser/platform. Though we only intend to test our own templates, we are actually implicitly forced to test our models and the browsers we support, as shown in this diagram:

Implementation

The trickiest part of implementation is finding a way to automatically gather web browser output. This is especially difficult because:

Web browsers were historically not designed to render off-screen, so “render a PNG of a web page” is generally a difficult task involving very different code on each browser.
Rendering is a very slow operation, compared to, say, parsing an HTML document’s DOM structure.
Some web browsers are mutually exclusive (for instance, Internet Explorer 7 and Internet Explorer 8) and cannot usually appear on the same computer.

A distributed testing environment is necessary, either through physical or virtual machines. It will look something like this:

This leaves plenty of programs for us to write:

The “Test Runner”
The “Render Server”
The “Thumbnailing Program” (which is different for each browser/platform)

Render Server, Thumbnailing Programs

I have decided to bundle (platform-specific) thumbnailing programs with the (cross-platform) render server. The project is hosted on GitHub: html_to_png_server, which includes compiled versions but currently only handles IE6/7 on Windows and Firefox 3 on GNOME. (New thumbnailing programs, while trivial to plug into the render server code, are difficult to implement.)

The server accepts a POST request with HTML data, and it returns a PNG response. The URL determines which thumbnailing program (i.e., browser) to use for rendering. What a wondrous black box!

Visit http://github.com/adamh/html_to_png_server for details.

Test Runner

The test runner is written in Ruby and hosted on GitHub: html_render.

It contains a Rails-agnostic component, which is simply a client to the Render Server, and a Rails-specific testing component.

Visit http://github.com/adamh/html_render for details.

Your Render Test Framework

The idea is for Rails rendering tests to be as simple as possible to write, while incorporating your own application’s specific needs. You, the test automation engineer, will have two tasks:

Application-specific (or, if you feel creative, sub-application-specific) HTML-preparation and render server configuration: your project-specific framework.
Actual tests which use your framework.

Your Project-Specific Framework

Here is a sample framework, in which we have decided the following:

Tests are stored in RAILS_ROOT/test/render/
Tests are invoked by running ruby test/render_test.rb
You will render on winxp-ie6.local, winxp-ie7.local, and ubuntu-ff3.local

Create the file test/render_test.rb with the following contents:

require File.dirname(__FILE__) + '/test_helper'

require 'html_render/render_test/rails'

class RenderTest < ActiveSupport::TestCase
  extend HTMLRender::RenderTest::Rails::TestCaseDefinitions

  PREFIX = File.join(File.dirname(__FILE__), 'render')
  SERVERS = {
    :winxp_ie6 => 'http://winxp-ie6.local:20558/ie',
    :winxp_ie7 => 'http://winxp-ie7.local:20558/ie',
    :ubuntu_ff3 => 'http://linux-ff3.local:20558/ff3_linux'
  }

  class ApplicationRenderTest < HTMLRender::RenderTest::Rails::RenderTest
    def css
      @css ||= File.open(File.join(File.dirname(__FILE__), '..', 'public', 'stylesheets', 'application.css')) { |f| f.read }
    end

    def javascript
      '' # We're not ambitious enough to test our JavaScript just yet
    end
  end

  define_tests(PREFIX, :servers => SERVERS, :base_class => ApplicationRenderTest)
end

Your Tests

To test the partial /foos/_foo.html.erb (or .haml), create the file test/render/foos/_foo/basic/setup.rb with the following contents:

def locals
  { foo: Foo.new }
end

(Yes, that is the entire test.)

Here, we name the test basic: we may want to create several tests of the same partial, and for those we would replace “basic” with something else. The path, foos/_foo, implicitly specifies that we mean to test app/views/foos/_foo.html.erb (or .haml, as the case may be). Tests may define locals, assigns, css, javascript, or any helper methods deemed necessary. Behind the scenes, the contents of setup.rb will be appended to a subclass of ApplicationRenderTest. Any methods, up to and including html (the only method the testing framework actually uses) may be overridden.

Running Your Test Suite

First, download html_to_png_server (specifically, html_to_png_server.jar) to each rendering machine and start the server on each machine:

java -jar html_to_png_server.jar

Next, on the test runner computer (which holds your Rails project code), install the html_render gem:

sudo rubygems install adamh-html_render --source http://gems.github.com

Now you can run the test suite from your Rails project directory:

ruby test/render_test.rb

(This command could just as easily be made into a Rake task, of course, and included as part of rake test.)

All your render tests will fail: the expected images have not been defined yet! But each render has actually taken place, and the results are now viewable.

Making Tests Pass

Look in test/render/foos/_foo/basic. Where before you only had setup.rb, you will now have a run/ subdirectory with html.html, winxp_ie6.png, winxp_ie7.png, and ubuntu_ff3.png.

Here is the manual validation part: make sure each of those images looks the way you want it. If it does not, you will need to fix the partial (or, if you made a mistake settings up the framework, you will need to fix the stylesheets you are supplying in ApplicationRenderTest).

Assuming the images look correct, you can set the expected value. Choose the best-looking image and copy it to canonical.png (in the same directory as setup.rb). Now, should you add any new servers to your testing framework, their output will be compared pixel-by-pixel with the image in canonical.png (and the test will fail if the pixels do not match). Typically, canonical.png will be chosen from a Firefox server. (Not that I’m opinionated against Internet Explorer or anything.)

Now, if you run your test again, you will probably find that it still fails. While your ubuntu_ff3.png matches canonical.png as expected, the Internet Explorer images are not identical to canonical.png and so they trigger the failure. This failure is expected and healthy. To mark the images as valid, create a valid/ subdirectory in the same directory as setup.rb and copy winxp_ie6.png and winxp_ie7.png into it.

In the end, your single test’s folder will look like this:

test/render/foos/_foo/basic/setup.rb
test/render/foos/_foo/basic/run/html.html
test/render/foos/_foo/basic/run/ubuntu_ff3.png
test/render/foos/_foo/basic/run/winxp_ie6.png
test/render/foos/_foo/basic/run/winxp_ie7.png
test/render/foos/_foo/basic/canonical.png
test/render/foos/_foo/basic/valid/winxp_ie6.png
test/render/foos/_foo/basic/valid/winxp_ie7.png

…and your test will pass. (Until you change the partial, that is: try editing it and watch the test fail!)

Test Policy Considerations

When adding your tests to version control, ignore the run/ subdirectories: the test only really comprises setup.rb, canonical.png, and the images within valid/. (The run/ directory can be considered a temporary directory, though it is useful enough that the html_render test framework leaves it behind.)

You may also choose to never set a canonical.png. I chose to include it because it seems idealogically appropriate and it will help when developing a nice user interface for the test runner.

Summary

You now have a test framework which validates, with pixel-perfection, the way your partial/template is output from major web browsers. It can work at any detail level, from a single input box to your whole home page. You can also write tests of your HTML-producing helpers, independently of any actual templates (by overriding html in the test’s setup.rb).

The system is somewhat fragile: all render servers must be running html_to_png_server.

The system is somewhat slow: your render test suite will take approximately as long to run as it takes your slowest render server to render each partial.

But despite the system’s faults, you now confidently catch those irksome 20px Internet Explorer hasLayout issues before your boss does.

Looking Forward

This system is currently prototypical. Here, I discuss some problems I intend to solve.

The test suite is difficult to manage, with its masses of images in several directories. A graphical front-end to this large directory structure is a huge priority and should make the process of fixing tests much easier.

More challenging, the process of writing a test is not as seamless as it could be. I envision a point-and-click interface running on top of the actual Rails application, though of course this is easier said than done.

Optimizing jQuery-based JavaScript

Tue, 10 Feb 2009 18:48:10 -0500

I write this article as much for internal use as for others. Its target audience is competent jQuery users who need their code to run faster.

jQuery is fantastic, but its documentation does little to distinguish best practices from worst. In particular, jQuery supports very-slow and very-fast ways to do the same thing. What follows is a collection of my notes during two days dedicated to optimizing a large, JavaScript-intensive website.

Before You Begin

Before you begin to optimize, do not optimize. Only optimize:

When somebody complains;
When you think somebody is only not-complaining out of courtesy;
When you suspect somebody is about to complain; or
When you have implemented all functionality and you want additional bragging rights

Why? Because optimizing early is a waste of time: you are not able to guess where the bottlenecks are. Trust me.

Also, bottlenecks may be slightly different on different web browsers.

Okay, disclaimer out of the way, here is where your bottlenecks may be.

Step 1: Define Metrics and Targets

Do not optimize until you have metrics and targets. A fair metric is, for instance, “how long it takes $(document).ready() to run, in milliseconds.” A fair target depends on your worldview; 50ms might sound good for some pages. Different projects or optimization sessions may call for different metrics. Every metric needs a unit (usually “milliseconds”).

In our particular project, we invoke $(document).ready() in two places: either from an “initializer JavaScript” file or in <script> tags. Any code we run is within $(document).ready(). Since each file implements a different feature, we decided to measure each file’s delay separately so we could determine, feature-by-feature, whether to optimize or scrap our functionality.

Step 2: Measure

Design some way to quickly measure all your metrics.

Firebug has a very useful “Profile” mode which is simple enough to use. We wrapped our code like this:


if (window.console) window.console.profile("path/to/file.js");
[ ... code to profile ... ]
if (window.console) window.console.profileEnd();

For non-Firefox browsers, we added some pseudo-profiling code which will alert() a list of metrics after the page has loaded. This lets us at least verify our work, though it does not help with the actual work like Firebug’s profiling data does.


if (!window.console || !window.console.profile) {
  window.PROFILES = [];
  window.console = {
    profile: function(name) {
      window.PROFILE_NAME = name;
      window.PROFILE_TIME = new Date();
    },
    profileEnd: function() {
      window.PROFILES.push(window.PROFILE_NAME + ': ' + (new Date() - window.PROFILE_TIME) + 'ms');
    }
  };
  $(window).load(function() {
    // Ensure we finish $(document).ready() first....
    window.setTimeout(function() {
      if (window.PROFILES.length) {
        alert("Profiled:\n\n" + window.PROFILES.join("\n"));
      }
    }, 100);
  });
}

Step 3: Profile

For our particular website, because we use Firebug for our metrics anyway, profiling in Firefox amounts to loading our web page. We can see which functions are slow, which functions are called too many times, and which functions are not worth investigating.

The slowdowns generally fall into some basic categories, and the solutions are often transferrable site-wide.

Step 4: Optimize

Speed Up jQuery Selectors

Selectors come in varying flavours:

ID-based selectors, such as $("#foo"). These are blindingly fast.
Element-based selectors, such as $("input"). These are fast, but slower depending on what element you use. $("label") will probably be faster than $("div") if you use more @div@s than @label@s.
Class-based selectors, such as $(".foo"). These are less fast but more versatile.
Attribute-based selectors, such as $("[name=bar]"), and virtual selectors, such as $(':input'). These are painfully slow.

If jQuery.find() is dragging you down, your first order of business is to move your selectors up the above list. If you are using $(":input") on your entire site during page load, the browser needs to walk the entire DOM tree; change that to an element-based selector such as $("button, input, select, textarea") and the search will be an order of magnitude faster. If you are dealing with a unique element on the page, consider giving it an ID and selecting with that.

It is worth explicitly mentioning: avoid writing selectors without either element names or IDs. In other words, never use $(".foo") when you can write $("div.foo") instead.

These rules apply for compound selectors, too. For instance, rewrite $(".foo .bar") as $("div.foo div.bar").

Remove Selectors

If your selectors are still too slow, you may be able to remove them altogether. For instance, if we start with code like this:


$('div.geo').each(function() {
  var $geo = $(this);
  var lat = parseFloat($geo.children('div.latitude').text());
  var lng = parseFloat($geo.children('div.longitude').text());
  var image_path = $geo.children('div.image_path').text();
  // ...
});

Because there were so many calls to $geo.children() in our website (with maybe 75 $geo elements on a page), these selectors were excessively slow. The rewrite was not too painful:


$('div.geo').each(function() {
  var $geo = $(this);
  var lat, lng, image_path;
  $geo.children().each(function() {
    var c = this.className;
    if (/\blatitude\b/.test(c)) {
      lat = parseFloat($(this).text());
    } else if (/\blongitude\n/.test(c)) {
      lng = parseFloat($(this).text());
    } else if (/\bimage_path\b/.test(c)) {
      image_path = $(this).text();
    }
  });
  // ...
});

By eliminating all the excess selectors, we sped up our code by 40%.

Notice, by the way, an interesting tidbit from the above code:


$element.hasClass('foo'); // slow
/\bfoo\b/.test($element[0].className); // fast

This optimization works when hasClass() is being called very frequently on a jQuery object which we are sure holds exactly one DOM element.

Cache Selectors

If you have code like this:


$('div.foo div.bar div.baz1').something();
$('div.foo div.bar div.baz2').somethingElse();
$('div.foo div.bar div.baz3').somethingElseEntirely();

Consider rewriting to this:


var $bar = $('div.foo div.bar');
$bar.find('div.baz1').something();
$bar.find('div.baz2').somethingElse();
$bar.find('div.baz3').somethingElseEntirely();

Add Redundant Selector Elements

Imagine your page is divided into #header, #page, and #footer. Also, imagine #header contains a significant portion of the website—maybe 1/5 of it. Then, changing $("div.foo") to $("#page div.foo") may, in some cases, give you a 20% speed boost. Check this with your profiler. (Hopefully you’re scraping the bottom of the barrel at this point, as we are talking of 1ms-2ms differences here….)

Upgrade to jQuery 1.3

While jQuery 1.3 has some regressions compared with 1.2.6, it boasts a rewritten selector engine with a noticeable performance improvement. The above rules almost certainly apply anyway.

Avoid DOM Manipulation

DOM manipulation is slow. For instance, imagine a calendar written like this:


var $calendar = $('<table></table>');
for (var i = 0; i < 6; i++) {
  var $tr = $('<tr></tr>');
  for (var j = 0; j < 7; j++) {
    $tr.append('<td></td>');
  }
  $calendar.append($tr);
}

Yes, it’s pretty, but it is impractical. You will have to rewrite it to something more pragmatic, such as:


var $calendar = $('<table><tr><td><td>...</tr><tr>...</tr>...</table>');

(Except When Escaping User Input)

In general, a good portion of DOM manipulation can be circumvented. The speedups are enormous, but make sure you escape user-supplied HTML portions through jQuery’s interfaces.

That is, do not do this:


var $calendar = $('<table class="' + $('input.calendar_name').text() + '"><tr>...</table>');

…because a quotation mark in the $('input.calendar_name').text() would lead to unpredictable results. Use jQuery’s attr(), text(), and html() when dealing with strings which may be partially or fully constructed by your end user’s input.

Push Static Stuff to the Server Side

We use semantic HTML on our particular website, so it seems horrendous to us to write multi-column lists on the server side. Idealists as we were, we served our navigation menus in pristene HTML ul elements, relying upon a jQuery plugin to split the menus into smaller sub-lists as they were received by the client. Our profiler reported this splitting code as taking 130ms (DOM manipulations are expensive). We caved and split the menus on the server side, removing all that JavaScript altogether.

Manipulate the DOM Directly

jQuery’s DOM manipulation methods are brilliant, but they present some overhead. For instance, imagine an unoptimized method to create an “excerpt” of text which fits within a certain bounds, working something like this (untested) jQuery plugin:


jQuery.fn.excerpt = function(max_height) {
  return $(this).each(function() {
    var $e = $(this);
    var words = $e.text().split('\s*');
    var longest_fitting_string = '';
    var cur_string = '';
    for (var i = 0; i < words.length; i++) {
      cur_string = cur_string + ' ' + words[i];
      $e.text(cur_string);
      if ($e.height() > max_height) {
        break;
      } else {
        longest_fitting_string = cur_string;
      }
    }
    $e.text(longest_fitting_string);
  });
};

This method is terribly slow: in particular, $e.text() and $e.height() eat up too much time, as they are called dozens or hundreds of times in the loop.

Because we know there is exactly one element in our inner loop and we assume the element only contains text, we can replace $e.text(cur_string) with the following:


$e[0].firstChild.nodeValue = cur_string;

Likewise, we can use similar assumptions to transform $e.height() > max_height into the following:


$e[0].offsetHeight > max_height

These will make the excerpt() function many times faster than before (but, frankly, still not enough to use in a production website).

Many aspects of jQuery are surprisingly slow. css(), show(), hide(), and hasClass() spring to mind immediately, as they are so easy to replicate with similar, yet much faster, behaviour. If you are calling jQuery methods in a loop your profiler highlights, consider using direct DOM manipulation, as long as you are certain you are not relying upon some of the wonderful guarantees jQuery gives you.

Reimplement Algorithms

Computer science geeks excel here. Using the above excerpt() code as a typical example, we can generally say that many algorithms are slower than they need to be: not because of their tiny details (such as using $e.text() instead of $e[0].firstChild.nodeValue), but because they go about things the wrong way.

I left this optimization for the end for two reasons: it is often time-consuming; and my readership is likely divided between the camp who have done this already and the camp who do not know how. Basically, this is the “computer science” aspect of JavaScript coding.

Improving the running time of algorithms can be fun, and the profiler will drop hints when it is necessary. For instance, the profiler will show us that excerpt() is still too slow. A savvy computer scientist may then rewrite it to use a more clever heuristic, possibly based on a binary search, to eliminate 90% of the calls to $e.text() in typical calls to excerpt().

Optimizing in this manner is often (usually?) more effective than circumventing jQuery, and it leaves legible code. It works particularly well if you coded using Test-driven development since rewriting algorithms will often break them.

Step 5: Test

Ensure you did not break anything. There are many, many things you could have broken. Hopefully you unit-test your code….

Step 6: Repeat

Engineering is all about steps. Depending on your allocated time, shifting priorities, and mood, after optimizing you should return to one of the following steps:

Return to Step 5 (Test)
Return to Step 4 (Optimize)
Return to Step 3 (Profile)
Return to Step 2 (Measure)
Return to Step 1 (Define Metrics and Targets)
Return to Step 0 (Do Not Optimize)

Working With and Around FormBuilder

Tue, 11 Nov 2008 01:19:05 -0500

My current work is entirely focused on Ruby on Rails; and once you put Rails into focus, a lot of well-intended, badly-implemented designs peer back at you. Today’s topic: FormBuilder.

FormBuilder is a class most Rails users use transparently and most Rails users do not realize exists. It is the class of object returned by the form_for() and fields_for() helper methods.

Rails’s FormBuilder code looks like it was abandoned before being completed, and it suffers from abysmal documentation. Best practices, aside from being undocumented, do not cover all use cases—or even common use cases. In fact, best practices are downright nonexistent. Fortunately, all that can change with about 20 lines of well-placed code.

What `FormBuilder` does well

In your templates (depicted here in Haml instead of Rails’s default, ERb, as Haml is less sucky), FormBuilder shines with simple fields:


- form_for(@thing) do |f|
  %p
    = f.label(:name, 'Name:')
    = f.text_field(:name)
  %p
    = f.label(:description, 'Description:')
    %br/
    = f.text_area(:description)

Here, f is the FormBuilder instance. It provides the label(), text_field() and text_area() methods.

What `FormBuilder` does badly

Inevitably, in the course of a project, you will want to use more than text fields and date fields. And here, FormBuilder falls flat. For instance, a select field might look like this:


  %p
    = f.label(:thing_type_id, 'Type:')
    = f.select(:thing_type_id, ThingType.all(:order => :name).collect{ |t| [ t.name, t.id ] }, :include_blank => 'None')

This verbose amalgamation of complicated code and simple markup is even, with no apparent embarassment, suggested in the Rails API documentation.

But even select() is elegant compared to other common solutions FormBuilder supposedly provides. Writing a functional date_field(), needed for virtually any serious Rails project, is a nightmare; compound fields are nonexistent; and, most common and worst of all, there is no way to specify, for instance, “title fields should look like this”: the very Don’t Repeat Yourself principle Rails fans so avidly embrace.

Mediocre Solutions

Rails plugins may ease the pain, but they almost never do exactly what you want. Even if you find a perfect date_field(), you are only postponing an inevitable need for a more general solution.

Helper methods can work around some of FormBuilder, but if you resort to your own system of helper methods, you end up essentially engineering your own mini-framework of form helpers. The problem here is the unintuitive divide (what is FormBuilder and what is custom?) and a very inconsistent method signature (that is, the types and orders of parameters). And if you do not plan things out ahead of time, your helper code which began as a helper or two will end as a confusing mess.

Developers also get tempted to use Rails’s alternate tagging library methods, such as text_field_tag(), select_tag(), and other non-incumbered helpers. The problem with these is that they do not do the wonderful things FormBuilder does: automatically figure out element names and IDs, for instance, magically calculate fields’ initial values, or (prior to Rails 2.2) output valid HTML. These methods should be avoided when building a form for a model.

The funny thing is, FormBuilder actually does solve all these problems, almost. It really does get 95% of the way there. This article presents the last 5%.

How `FormBuilder` works

FormBuilder, implemented mostly in action-pack/lib/action_view/helpers/form_helper.rb (but spread into date_helper.rb and form_options_helper.rb in a textbook example of “when not to use Ruby’s class-reopening feature”), is implemented with three main pieces:

First, the helper methods: ActionView::Helpers::FormHelper defines several methods which are available to every view: text_field(), text_area(), select(), and so on. Most of them follow a simple calling convention:


  def something_field(object_name, method, options = {})

And here’s the magical part: if options[:object] is set, the helper will call options[:object].send(method): it will use the actual object to determine the initial value of the field.

Second, ActionView::Helpers::FormBuilder (which is not a helper at all): essentially, FormBuilder holds the object you are building a form for and serves as a proxy to the helper methods. It redefines every helper method with its own version:


  def something_field(method, options = {})

FormBuilder uses its own object_name and object methods, so that when your template calls:


= f.text_field(:name, :size => 15)

FormBuilder (in the above example, f) passes the call through as:


= text_field(f.object_name, :name, :size => 15, :object => f.object)

Lastly, InstanceTag: an implementation detail which may strike you at first as over-engineered, but upon further inspection rubs off as sideways-engineered at best and upside-down-engineered the rest of the time. Try to avoid calling its methods or modifying it whenever possible.

What We Want

We want a simple way to add new types of fields to FormBuilder: perhaps, extending the above examples, a thing_type_field() or a date_field(). I will showcase both below.

Implementation

I shy away from plugins and I encourage others to act the same way. If you do not understand this code, you should not use it: that would be applying my solutions to your problems, which only works if you have the exact same problems as I do, which is unlikely. Continue on your merry way until you reach the aforementioned impasses, and use this guide as an aid in solving your problems.

That said, here is my solution:

Create the directory lib/forms/ and add a file, lib/forms/application_helper.rb, and leave it nearly empty for now:


module Forms::ApplicationHelper
  # Helper methods go here
end

At the top of app/helpers/application_helper.rb, include it:


module ApplicationHelper
  include Forms::ApplicationHelper
  #... leave whatever was here before ...
end

Create a custom FormBuilder, in lib/forms, in a file called lib/forms/application_form_builder.rb:


class Forms::ApplicationFormBuilder < ActionView::Helpers::FormBuilder
  # Copied from FormBuilder. FormBuilder looks like it has some bright
  # engineering ideas but never finished implementing them. This *should*
  # be automated by defining "self.field_helpers", but it's used before
  # this class is loaded.
  Forms::ApplicationHelper.instance_methods.each do |selector|
    src = <<-end_src
      def #{selector}(method, options = {})
        @template.send(#{selector.inspect}, @object_name, method, objectify_options(options))
      end
    end_src
    class_eval src, __FILE__, __LINE__
  end
 
  private
 
  def objectify_options(options)
    @default_options.merge(options.merge(:object => @object))
  end
end

The above code is copied, almost word-for-word, out of Rails’s original FormBuilder.

Finally, we want to use our own ApplicationFormBuilder instead of Rails’s default, so add the file config/initializers/application_form_builder.rb with the following contents:


ActionView::Base.class_eval do
  def self.default_form_builder
    Forms::ApplicationFormBuilder
  end
end

Restart your Rails application (this is the last time—a subtle bonus) and we are now using our new-and-improved ApplicationFormBuilder. At the moment, it behaves exactly like Rails’s regular FormBuilder: it maintains all the great features, adding no overhead. But now we can extend it!

A better `date_field`

When engineering, always determine what the customer wants (which should be easy here, because you are the customer). In my case, I want to write this:


%p
  = f.label(:start_date, 'Start Date:')
  = f.date_field(:start_date)

Now, let’s make this work!

I use jQuery for all my JavaScript needs, which is fantastic because it lets me work without HTML IDs and it lets me remove 100% of my JavaScript code from my templates, which is good because code does not belong in templates. There are a couple of jQuery date-select plugins which border on acceptable; let us use the one which comes as part of the jQuery-UI download, ui.datepicker.js.

Our first hurdle: all these JavaScript date-picker fields submit text (for example, "Nov 10, 2008") which the model does not expect. We will need to edit our model to allow reading and writing a text representation. So let’s take our model, for instance, app/models/thing.rb


class Thing
  # ... typical model stuff ...

  def start_date_string
    d = self.start_date
    d.blank? ? nil : d.strftime('%b %d %Y')
  end

  def start_date_string=(value)
    d = nil
    begin
      d = Date.parse(value)
    rescue ArgumentError
      # d = nil
    end

    self.start_date = d
  end
end

(In a real project, this behaviour should be extracted into a mix-in.)

Now we have Thing::start_date_string() and Thing::start_date_string=(), which behave like Thing::start_date() and Thing::start_date=() except they allow human-readable strings that can be submitted via HTML forms.

We should add the necessary JavaScript includes, as well, in app/views/layouts/application.html.haml (or, as the case may be, app/views/layouts/application.html.erb—or whatever layout you happen to be using), in the header:


= javascript_include_tag(['vendor/jquery.js', 'vendor/jquery.ui.all.js', 'date_field.js'])

To tie our HTML into jQuery (since, as mentioned above, we write no JavaScript in our HTML), our templates will need to output text fields with a special class attached. Let us arbitrarily choose the HTML class, date_field.

Create public/javascripts/date_field.js with the following contents:


$(document).ready(function() {
  $('input.date_field').datepicker({
    // Add options here...
    dateFormat: 'M d yy'
  });
});

This JavaScript, when loaded, will turn any HTML input with class date_field into an interactive date selector.

Now, let us use our magical ApplicationFormBuilder to create the missing piece of the puzzle, by adding the following to lib/forms/application_helper.rb:


  def date_field(object_name, method, options = {})
    options = options.reverse_merge(:size => 12)
    if options[:class].blank?
      options[:class] = 'date_field'
    else
      options[:class] += ' date_field'
    end
    method = "#{method}_string".to_sym
    text_field(object_name, method, options)
  end

Presto! Your template should work now. If you have forgotten, your template looks like this:


- form_for(@thing) do |f|
  %p
    = f.label(:start_date, 'Start Date:')
    = f.date_field(:start_date)

And internally, that behaves as follows:

The template calls form_for(@thing) to create a new ApplicationFormBuilder, f. (The initializer we wrote specifies ApplicationFormBuilder instead of the regular FormBuilder.)
The template calls f.date_field(:start_date) (and maybe passes optional parameters, such as an HTML class or style). ApplicationFormBuilder::date_field() is not the helper you wrote: it is a new method, generated automatically because you wrote the helper method.
ApplicationFormBuilder::date_field() calls the helper you wrote, date_field(:thing, :start_date).
Forms::ApplicationHelper::date_field() calls text_field(:thing, :start_date_string, :object => thing, :size => 12, :class => 'date_field')
FormHelper (the Rails helper) creates the text field, with the date_field HTML class.
When the page loads, jQuery selects all text fields with the date_field HTML class and calls datepicker() on them, turning them into date pickers.

Whew, a lot of work, right? yes. But here are the advantages:

We can pick another jQuery-style date-picker implementation, and all we need to change is public/javascripts/date_field.js.
We can change the HTML completely, for instance reverting to Rails’s three-dropdown-box behaviour, by simply changing lib/forms/application_helper.rb.
It is easy to unit-test the helper, the template, and the JavaScript, since each has an isolated and distinct role.
Views receive the real benefits:
- There is no bloat in a simple call to f.date_field(:method): we remain readable and concise.
- Site-wide changes to date field behaviour can transpire without a need to edit templates.

As for drawbacks? This procedure is certainly complicated, so some people may prefer to just install a Rails plugin and be done with it. I have tried a few date selector plugins and found them all lacking in one way or another compared with this implementation, but your mileage may vary.

A better `select()`

While ranting above about FormBuilder shortcomings, I produced this piece of code, which the Rails API documentation shockingly presents as exemplary.


  %p
    = f.label(:thing_type_id, 'Type:')
    = f.select(:thing_type_id, ThingType.all(:order => :name).collect{ |t| [ t.name, t.id ] }, :include_blank => 'None')

What do we, the customer (template writer), want to type? Certainly not all that, since we may have ThingType selectors elsewhere in our application and we are almost guaranteed to need to change the code within as our application evolves. Helper methods are a common but unsatisfactory compromise. What we really want is this:


  %p
    = f.label(:thing_type_id, 'Type:')
    = f.thing_type_field(:thing_type_id)

And the implementation? Easy: Just add this to lib/forms/application_helper.rb:


  def thing_type_field(object_name, method, options = {})
    select(object_name, method, ThingType.all(:order => :name).collect{ |t| [ t.name, t.id ] }, options.reverse_merge({:include_blank => 'None'}))
  end

Advantages:

Our template is simple.
We reuse code and ensure consistency across the site by only writing our model query once.
Because the model query code is now in a helper (where it belongs) and not in the template (where it does not), it can (and should) be easily split into multiple lines and have more logic added.

Disadvantages:

We lose the nice html_options parameter in the select() method’s signature, which is inconsistent across the Rails API but useful nonetheless. It is straightforward to pass a :html_options item within our options hash to work around this problem.

A Nested Object Field

Rails is unhelpful when you want to include a nested object in your form. The best you can do is render a partial, but I find that annoying because it melds interface with implementation. How about this:


  def foo_field(object_name, method, options = {})
    foo = options[:object] && options[:object].send(method) || Foo.new
    fields_for("#{object_name}[#{method}]", foo) do |f|
      render(:partial => '/foo/form', :object => f)
    end
  end

And because at this point all your form partials use FormBuilder exclusively and never rely upon an @object or @foo or anything like that, this works without a hitch.

Parting Thoughts

This guide is the result of many of hours of hard work with FormBuilder, and I hope it helps save others that time. If you have any questions, comments, or even completely different and possibly better solutions, I would love to hear them.

Suckiness Engineering

Sun, 02 Nov 2008 01:24:23 -0500

Everything sucks.

This is a website dedicated to engineering and programming: to learning why computers do what they do, to understanding how to trick them into doing what you want, and to making the world (or at least, the computer programs therein) better.

Yet nobody seems to realize everything sucks, so I cannot resist a diatribe.

Computers Suck

Before computers were even invented, scientists already knew there are some problems computers can never solve. (They all stem from the halting problem: writing Program A to run Program B and determine whether Program B has gotten into an infinite loop: Program A, an idiot like all computer programs, is doomed to sit idly by and twiddle its thumbs while Program B goes crazy and explodes.)

Computers are very bad at certain tasks, too, like organizing an airplane-confined salesperson’s itinerary. They know what to do, yet they insist on taking millions of years in which to do it: a rather inconvenient delay, since they are bound to break before they finish.

But it might be a good thing computers are so inept. If computers could organize efficient itineraries, they would also break all our encryption algorithms (since if you knew how to organize efficient itineraries you could use that knowledge to decipher encrypted text).

Computer Programs Suck

A current computer can run billions of calculations every second, and yet it still takes a minute from when I turn it on to when I have Google open. I implicitly run this same test on every computer I use, with different suites of programs (written by Apple, Microsoft, or freedom fighters), and the story is always the same. Wait, wait, wait. Except for the times when something breaks and I never get my Google prompt after all.

A lot of things can break, too: viruses can latch onto your computer, physical defects can infest it, or, more enraging, you can click on that one pixel the developers thought you would never click on, drag your mouse over three pixels to some other pixel they assumed you would miss, and let go. I do house calls to fix computer problems from time to time, and the most common annoyance I see is the Microsoft Windows taskbar floating to the left of the screen or toolbars in Microsoft Word floating nowhere in particular.

And then… the crashes. Oh, Lord, the crashes.

Then, not to be forgotten, the inevitable “security bug” dilemma, which can render your wonderful web server into a spam-sending turd in the span of five minutes. Most of these bugs are extremely stupid; yet because of the halting problem, computers cannot help us find them all.

User Interaction Sucks

“But wait!” you say. “What about Google? Google Search doesn’t suck, does it?”

Yes, it does. If Google Search did not suck there would be no need for the “Search” button because the “I’m Feeling Lucky” button would always take you exactly where you want. In fact, there would be no button at all, because there would be no text field, because Google would already know what you want. Even better, you would never need to open the Google website at all, as Google would simply infuse the knowledge into your mind a millisecond before you even realize you want it.

Keyboards and mice and user interfaces and web pages are the first signs of imperfection in our computer programs. They make the user do something, which sucks because the user wants the computer to do it.

I hate using computers, and you should, too. But assuming you are a programmer programming programs for non-programmers, never forget that your users hate using computers even more.

Where Engineering Comes In

Everything about computers sucks. But some aspects suck less. And therein lies my approach to software engineering. This is a rehash of textbook processes, yet I have never seen it written down with this wording: probably because textbook writers consider the word “suckiness” taboo.

To write a program:

Define perfection. Because computers suck, perfection does not necessarily involve computers. Perfection involves users getting what they want with a minimum of hassle.
Define and measure how far your program is from perfection: your “suckiness”, if you will.
Remove as much suckiness as you can as quickly as you can.
Repeat steps 1-3 until you have overspent your resources or your program’s suckiness level is below your acceptance threshold.

Good engineering is all about cycles. The cycles can have any length (1-2 week cycles seem to work in several contexts) and you can repeat as many times as you like. You can alter requirements (Step 1) and their priorities (Step 2) as often as you like, and you can run as many or as few cycles as you like.

In the spirit of negativity, here is what happens if you try to skip the above steps:

If you neglect to define what you are trying to program, then you have implicitly decided to program “something”. And your “something” does exactly that, which is annoying because you realize soon afterwards that you meant for it to do what you want it to do, and meanwhile it is off “something”-ing. Then you need to change it.
If you neglect to prioritize, you spend ten days working on Internet Explorer 4 compatibility or something equally unimportant (for your purposes), leaving the important work unfinished and untested.
If you neglect to code, your code does not get written. More subtly (because more people seem to ignore it), if you neglect to test your code, your program’s suckiness does not change because you have no proof that the suckiness indeed did decrease as outlined in Steps 1 and 2. (How you test is up to you, but rest assured, you are testing.)
If you never repeat your cycle, you never incorporate new insights into your process. If a piece of code turns out to be difficult (Step 3), you may want to reprioritize it (Step 2) or even drop it completely (Step 1). Conversely, if you have hit your goals, you can stop repeating and go outside and play, confident that your program does not suck as much as, say, remaining indoors.

It may seem cynical, but this loop is very effective at suckiness-removal. And that is what software engineering is all about.

Perfect programs belong to computer programmers. Proofs that they do cannot exist belong to computer scientists. Real programs for real people are, whether you like it or not, whether you realize it or not, generated through software engineering.

“Why are you telling me what to do?”

I am not telling you what to do. I am showing you what you do already.

Everything from basement-tinkered tools to bank security software is built in this same manner. Not just with computers: with everything we create. In fact, it can be carried to things we do not create. Evolution, for instance, is a constant striving towards the perfection of remaining alive, with priority number one being having lots of babies.

Don’t be a wooly mammoth. Engineer your software.

Adam Hooper's Engineering Tips

refinery: a fast RAW decoding library

SVG/JavaScript mapping

Example

How it works

1. Create an SVG map

2. Output the data

3. Write the JavaScript

Pleasures

Limitations

Conclusions

Legal

Bodacity

Ruby's Object#respond_to?: String or Symbol?

HTML Namespacing in Rails

Full Sentences in Rails Validation Messages

Rationale

Why String Concatenation Breaks…

… Even in a Single Language

fix_active_record_validations_full_messages.rb

config/locales/en.yml

Why

The Last Word

Appendix: Other Confusing Rails Methods

HTML Rendering Tests with Rails

The Problem

Assumptions

Overview

Implementation

Render Server, Thumbnailing Programs

Test Runner

Your Render Test Framework

Your Project-Specific Framework

Your Tests

Running Your Test Suite

Making Tests Pass

Test Policy Considerations

Summary

Looking Forward

Optimizing jQuery-based JavaScript

Before You Begin

Step 1: Define Metrics and Targets

Step 2: Measure

Step 3: Profile

Step 4: Optimize

Speed Up jQuery Selectors

Remove Selectors

Cache Selectors

Add Redundant Selector Elements

Upgrade to jQuery 1.3

Avoid DOM Manipulation

(Except When Escaping User Input)

Push Static Stuff to the Server Side

Manipulate the DOM Directly

Reimplement Algorithms

Step 5: Test

Step 6: Repeat

Working With and Around FormBuilder

What FormBuilder does well

What FormBuilder does badly

Mediocre Solutions

How FormBuilder works

What We Want

Implementation

A better date_field

A better select()

A Nested Object Field

Parting Thoughts

Suckiness Engineering

Computers Suck

Computer Programs Suck

User Interaction Sucks

Where Engineering Comes In

“Why are you telling me what to do?”

What `FormBuilder` does well

What `FormBuilder` does badly

How `FormBuilder` works

A better `date_field`

A better `select()`