urgetopunt technologies blog

Better Conditional Sum in Google Spreadsheet

2009-11-06T00:00:00+00:00

In the original article I described using SUM() and FILTER() in a Google Docs spreadsheet to calculate the sum over a subset of cells within a column. Turns out, there’s a better way:


=SUMIF('Worksheet name'!E:E, "food", 'Worksheet name'!D:D)

In this version, the SUMIF() function combines the behavior of the SUM() and FILTER() functions I was using before. The first argument is the column to be compared to the filter value, the second argument is the filter value itself (or the cell address containing the filter value) and the third argument is the column over which the sum is to be calculated. So in the above example, in column D of the worksheet named “Worksheet name” all cells for which the corresponding cell in column E contains the value “food” are selected, and the sum of those select cells is returned.

In addition to being more succinct and easier to read, the use of SUMIF() has the added benefit of returning 0 if there are no rows matching the filter. Using SUM() and FILTER() instead returns #N/A.

Using Passenger with rvm

2009-09-27T00:00:00+00:00

I’ve recently starting using Ruby Version Manager (aka rvm) to manage multiple ruby versions on my workstation. So far I’m quite pleased with the ease of use. (My only gripe so far is that the rvm command is uncomfortably close to rm — it’s only a matter of time before I shoot myself in the foot).

There is one gotcha. In production I use Phusion Passenger in conjunction with Ruby Enterprise Edition. I would like to use the same environment in development. However, when installing the passenger gem into an rvm-managed ruby installation, the path to the ruby binary (which you need to plug somewhere into your Apache config) will not work. It will have an incorrect gem path, and in my case, that resulted in passenger failing to spin up because it was unable to load the fastthread gem.

/Users/jparker/.rvm/ruby-enterprise-1.8.6-20090610/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- fastthread (LoadError)
        from /Users/jparker/.rvm/ruby-enterprise-1.8.6-20090610/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
        from /Users/jparker/.rvm/gems/ruby-enterprise/1.8.6/gems/passenger-2.2.5/lib/phusion_passenger/utils.rb:28
        from /Users/jparker/.rvm/gems/ruby-enterprise/1.8.6/gems/passenger-2.2.5/bin/passenger-spawn-server:53:in `require'
        from /Users/jparker/.rvm/gems/ruby-enterprise/1.8.6/gems/passenger-2.2.5/bin/passenger-spawn-server:53

The solution is very clearly printed on the rvm web site. Instead of passing the path to the actual ruby binary to the PassengerRuby directive, pass in the RVM_ROOT/bin/ruby-VERSION (replacing RVM_PATH with the root of your rvm installation [usually $HOME/.rvm] and VERSION with the ruby version you want to use [in my case ruby-enterprise-1.8.6-20090610]).

# Don't do this
PassengerRuby /Users/jparker/.rvm/ruby-enterprise-1.8.6-20090610/bin/ruby

# Do this instead
PassengerRuby /Users/jparker/.rvm/bin/ruby-enterprise-1.8.6-20090610

The latter is a shell script which execs the actual ruby binary after setting an appropriate GEM_HOME and GEM_PATH (among other things).

Using ERB in YAML Configuration File

2009-09-12T00:00:00+00:00

A while back in Railscast #85 Ryan Bates demonstrated how to add a YAML-based configuration file to a Rails application. You start with the configuration file — say RAILS_ROOT/config/app_config.yml — containing your configuration data:

And then you load the file from an initializer — say RAILS_ROOT/config/initializer/load_config.rb — containing the following:

And from then on a Hash named APP_CONFIG will be available throughout your application containing the configuration specific to the environment in which your application is running, i.e., development, test or production.

But what if you want to dynamically configure one or more values in your configuration file? Other YAML files loaded by Rails such as fixture files or database.yml are processed through ERB before being loaded. Wouldn’t it be nice to be able to do the same in your application configuration file?

As it is Rails will not process those ERB snippets, but you can change that with one small change to your initializer:

Now, instead of loading the file directly, YAML loads the string returned by ERB#result which will contain the contents of the app_config.yml after the ERB snippets have been evaluated.

Launchd for Cron Jobs

2009-08-30T00:00:00+00:00

Although cron is nominally supported on OS X, the preferred alternative seems to be launchd. It is often used to run jobs at startup (much like an init script), but it also has configuration options to produce cron-like scheduling. For more information, check out the launchd(8) and launchd.plist(5) man page, and pay particular attention to the StartCalendarInterval attribute.

This is a plist file I’m using to get launchd to periodically run a script which pulls down copies of database dumps from remote servers and then uploads them to S3:

The file lives in $HOME/Library/LaunchAgents/com.urgetopunt.backup.plist, and it was loaded into launchd by running the following command (no root privileges necessary):

$ launchctl load -w $HOME/Library/LaunchAgents/com.urgetopunt.backup.plist

Crontabs are still, in my opinion, much easier to deal with than XML documents, but launchd does offer certain advantages for desktop platforms. If my Mac is asleep when a cron job is scheduled to run, that run is missed. If launchd determines that a run was missed because the system was asleep, it runs the job when the system wakes up. If multiple consecutive runs are missed, it only runs the job once to get caught up.

Fibonacci Sequence in a Hash

2009-08-13T00:00:00+00:00

Being able to instantiate a Hash that calculates Fibonacci numbers is just another reason I like Ruby…

It may not be as compact as what Perl 6 promises, but it’s a lot more legible.

OSCON Highlights

2009-07-25T00:00:00+00:00

OSCON is over. While there were a number of interesting sessions, the following two struck me as particularly interesting.

7 Principles of API Design

Damian Conway’s API design tutorial used Perl for the examples, but the ideas can reasonably be applied to most other languages as well. The talk was organized around Arthur C. Clark’s third law of prediction:

Any sufficiently advanced technology is indistinguishable from magic.

The idea is to develop software modules that are so clever they can be useful with a minimal interface or even no interface at all. A good example of this is Perl’s strict pragma which does everything expected just by adding use strict.

The seven principles follow:

Do one thing well. Keep methods small and tightly focused.
Design by coding. Designing APIs around expected usage leads to intuitive, easy-to-use APIs.
Evolve by subtraction. Squash needless complexity whenever you can. Find better defaults.
Declarative beats imperative. Let users say “what” rather than “how”.
Preserve the metadata. If a module knows something useful at one point, it should remember that information later on.
Leverage the familiar. The easiest interface to understand and use is one you are already using.
Best code is no code at all. Modules that can be used with very little conscious effort can be a delight to use.

The HTML 5 Experiment

Bruce Lawson discussed the future with HTML 5. All of the major browser players — Microsoft (IE), Mozilla (Firefox), Apple (Safari), Opera (Opera) and Google (Chrome) — support some subset of the features of HTML 5 already, so you can start playing with it today. HTML 5 is (mostly) a superset of HTML 4 which means HTML 5 pages will usually degrade with some semblance of grace. Because it is so common for page structures to include distinct areas for headers (not to be confused with HEAD), footers and navigation menus, HTML 5 has header, footer and nav tags. Unlike a div with a specific id attribute, these tags have semantic meaning that can be used by page readers. Forms in HTML 5 have a whole slough of features that previously could only by achieved with a lot of Javascript. Among these features are common validations (required fields, valid email formats, valid numbers, etc.), an autofocus attribute and a calendar picker.

Open Source Bridge -- Day One

2009-06-17T00:00:00+00:00

Amber Case on our gradual transition into cyborgs. Very funny talk. I wouldn’t normally think of myself or any of my fellow human beings as cyborgs, but when you stop to think about the gadgetry that we keep on our persons at just about all times nowadays…

Kurt von Finck and the “hacker business model”. What if, instead of specifically asking employees to put in 7.5 hours a day, you asked them to put in 75 hours every two weeks? It’s an interesting idea. While many programmers do their best work with fixed-width, evenly-spaced work shifts, others thrive on binge coding. Perpetual binging is unmaintainable, but with nice, long breaks between sprints, it can be quite productive.

J. Chris Anderson presented Deploying to the Edge from CouchDB. A good overview of CouchDB which seems to get more and more attention these days. CouchDB is not a traditional relational database but rather a document-based database. It’s written in Erlang and communication is based on HTTP and JSON. Among its most appealing features is its capacity for distributed and even embedded operation. It has enticing potential for offline applications.

Scott Becker spoke about Agile Javscript Testing. Informative presentation on a topic on which I really need to get up to speed. He demonstrated Screw.Unit, a BDD framework inspired by RSpec. He also demonstrated Blue Ridge, a Rails plugin which combines several Javascript-testing tools (including Screw.Unit) and paves the way for integrating Javascript tests into the rest of the test suite for Rails applications.

There were some promising sessions in the afternoon schedule, but, tragically work beckoned and I spent the better part of the afternoon on the computer.

The Importance of Being Specific

2009-06-05T00:00:00+00:00

Do not do this in your .autotest file.

Instead, be specific.

One of the benefits of autotest is that it can save you time. It’s not saving you time when you spend 15 minutes trying to figure out why it isn’t running any of the tests from vendor_controller_test.rb. Be good to your tools, and they will be good to you. Tell them exactly what it is you’d like to do, and they will be happy to oblige. Read the documentation about a feature when you use it — then maybe you’ll realize that a method actually expects a Regexp as an argument.

If you don’t read the documentation, you risk being a tool… Like me.

Refactoring Views for Clarity

2009-05-28T00:00:00+00:00

I recently found myself doing some work on the views of an application I developed more than 18 months ago. As seems so often the case when looking at something I wrote long ago, I found myself somewhat dissatisfied with the way some of the code had been written. The partials I was looking at were DRY but long. In Ruby I’ve learned to extoll the virtues of shorter, more focused methods, and in Rails, by extension, I think these same virtues apply to smaller, more focused views.

The controller I was working on handled the usual RESTful actions for a resource named billable, but in addition to the usual suspects, the billable resource also had several custom collection actions named day, week and month. As you might guess from their names, they returned collections of billables by day, week and month, respectively. The base views were simple — they displayed a title and rendered a _billables partial which displayed a table summarizing the collection. The _billables partial was my primary concern as it had grown too long. (See the original code below).

I decided to address the problem breaking the _billables partial up. I created _head and _foot partials to display the header and footer sections of the table, and I created a _billable (singular) partial which displayed a row for individual billables. I opted to shed some of the original approach’s DRYness by replacing the one call to #render in each view with two calls for the _head and _foot partials and a third call for the _billable partial for the entire collection. (See these changes below).

While I am not necessarily wed to the details of this new approach, I do think the result is a net gain. There are more files, more lines of code and there’s more duplication. But I believe everything is clearer. I also believe the duplication I’ve added is unlikely to add maintenance headaches — it’s hard to imagine it changing in any way that wouldn’t require significant updates with or without the duplication. (As long as I am using a table to display the collection, I will have a header, footer and main body).

These are the partials I started with:

These are the partials I ended with:

Conditional Sum in Google Spreadsheet

2009-05-09T00:00:00+00:00

It’s the simplest things I always forget. Hopefully I’ll remember this article next time. When traveling to a conference I record and categorize my expenses in a Google Docs spreadsheet, usually creating a new worksheet for each trip. I have a separate worksheet which contains a cumulative summary of each category of expense. This is the formula I use to keep the running total for specific categories.


=SUM(FILTER('Worsheet name'!D2:D; 'Worksheet name'!E2:E="food"))

The SUM function is self-explanatory. The FILTER function selects all cells from a given range which match particular conditions. The range of cells to choose from is the first argument. Subsequent arguments describe the conditions. In the above example we select all cells from column D from row 2 onwards for which the value of the corresponding cell in column E equals “food”.

UPDATE: As is so often the case when I learn something, there is a better way.

RailsConf Day Four

2009-05-07T00:00:00+00:00

I attended HTTP’s Best-Kept Secret: Caching with Ryan Tomayko for the early morning session. Ryan gave a brief overview of different types of HTTP caching (client, shared proxy and gateway), eventually focusing on the server-side gateway caching. This form of caching still sends data over the wire, but allows you to avoid hitting Rails entirely, or else — using something like ETags — allows you to minimize the amount of work done within Rails to generate the content. It sounds like HTTP caching in its current form is still a bit awkward to handle when the content of the response varies based on session state, e.g., whether or not the user is logged in.

Mid morning session found me at When to Tell Your Kids About Presentation Caching with Matthew Deiters. This covered some of the same material as the previous caching talk, but Matthew focused more on minimizing the amount of data sent over the wire. In addition to client-side caching, he covered some general tips on reducing the size of the server responses. Tips included reducing the number of resources (e.g., use Rails’ asset caching functionality to condense multiple javascript/CSS files into fewer [but larger] files) and reducing the size of resources through minification (obfuscation) and compression (e.g., Apache’s mod_deflate).

My late morning session was It’s Not Always Sunny in the Clouds: Lessons Learned with Mike Subelsky. Mike described some of the surprises and challenges he encountered over the past year working with Amazon EC2. He’s still a fan of the power and convenience introduced by cloud computing, but he’s developed a healthy respect for the complications and expenses it introduces over the old tangible, colocated server route. Turnkey provisioning rocks, but it involves a lot of work.

RailsConf Day Three

2009-05-06T00:00:00+00:00

My early morning session was Using metric_fu to Make Your Rails Code Better with Jake Scruggs. Aside from rcov and the stats rake task in Rails I haven’t yet spent much time studying code metrics. Jake touched on a number of different tools available today, each of which looked quite interesting and each of which is (or can be) used within metric_fu. The metric_fu rake tasks generate reports which can provide useful information on where you might need to focus your refactoring efforts by identifying potential problem spots like overly complex methods and repetitious code. I’ve seen the announcements about metric_fu, but hadn’t taken a close look at it until this session. I find I’m eager to try it out but dreading what I will discover.

In the late morning I attended Are You Taking Things Too Far? with Michael Koziarski. The basic message of this session is pretty easy to extract from the title, and it’s one that I’ve heard in several sessions this week — don’t be overly dogmatic. Rails has introduced plenty of conventions that, in general, make our jobs easier, but it’s important to remember that, sometimes, the conventional way may not be the best way. Sometimes fiercely sticking to convention can mean writing more code than you really need that requires more maintenance than you have time for and which expresses your intent less obviously than a simpler, less conventional approach. Conventions are good, but sometimes you need to be willing to break with them.

Early afternoon session probably would have been interesting, but I had to catch up with work (coffee in Las Vegas isn’t cheap).

For the mid afternoon session I attended Working Effectively with Legacy Rails Code with Pat Maddox and BJ Clark. This was a particularly appealing talk, as it’s a challenge I’ve spent a lot of time wrestling recently — taming Rails code I wrote before I’d learned many of Rails most helpful features. One of their suggestions was to keep an eye out for occasions where complex, semi-redundant code can be abstracted into a “mini framework” that you can mix-in as needed, and, as a corollary, keep an eye out for occasions where convoluted code may be unnecessry because Rails already provides a facility to do what needs to be done. They also announced their new blog.

Late afternoon session %w(map reduce).first — A Tale About Rabbits, Latency and Slim Crontabs with Paolo Negri. Paulo gave an interesting overview of how and why to use RabbitMQ from Ruby. This is one of those times where the topic is fascinating, but I really don’t have any immediate application for it.

The evening keynote with Bob Martin was a particular treat and one of the most entertaining I’ve attended in years. The value of testing was the core of the talk. He presented the three rules of true test-driven development:

You are not allowed to write a single line of production code until you have written a failing test.
You are not allowed to write a single line of additional test code once you have a failing test.
You are not allowed to write a single line of production code beyond what is needed to make the failing test pass.

Following those rules to the letter requires extraordinary discipline, but he asked us to think what our development world would be like if we followed through. Tests eliminate fear and risk — if you have the tests, you can refactor the production code with confidence. Ruby (and Rails) can spare itself the fate of Smalltalk (which he said basically died of hubris) by three things:

Professional discipline — specifically, writing well-tested code.
Professional humility — not encouraging and adversarial relationship with other languages.
Professional responsibility — not shying away from fixing dirty coding problems when they are discovered.

“Uncle Bob” is a marvelous presenter and showman in general, but one comment near the end struck me as particularly funny:

[When Smalltalk died, its programmers] had to start writing Java, and it nearly killed them.

RailsConf Day Two

2009-05-05T00:00:00+00:00

Early morning session was The Even Darker Art of Rails Engines by James Adam. Good overview of Rails Engines as they are implemented in Rails 2.3, including caveats of some of the shortcomings. One issue is that when application routes and engine routes collide, the engine route takes precedence. James considers this a bug (I’m inclined to agree). Another gotcha is migrations. Migrations in an engine are not visible unless they are copied into the top-level migrations directory, but this is tricky because the version number/timestamp on a migration in the application may collide with that of a migration in the engine. Finally, public assets that are bundled with an engine must be copied into the application’s public directory in order to be visible to the web server. Overall, engines have a lot of potential as well as room for improvement. As things stand, they are reasonably easy to deal with across in-house applications, but it becomes much more complicated when they are published by third parties. It’s also worth noting that Rails 3 promises to have mountable applications based on Rails Engines and Merb Slices.

Late morning session was In Praise of Non-Fixtured Data by Kevin Barnes. It was interesting to see some of the options for avoiding fixtures that have arisen although for day-to-day work I have already switched over to Factory Girl. I doubt it’s rational, but the idea of data generators being added directly to the model class bothers me. That’s one of the reasons I stick with Factory Girl.

Early afternoon session was “The Future of Deployment: A Killer Panel” with Marc-André Cournoyer, Christian Neukirchen, Ryan Tomayko, Adam Wiggins, Blake Mizerany and James Lindenbaum. The panel members represented different layers of the deployment stack including Thin, Rack, Rack::Cache, Sinatra and Heroku. Discussion centered around how the different components came into being, how they’ve come to interact and how they might need to develop for the future. If I spent more time working with complex deployment issues, I probably would have gotten more out of this session, but as it is my needs are quite simple.

Mid afternoon session was I Rock, I Suck, I Am — Jumpstart Your Journey to Agile by Davis W. Frank. Davis went over some practices and guidelines that helped him adapt to the Agile workflow. Sometimes it seems that Agile pays off the same way that REST does — it’s a simple, well-structured way to breakdown complex tasks.

Late afternoon session was Obie Fernandez’s Blood, Sweat and Rails. An interesting talk about some of the perils of launching a formal consultancy. Dealing with contract lawyers and keeping up with collections are serious drawbacks.

RailsConf Day One

2009-05-04T00:00:00+00:00

The tutorial day for RailsConf 2009 has drawn to a close. I spent the morning in Running the Show: Configuration Management with Chef by Edd Dumbill. The first half of the tutorial was a bit awkward when the live demo failed to get off the ground. While I don’t feel I walked away with any concrete training, I did come away with enough appreciation of Chef that I’m looking forward to learning and using it to rein in the increasing number of nearly identical systems I find myself managing.

In the afternoon I attended Joe O’Brien’s and Jim Weirich’s Testing, Design and Refactoring. The first half of the tutorial consisted of presentation by both Jim and Joe. Part of Jim’s talk included the material from his “Writing Modular Code” talk from Scotland on Rails. Joe’s portion covered some of the general methods of refactoring and why you might use them. One concept that struck a chord with me was the concept (or goal) of simplicity. Simple code has the following characteristics:

Passes tests, i.e., the code works
No duplication
Clearly expresses intent
Minimal number of classes and methods

And those characteristics are in order of importance. If the code doesn’t work, it doesn’t matter whether or not it’s clear. (I’m a little fuzzy about the order of duplication and clarity in that list though. If eliminating duplication results in loss of clarity, that may not be a net win.)

Scotland on Rails 2009

2009-04-01T00:00:00+00:00

Scotland on Rails 2009 was held at the University of Edinburgh 26-28 March. I confess that when I registered and made travel plans I regarded the conference as a good excuse to visit Scotland but not much more. It ended up being one of the finest conferences I have attended in a while. Good assortment of sessions, fascinating speakers, a good venue and superlative organization made the conference as valuable an experience as it was enjoyable. The conference organizers deserve a lot of credit for the job they did.

For me, the highlight sessions included:

Marcel Molina’s opening keynote — fascinating and humorous insight into the early development of Rails.
Jim Weirich’s session “Building Blocks of Modularity” — turned out to actually be a thought provoking “Grand Unified Theory of Software Development”.
Joe O’Brien’s and Jim Weirich’s “Ruby Code Review” — shined a light on taking simple steps to rein in untested code (and earned a good many guffaws in the process).

But, yeah, it was still a good excuse to visit Scotland.

Connascence and Software Development

2009-03-27T00:00:00+00:00

This morning at Scotland on Rails Jim Weirich gave a fascinating talk covering a software development pattern called connascence_. His slides are available on Githubconnascence.

Connascence describes a kind of coupling within your code. Jim covered several forms:

Connascence of Name (CoN)
Connascence of Position (CoP)
Connascence of Meaning (CoM)
Contranascence (CN)
Connascence of Algorithm (CoA)
Connascence of Timing (CoT)

In addition to the various forms of connascence he described two rules for dealing with connascence in your software.

Rule of Locality — As the distance between software elements increases, use weaker forms of connascence.
Rule of Degree — Convert high degrees of connascence into weaker forms of connascence.

His slides have good examples of each of these forms, but consider a quick example. If a method has several — say 4 or 5 — positional parameters, it has a high degree of connascence of position. Over time, if parameters have to be added, removed or just reordered, this code will prove to be brittle as any code that calls that method would also have to be updated. The situation can be improved by replacing those positional parameters with a single hash parameter. The hash keys introduce connascence of name but eliminate concern over position, decoupling the method defintion somewhat from the code that calls it.

I haven’t yet attended a talk by Jim that wasn’t enlightening (I got started with Ruby because of his “Ruby for Java Programmers” talk at OSCON 2005), and this talk was no different. Seeing these patterns isolated and explained helps me with the ongoing goal of achieving greater discipline as a software developer. Thanks, Jim.

Hidden Files and the Finder

2009-03-23T00:00:00+00:00

If you are working on a Mac and have an unfortunate run-in with a mistyped rm -r command, you may find yourself in need of restoring files, including those elusive dot files which don’t show up in the OS X Finder by default. If you’ve been using Time Machine to backup your data, doing restores is easy, but before you can restore hidden files, you have to tell the Finder to display them.

The following commands will do just that:

After the Finder has restarted, you can go into Time Machine and restore hidden files to your heart’s content. When you’re done, if you want to turn display of hidden files back off (they do clutter up Finder windows and the Desktop), the reverse is intuitive:

Voilà.