Craic Computing Tech Tips

Migrating from Rails HABTM to has_many :through

2013-06-13T15:29:00.001-07:00

I had a has_and_belongs_to_many (HABTM) association that I needed to convert to has_many through with an explicit model that contains additional columns.

In my example a User knows about many Words and each Word is known by many Users.

You can find many pages on the web about the differences. The consensus is that has_many :through is the way to go from the start - and after this process I agree.

Making the change in itself is not a big deal - drop the existing joining table 'users_words', create the new one, update the User and Word models and you're good to go.

Problem is that I already had data in the joining table...

And because the standard way of setting up a HABTM joining table does not include an id column, you can't just use that table or directly copy each record from it. Dang it...

Here were my steps - hopefully I got them all - don't skip any !

1: Backup your database and prevent users from accessing it

2: Do not touch any of the old associations, table, etc

3: Create the new table and model with a different name from the old joining table.
My HABTM table was users_words and my new table is user_work_links

4: Update the two models
My original association was this - do not change it yet !

  has_and_belongs_to_many :words, :uniq => true

The new association is this - NOTE the second line is commented out for now - VERY important !

  has_many :user_word_links, :dependent => :destroy
  # has_many :words, :through => :user_word_links, :uniq => true

5: Copy over the data from the old joining table with a rake task
You need to go through the existing associations one by one to get the ids for records in the two tables.
Here is my script:

namespace :craic do
  desc "move user word data"
  task :move_user_word_data => :environment  do
    users = User.all
    users.each do |user|
      user.words.each do |word|
        record = UserWordLink.new({ :user_id => user.id, :word_id => word.id })
        record.save
      end
    end
  end
end

6: Update the two models
Now you can comment out the old associations and uncomment the new ones

  # has_and_belongs_to_many :words, :uniq => true
  has_many :user_word_links, :dependent => :destroy
  has_many :words, :through => :user_word_links, :uniq => true

In the attr_accessible lists in the two models be sure to add :user_ids in the Word model and :word_ids in the User model. If your forget this it will silently fail to create the records

7: Test Test Test
You should be back up and running with the new model

8: Remove the old table
Finally create a migration that drops the old table and run it

Not too bad as long as you think it through before you start and don't rush the steps

Unix command 'comm' for comparing files

2013-06-07T14:23:00.001-07:00

I needed a simple way to compare two similar files and only output lines that were unique to the second file. Sounded like a job for 'diff' but I was not finding the right options to give me what I needed. And then I stumbled across 'comm' - a standard UNIX command that I don't think I have ever used. That does exactly what I needed. My two files look like this
File A

A
B
D
E

File B

A
B
C
D

I want the command to just output 'C' By default comm compares two files and produces 3 columns of text - lines that are only in file A, lines that are only in file B and lines that are in both. So with these two files I get:

$ comm tmp_A tmp_B
      A
      B
   C
      D
E

Ugly, and not what I want... But then you can suppress the output of one or more of these columns using -1, -2, -3 options and combinations of those. I want to suppress lines that are only in file A and those in common:

$ comm -13 tmp_A tmp_B
C

Simple - does exactly what I need - can't believe I didn't know about it...

Listing all the models and numbers of records in a Rails application

2013-05-31T09:51:00.000-07:00

Here is a rake task that will list all the ActiveRecord models in a Rails application along with the number of database records in each and the last time that database table was updated. Note the call to eager_load which ensure that all models are loaded.

This is a simple but very useful way to assess the contents of your database.

  desc "List model in application"
  task :list_models => :environment  do
    Rails.application.eager_load!

    ActiveRecord::Base.descendants.each do |model|
      name = model.name
      count = model.count
      last_date = count == 0 ? 'no data' : model.order('updated_at desc').first.updated_at
      printf "%-20s   %9d   %s\n", name, count, last_date
    end
  end

Gotchas when working with HTML5 Audio

2013-04-17T10:02:00.002-07:00

I've run into a few issues while building a demo web app that plays audio files, so I wanted to post my experiences...

1: Firefox does NOT play MP3 files (as of 2013-04-17)

This is because the MPEG format is proprietary and the Firefox/Mozilla folks insist on only open source formats. You need to use .wav, .ogg or .webm. See https://developer.mozilla.org/en-US/docs/HTML/Supported_media_formats for the allowed options.

Unfortunately they do not seem to produce a useful error message if you try and play an MP3 file.

Google Chrome plays anything - so your application may appear to work fine but may fail on Firefox.

2: Firefox is VERY strict in regard to Cross-Origin Resource Sharing (CORS)

Firefox will refuse to play an audio file on a host other than the one that served the original web page. So you can't store an audio file on, say, Amazon S3, and play it on your web site. The same goes for Ajax requests to another site. The Firefox console will, or may, display an error but this is not necessarily clear.

Google Chrome doesn't seem to care about this restriction.

See this page for more details https://developer.mozilla.org/en-US/docs/HTTP/Access_control_CORS

To work around this you can specify a header for the audio file which means that as the owner of the audio file you allow it to be referenced from any other site.
"Access-Control-Allow-Origin" = "*"

To work around the problem with Ajax requests you can use JSONP instead of JSON as I have described.

The special header ought to work with AWS S3... except....

3: AWS S3 does NOT allow the Access-Control-Allow-Origin header

You can set metadata headers with S3 files but not this one. The Amazon support forums are full of requests/complaints about this but it does not seem to have been resolved.

There does seem to be a way round this on Amazon Cloudfront, which pretty much exists to server assets for other sites, but from what I've seen this is pretty ugly.

My workaround for this was to add a proxy method to my server. The client requests a file that resides on S3 from the proxy. It fetches it from S3 and echoes it to the client, which sees it as coming from the originating server. This introduces an unnecessary delay but works file. Here is a snippet of code in Ruby for a Sinatra server. It uses the aws-sdk gem and leaves out set up for that.

get '/proxy' do
s3obj = bucket.objects[params['file']]
response["Access-Control-Allow-Origin"] = "*"
content_type 'audio/wav'
s3obj.read
end

Apparently Google's equivalent of S3 does not have this issue - I've not tried it.

4: You cannot play audio files from localhost

This makes testing a pain. Your audio files must be hosted on a server other than localhost. I recommend Heroku as a way to get development servers up and running quickly and at no initial cost.

If you don't realize this, then there is no error message in the browser console - it simply doesn't work and you are left scratching your head until you figure it out. I hate stuff like that...

Web Text To Speech using Bing Translate - Demonstration Sinatra App

2013-04-16T15:01:00.000-07:00

Given the variety of sophisticated web services available today, it is surprising that a good Text To Speech API (TTS) is not readily available.

Google has one as part of its Translation API but it is not publicized or actively supported. The Bing/Microsoft Translate API also has a TTS feature and this is supported. In my experience this works very well and allows you to specify the language of the text, which changes the voice and pronunciation that is used.

Accessing this API is easy enough using a wrapper library, such as https://github.com/CodeBlock/bing_translator-gem (disclaimer: I added the speak() function to this ruby gem) but it returns binary data that represents an MP3 file. The HTML5 audio tag allows you play audio files, but not, apparently, to play the data itself.

The result is that the TTS output must be first written to a file and then played via the audio tag.

To demonstrate how these different pieces fit together, I have written a TTS demo app that consists of a Javascript in a web page that sends the query text to a Sinatra app, using ajax. The server in turn sends this to Bing and get back the audio. The server then writes this to a file on Amazon S3 and returns the URL for this back to the web page where the audio is played.

The Live Demo for this is at http://bing-translate-tts-demo.craic.com/ and the code required to implement the whole thing is freely available at https://github.com/craic/bing_translate_text_to_speech

The demo has several moving parts and setting it up for yourself requires experience with Sinatra, S3, etc.

Zillow Neighborhood Boundaries in GeoJSON and KML for Seattle

2013-04-12T13:55:00.000-07:00

The good people at Zillow have invested a lot of time defining the boundaries of around 7,000 neighborhoods in cities around the USA.

These are in the form of ESRI ARC Shapefiles that can be used as overlays on maps.

You can find these at Zillow Neighborhood Boundaries.

Zillow have kindly made these available to the rest of us under a Creative Commons license. This allows you to share and modify the data but you need to attribute Zillow and must distribute any derivative forms of the data under the same or similar licenses. Attribution should take the form of including their logo with a link back to their site.

ESRI shapefiles can be read by many types of mapping software, but one big exception to this is Google Maps. Overlays in Google Maps are loaded from KML format files. In addition GeoJSON is a JSON-based format that is gaining in popularity.

I needed neighborhood boundaries for the City of Seattle in KML format so I can use them in Google Maps. I couldn't find a convenient way to convert from Shapefile directly to KML (although these do exist in some GIS packages) and the solution I came up with involved converting first from Shapefile to GeoJSON and them from GeoJSON to KML. I'll make that software available elsewhere.

My Github Project zillow_seattle_neighborhoods contains the GeoJSON and KML files for the 78 Seattle neighborhoods defined in Zillow's dataset.

The same approach could be applied to derive GeoJSON and KML files for the entire Zillow US city dataset. I'll see if I can do that in the future... for now it just covers Seattle.

Using the Files

You can use either format of file in many GIS packages and mapping programs. Here are a few ideas for people who are new to this area of coding.

You can load KML files into your own maps using Google Maps directly. Look under 'My Places' -> 'Create Map' -> 'Import'. See Google Maps documentation for details.

You can load the KML files into Google Earth for a different experience. The command 'open .kml' on a Mac with Google Earth installed should be enough.

To use these overlays in custom Google Maps you want to use the Google Maps JavaScript API.

I'll add an example HTML page that shows how to do this in due course...

Two important things to note about testing KML overlays in Google Maps in your own HTML pages

1: Your KML files MUST be hosted on a remote server. They will be ignored if you give try and host them from a server on localhost !

2: KML files are CACHED in web pages, just like images. So if you are changing the files, such as changing the colors, these will not be visible unless you work around the caching issue. The easiest way to do this is to add a question mark followed by a random string at the end of the url for the KML file, and change this on every page reload.

For example: http://yourserver.net/yourfile.kml?1234567

Fixing incorrect timezones in Ruby

2013-04-05T15:21:00.000-07:00

Timezones can be tricky to manage and sometimes I just get them wrong - like now...

I had been parsing strings representing local times in Seattle and then storing these without an explicit timezone on a machine which uses UTC by default - so they got stored as that time in UTC (not converted to UTC - the exact time string in UTC.

For example: Fri, 05 Apr 2013 14:24:39 (the local time in Seattle) was stored as Fri, 05 Apr 2013 14:24:39 UTC +00:00 which is 7 hours earlier than it should be.

The easiest way to fix this, that I found is to convert back to a string, strip of any record of a timezone and then parse it back into a time, having temporarily set the TZ environment variable to the correct local timezone.

NOTE that this works on UNIX/Mac OS X systems - not sure about Windows...

puts ENV['TZ'] => nil
puts t   => 2013-04-05 14:24:39 UTC
s = t.to_s
puts s   => "2013-04-05 14:24:39 UTC"
s.sub!(/\s+UTC/, '')
puts s   => "2013-04-05 14:24:39"
ENV['TZ'] = 'US/Pacific'
t1 = Time.parse(s)
ENV['TZ'] = nil
puts t1 => 2013-04-05 14:24:39 -0700

Remember to reset the TZ environment variable back to nil

Be very careful about using gmtime and localtime as these methods operate in place. So ALWAYS make a copy of your time before applying those conversions

t => 2013-04-05 15:25:41 -0700
t1 = t.dup

t.gmtime => 2013-04-05 22:25:41 UTC
t => 2013-04-05 22:25:41 UTC
t1 => 2013-04-05 15:25:41 -0700

Using Latitude and Longitude with Google and Bing Maps

2013-04-03T14:13:00.003-07:00

When you work with maps you sometimes want to extract a Latitude/Longitude pair from the map, or you want to enter a pair and see the corresponding map.

Google Maps and Bing Maps do things slightly differently. In general I go with Google but there is not a lot in it. Here are how you work with Latitude/Longitude in both of them:

1: Enter an Address and get the Latitude/Longitude pair

Bing - Enter the address - the Lat/Lon pair will be shown below the Address in the left hand panel

Google - Enter the address - then Right Click the marker and select What's here? - the Lat/Lon pair will appear in the address entry box

2: Browse to a Location and get the Latitude/Longitude pair

Bing - Browse to the location - then Right Click and the Lat/Lon pair will be shown in the popup

Google - Browse to the location - then Right Click the marker and select 'What's Here - the Lat/Lon pair will appear in the address entry box

3: Enter a Latitude/Longitude pair and view the Location

Bing and Google - Enter the Lat/Lon pair, separated by a comma, into the address box (e.g. 47.619905,-122.320844) - Google makes a better choice of zoom level in my opinion.

SODA - Socrata Open Data API

2013-03-21T08:39:00.001-07:00

Socrata is a company that helps organizations make their data available to outside users. They work with people like the World Bank and a number of city governments, including the City of Seattle.

You can access these public data sources via SODA - the Socrata Open Data API which provides a SQL-like query language (SoQL) that you submit via HTTP request to an API endpoint for that resource. The default response is JSON but you can get CSV, XML and RDF.

They provide quite a bit of documentation but it is a bit short on examples and I had to poke around a bit to get the right syntax for the data sources that I am interested in.

A very useful resource is the 'live' query console they provide for a test dataset, which lets you try various queries.

I access their resources from Ruby. I generate the URL, use open-uri to fetch the data and then parse the JSON response. Here is an example fo the process:

The data source I want to query list the Seattle Fire Department 911 calls. That page lets you create custom views and download the data. If you click on 'Export' you can see the column names used in the database as well as the url for the API endpoint.

I want to get all records in the past hour, so I need a query along the lines of 'where datetime > my_timestamp'. Specifically I need to build a URL similar to this:

http://data.seattle.gov/resource/kzjm-xkqj.json?$where datetime > my_time_stamp

This is not a real query just yet but note a couple of things...

The API endpoint is http://data.seattle.gov/resource/kzjm-xkqj.json

This version ends in '.json' and so it will return the response in JSON. If you want CSV then replace '.json' with '.csv' - same goes for XML and RDF.

Also note the dollar sign before the 'where' - it is easy to overlook - that signifies the following word is a SoQL clause and not the name of a column.

A big problem I ran into was figuring out the format for timestamps. The web page showing the table formats them as '03/21/2013 07:35:00 AM -0700' whereas the JSON response uses "datetime" : 1363872960. But neither of these works in a query... and the documentation on data types does not describe what you should use. Trial and error with that console page led me to the correct format of '2013-03-21 07:00:00' which is similar to the ISO 8601 format.

Also note that the time needs to be quoted... again, apparently not in the documentation

So my query should now look something like this:

http://data.seattle.gov/resource/kzjm-xkqj.json?$where datetime > '2013-03-21 07:00:00'

But that URL needs to be escaped before submission. That was the other issue - URI.escape() handles the spaces, quotes, etc but for some reason it does not escape the dollar sign. So I had to do that for myself.

Here is a chunk of Ruby code that gets all the 911 calls in the past hour:

require 'open-uri'
require 'time'
require 'json'

endpoint = 'http://data.seattle.gov/resource/kzjm-xkqj.json'
timestamp = (Time.now - (60 * 60)).strftime("%F %H:%M:%S")

query = "$where=datetime > '#{timestamp}'"

url = "#{endpoint}?#{query}"
url = URI.escape(url)
url.gsub!(/\$/, '%24')

puts "query: #{query}
puts "url: #{url}

json = open(url).read
puts json

result = JSON.parse(json)
puts result.length

That will show you the query, the escaped URL, the JSON returned and the number of records in the parsed JSON - enough to get you started. Different data sources will use different column names and perhaps different formats for date/time (always be aware of time zone issues).

Having a common API to many government/public data sources is great but the Socrata-backed sources that I've looked a so far don't seem to be quite as consistent as one would hope. This is an issue for the different data providers - it's not a Socrata issue.

I would hope that one could query the endpoint in some way and get a listing of all the column names and data types. There is one for SODA v1.0 but this now deprecated... so perhaps there is a newer version but it has not been documented.

Part of the answer lies in two custom HTTP response headers (X-SODA2-Fields and X-SODA2-Types) that are returned with your data. Here they are for the data source used here:

X-SODA2-Fields: [":updated_at","address","longitude",":id","latitude","incident_number","datetime","type",":created_at","report_location"]
X-SODA2-Types: ["meta_data","text","number","meta_data","number","text","date","text","meta_data","location"]
X-SODA2-Legacy-Types: true

You can see the headers if you use 'curl -i "your_url"' on the command line with your escaped url.

HTML5 and web API code examples

2013-03-08T14:31:00.000-08:00

Whenever I am learning a new feature of HTML5, JavaScript/Jquery or a new API, such as Google Maps, I always look for example code that I can learn from.

In many cases the examples are great, but in others they can be too clever and too heavily styled, such that it can be hard to understand the core of the feature that they are demonstrating.

So in writing my own example code I try and strip things down to the bare minimum - very little styling and code that tries to do one, and only one, thing.

I've been collecting examples that I think have some substance and that can help others learn about a feature with the minimum of confusion.

Take a look at http://html5-examples.craic.com

This site has working examples of a number of HTML5-related features, including Data Attributes, Geolocation, Web Audio and Speech Recognition.

Look at the Source for each of these pages to see annotated JavaScript, etc. which illustrates the target feature of each page.

All the code is distributed freely under the terms of the MIT license and you are encouraged to build your own applications using it.

Note that some of the examples involve new web technologies, so they may not work in some browsers and what does work now may not work in the future as the APIs mature.

General approach to building Rails applications with Devise

2013-02-26T13:23:00.000-08:00

I have built a number of complex Rails applications that include user authentication. I want to describe my general approach to getting these up and running.

In most of them I chose the Devise gem to handle authentication and added a number of custom fields to the User model (first name, last name, etc). The gem documentation and this Railscast can help you get started with all that.

One approach is to build your core application without authentication and then add it in at a later stage but I don't like that.

Authentication can quickly get very complicated once you start adding user confirmation via email, an admin role and other custom fields. I prefer to get all that machinery up and running in the context of a very simple application and only then add my real application code into it.

Similarly, when I'm starting with a new application I don't usually hit on the right data architecture right away. That leads to me adding new columns to my database tables and models over time and it always involves quite a lot of 'rake db:rollback / rake db:migrate' operations. I prefer to go through this phase without the extra baggage of authentication, and often, without worrying a lot about CSS etc. I just need to try several alternate views of the problem before I settle on a final design.

So may approach is to build two applications side by side. The first is a minimal application with, say, a single table (usually called 'posts') to which I add Devise and configure all the options that I want.

This is a very useful piece of code for other applications, so I always create a good README and store a copy of the code for future use.

Separately I work through my core application and, once that is working, I condense all the migrations that add/remove/rename columns into a few 'clean', simple migrations.

Finally I add the core application code to the minimal authenticated application. That way it is easier for me to update the controllers, etc. to use authentication.

The bottom line is that you want to minimize the number of moving parts in the early stages of a new application design. This is a simple approach that works very well for me.

Street Intersections in Google Maps

2013-02-14T10:48:00.002-08:00

I need to display Google Maps with markers placed at street intersections, starting with an address that looks like '13 Av E / E John St, Seattle, WA'.

Google Maps doesn't know how to parse this and gives me a location on John Street but it's not correct.

Various sources on the web tell you use 'at' as the conjugation between the two streets - so '13 Av E at E John St, Seattle, WA'

This works fine in most cases but in some, like this one, it fails - this gives the intersection of 13th Ave East and E Prospect (7 blocks away).

What seems to work much better is to use 'and' ... '13 Av E and E John St, Seattle, WA' is spot on.

Jquery snippet to disable / enable links on a page

2013-02-08T15:54:00.000-08:00

Here are two approaches to hiding all the anchor (a) tags on a page while retaining the html that was contained within the tags. They both use Jquery.

Approach #1 - Remove all the 'a' tags while retaining their content - no way to retrieve the links, etc.

This lets you effectively disable all the links without losing any text. Using the .html() call instead of simply .text() means that any images or other formatting is preserved.

    $("a").each(function() {
      var t = $(this).html();
      $(this).replaceWith(t);
    });

Approach #2 - Reversibly Hide and Show the tags

Replace the 'a' tags with 'span' tags and store the original tag pair in a custom data attribute

Custom data attributes all have the prefix 'data-' followed by a unique name. You can store arbitrary data in these. With this approach I am storing the original 'a' tag pair and its contents.

In the example, my custom data attribute is called 'data-craic' and note that the 'span' tags are given a unique class (I'm using 'hidden-link') that allows you to identify them

To hide/disable the links:

  $("a").each(function() {
    // Get the html within the 'a' tag pair
    var t = $(this).html();

    // Get the entire tag pair as text by wrapping in an in-memory 'p' tag and fetching its html
    var original_tag = $(this).clone().wrap('<p>').parent().html();

    // Replace the 'a' tag with a 'span' - put the original tag in the data attribute
    $(this).replaceWith("<span class='hidden-link' data-craic='" + original_tag + "'>" + t + "</span>"); 
  });

To show/enable the links:

  $(".hidden-link").each(function() {
    // Retrieve the original tag from the data attribute
    var original_tag = this.dataset.craic;

    // Replace the 'span' with it
    $(this).replaceWith(original_tag);
  });

At least in my use cases I have not had to encode the original 'a' tag pairs but base64 encoding might be a good idea to avoid any possible unintended consequences.

Custom data attributes are extremely useful and were the perfect to solution to this problem

You can find a self contained web page with these scripts at https://gist.github.com/craic/4987192

JSON versus JSONP Tutorial

2013-01-28T15:45:00.001-08:00

I recently found myself a bit confused on how to convert a JQuery $.getJSON() call to use JSONP (JSON with Padding).

JSONP is a way to get around the Same Origin Policy restriction when a script in one domain wants to fetch JSON data from a server in a different domain.

The approach is simple enough but I had trouble finding a clear explanation. So I wrote up a simple example in both JSON and JSONP with server and client code.

The live demo is at http://json-jsonp-tutorial.craic.com and all the code is on Github at https://github.com/craic/json_jsonp_tutorial

I hope that you find it useful...

Tool to help Count Features in Images

2012-12-05T13:12:00.000-08:00

Craic has released a new tool to help count features in images. This can be used in a range of applications such as counting buildings in satellite images, bacterial colonies and cell types in histology images.

It takes the form of a Ruby Sinatra application that fetches remote images and Javascript in the client web page that implements the marking and counting functions.

The user can choose the shape and color of the marker. The current total is updated as features are clicked and the coordinates of each point are stored internally.

The image with the user's marks can then be saved to a PNG format file and the list of coordinate pairs can be displayed in a separate window.

The graphics and interaction are implemented with Canvas and Javascript and make use of the wonderful Canvas2Image code from Jacob Seidelin which allows the current state of a canvas element to be saved to an image file.

The live application is hosted at Heroku and can be accessed at http://counter.craic.com

The code is distributed freely under the terms of the MIT license and is archived at Github

Counting Temporary Shelters in Satellite Images using OpenCV

2012-12-04T08:29:00.000-08:00

A number of areas of data analysis are dominated by sophisticated algorithms, intensive computation and, in some cases, limited access to raw data. Examples include the analysis of satellite imagery, feature extraction from video, protein structure analysis and language processing.

I am interested in how simple, approximate methods can be used to extend the application of these technologies. While simple approaches will clearly not match the accuracy and resolution of complex methods, they can, by nature of their simplicity, be implemented and deployed more easily and hence more widely.

I have just posted the code for one of these projects to the Craic Github site. The application involves image processing of satellite images downloaded from Google Maps in order to estimate the number of temporary dwellings or shacks in the slums of Port-au-Prince in Haiti.

The satellite images of Port-au-Prince show large areas covered with small white, blue and rust colored squares. These are the slums of the city in areas such as St.Martin, Cite Soleil and others.

The blue features are very distinctive and most likely represent the ubiquitous blue plastic tarpaulins the you can find at any hardware store.

Using the Python interface to the wonderful OpenCV image processing library, I wrote a simple application that identifies blue features below a cutoff size and then computes their area. By fetching adjacent squares that tile across the city, and calculating the area of blue features, I can come up with an matrix showing the relative density of the slums.

The approach is undoubtedly simplistic (the code is only about 35 lines of Python) but it demonstrates how simple approaches can be successfully applied to complex and sophisticated types of data. You do not always need access to expensive commercial software and proprietary datasets in order to work in these fields.

The code is made freely available at https://github.com/craic/count_shelters and you can read my write up of the work so far, with example images HERE.

Tutorial on JS/JQuery Bookmarklets

2012-11-01T13:09:00.001-07:00

I recently needed to implement a JavaScript bookmarklet for a new project. There are many web resources available but even so I had a bit of a learning curve to contend with.

In the hope of lessening that burden on future users, I've written up a tutorial on two types of bookmarklets with simple, annotated examples and posted all the code on our Github page.

The simplest form of bookmarklet is a chunk of JS code contained in the Bookmarklet URL that invokes some action. This is a direct approach - you select some text, you click the bookmarklet and you get a result. If you want to repeat the process with some other text on the page then you have to click the bookmarklet link again.

However, for many applications you need to invoke an action multiple times on a single page - looking up words in a dictionary would be a good example. Having to click on a Bookmarklet link every time is not a good solution. Instead you want to modify the behaviour of the page by injecting a custom JS script. A bookmarklet can be used to initiate this. So you end up with three scripts, two of which are slightly modified, general purpose scripts.

The bulk of the code for this second approach comes from how-to-create-a-jquery-bookmarklet, written by Brett Barros of latentmotion.com, with some code input from Paul Irish.

To fully benefit from the tutorial, you should download the project, run 'bundle install' and 'rackup -p 4567' to set up a local Sinatra server, then point your browser to http://localhost:4567

Rails 3.2 Asset Pipeline - 'require_tree' is evil !

2012-10-29T14:09:00.000-07:00

I have been battling the Asset Pipeline in Rails 3 for a couple of days - and finally I've got it working

An important part of the Asset Pipeline is the use of Manifest files in app/assets/javascripts and stylesheets which determine the specific files to include and, in some cases, the order in which they are processed.

The default javascripts Manifest file has these lines:

//= require jquery
//= require jquery_ujs
//= require_self
//= require_tree .

That reads as:
- process jquery, then jquery_ujs
- process anything in this file (self)
- process all the files in this directory tree (require_tree)

I don't have many JS files in my app - three custom JS files written by myself and a couple of third party libraries.

I try and follow defaults as far as possible but require_tree has two BIG problems.

First is that it processes ALL the files in the directory tree.
Second is that the ORDER is UNDEFINED.

It is common for JS libraries to distribute their code along with examples and minimized versions of the code. In the past I would just drop these distributions in the javascripts directory and then specify which of the files should be included in my app.

So one problem with require_tree is that it will process both a 'native' and minimized version of the same library is both are present. That results in the same functions being defined multiple times - and that seems to be a real problem if a library includes it's own copy of jQuery.

Not specifying the order of processing is the other BIG issue. For libraries with a 'core' JS file along with others will typically require that the core library is processed first.

The solution is to avoid require_tree at all costs. Put only the JS files that you absolutely need in app/assets/javascripts and specify them EXPLICITLY in the order they should be processed.

In my example, my application.js file becomes:

//= require jquery
//= require jquery_ujs
//= require ./tabs_timeline_simple
//= require ./tandem_select
//= require ./raphael-min
//= require ./facebox

I strongly recommend that you start out with everything clearly defined like this. Once you have it all working then you can remove unused files and simplify application.js - if not, you risk going through the same debugging misery that I just went through.

Upgrading from Rails 3.0 to 3.1 - Issues with Javascripts

2012-10-25T11:35:00.001-07:00

I have been upgrading a Rails app from 3.0.10 to 3.1.5.

The BIG difference with 3.1 is the use of the Assets pipeline which requires that you move images, stylesheets and javascripts from public/* to app/assets/*. In the case of Stylesheets and Javascripts you also put these under the control of a Sprockets Manifest file: application.js or application.css.scss

That Manifest file caused me problems with my javascripts...

I followed the advice in the Railscasts Episode 282 (which I recommend you look at) and put these lines into my application.js

//= require jquery
//= require jquery_ujs
//= require_self
//= require_tree .

But when I ran my app none of my javascript functions were working. Looking at the source for one of the pages I saw this link

  <script src="/assets/application.js" type="text/javascript"></script>

Clicking on brought up a message, instead of a JS file, saying that Sprockets was unable to find the file 'jquery'

Jquery is supposed to 'just be there' if you have the jquery-rails gem installed.

I think there were a two, maybe three, issues.

1: Sprockets requires that there are NO BLANK LINES in the Manifest file BEFORE the end of the lines that should be processed. In Rails 3.2 the default file says that explicitly but if you are creating your own that is an easy thing to overlook.

2: Look at the 4 lines shown above. The first two are 'require', followed by a SPACE and then the name of the javascript file.

The Third is 'require_self' - underscore, no space - that means process any JS code in this file.
The Fourth is 'require_tree' - underscore, no space - and that means process ALL JS files in this directory.

If you type these lines in and happen to use a space instead of the underscore then Sprockets will look for JS file called 'self' and 'tree' - and won't find them. So be very careful tying those in.

3: The main problem was the require_tree line

I have a number of JS files that are included as needed in certain views. At the top of each view page I specify those with a content_for block like this:

<% content_for :custom_javascript_includes do %>
  <%= javascript_include_tag "raphael.js" %>
  <%= javascript_include_tag "tabs_timeline_simple.js" %>
<% end %>

and in my application.html file I have

  <%= javascript_include_tag "application" %>
  <%= yield :custom_javascript_includes %>

That has worked just fine in the past and I kept all the custom JS files in the javascripts directory. But require_tree will load ALL of them on EVERY page. I assume that there are some variable/function names that are duplicated and/or conflicting and so with all that loaded, nothing worked.

The solution is to get rid of the require_tree line completely.

If I had JS files that should be included in each page then I would 'require' each of them explicitly. This is the same approach that Ryan Bates suggests for CSS files. It just gives you more flexibility.

Uploading contents of a PostgreSQL database to Heroku without db:push

2012-10-24T17:00:00.000-07:00

I just created a new Rails app on my local machine and want to deploy it to Heroku. I've done this before and the uploading of the code typically works great - and it did today.

Then I wanted to upload the contents of a small PostgreSQL database to Heroku - and that's where I ran into issues.

Heroku has recently switched from a Heroku gem to Heroku Toolbelt which I installed onto my Mac OS X machine.

I should be able to run 'heroku db:push' from my Rails app and the data would get moved over using the taps gem. But it failed with this error:

$ heroku db:push
! Taps Load Error: cannot load such file -- taps/operation
! You may need to install or update the taps gem to use db commands.
! On most systems this will be:
!
! sudo gem install taps

I have taps installed, so what is going on? Well, I'm not sure but I run my rubies under rbenv which allows you to manage multiple versions - and Heroku's client doesn't seem able to figure out where the gems are located with rbenv.

I dug around in /usr/local/heroku and hacked /usr/local/heroku/lib/heroku/command/db.rb to tell me what it thought the load path for ruby gems was, using this line:

STDERR.puts "DEBUG: load path  #{$:}"

The paths given in that were all in /usr/local/heroku/vendor/gems and my gem paths were nowhere to be found. I tried installing the taps, sqlite3 and sequel gems to that directory and I was able to get a bit further but I still couldn't get all the way - so I gave up on that path.

And then I found mention of a page on Heroku's site that deals with the pgbackups addon. Part of that describes how to import from a PostgreSQL backup directly, without having to use the heroku clients or taps.

I followed that and it worked great - here are the steps involved. Your starting data must be in PostgreSQL and you need to have an account on Amazon S3 (and know how to use it)

1: On your machine create a backup of your PostgreSQL database - 'mydb' is your database name and 'myuser' is your database user.

$ pg_dump -Fc --no-acl --no-owner -h localhost -U myuser mydatabase > mydb.dump

2: Transfer that to an Amazon S3 bucket and either make it world readable or set a temporary access url.

3: Check that you can get the file back out of S3 using curl

$ curl 'https://s3.amazonaws.com/mybucket/mydb.dump' > tmp
$ ls -l tmp

4: Enable the pgbackups addon in Heroku

$ heroku addons:add pgbackups

5: Get the name of the Heroku database that you will restore to, by capturing a backup from the empty heroku db:

$ heroku pgbackups:capture

HEROKU_POSTGRESQL_CRIMSON_URL (DATABASE_URL)  ----backup--->  b001

Capturing... done
Storing... done

6: Restore to that database from the S3 URL

$ heroku pgbackups:restore HEROKU_POSTGRESQL_CRIMSON_URL 'https://s3.amazonaws.com/mybucket/mydb.dump'

7: Test it out - my app was already up so I just had to browse to the appropriate page and there were all my records.

8: Remove your S3 file

DONE !

Rails 3 - New Model is not recognized

2012-09-27T16:11:00.001-07:00

I created a new Model in a Rails 3 application, along with a database table - but because I was not creating a controller or views I did not use a rails generator and just created the file by hand.

The contents followed exactly the format of other Model files and I was careful to use a Plural name for the database table and a Singular name for the model class.

But when I ran the application, either in the console or as a server, I kept getting an error reporting the Model name as an 'uninitialized constant'.

I double and triple checked everything but no luck getting it to work. Poking around the web I found mention of the Lazy Loading feature in Rails whereby classes are not loaded until they are needed.

Along with that was mention that using the command 'Rails.application.eager_load!' in the console might solve the problem. indeed it did - but only in the Rails console...

The problem turned out to be that the model FILE name was PLURAL. The Model class itself was singular - which is correct - but the lazy loading mechanism was using this (I assume) to define the filename in which to find the Model. Because my was plural it could not find the file and because it did not find a model it assumed that my Model name was a constant and then found that to be uninitialized.

It's a confusing error message and I wonder if there might be a way for Rails to figure out that I wanted a model. But I figure this is a side effect of the Rails machinery for inferring model, controller, etc. names.

Ah well... live and learn...

Microsoft AutoUpdate doesn't...

2012-09-20T13:46:00.000-07:00

I got this message when I started up MS Word on my Mac this morning:

In other words, Microsoft AutoUpdate does not auto update... :^{

Compare that to the Google Chrome web browser which auto updates itself without requiring any action from myself and without requiring a browser restart.

Preventing Excel from wrongly interpreting Strings in CSV files as Exponential numbers

2012-08-27T13:37:00.002-07:00

MS Excel and Apple Numbers both attempt to guess the format of the contents of each cell when they open up a CSV file. Most of the time this is fine but in some cases it causes a big problem.

If you have a string with numbers and the letter E then both programs will treat this as a number in exponential notation.

For example, the string 229E10 will be converted to 2.29E+12

What makes it worse is that you cannot change it back to the original text using the cell format options.

I run into this problem with strings that represent microtiter plate and well assignments. For example, 229E10 refers to plate 229 row E column 10.

I just want the spreadsheet program to take what I give it as text. Putting the string in single or double quotes can work but then the value remains in quotes in the spreadsheet.

The best way to do this seems to be to use double quotes around the string AND prefix that with an equals sign - like this:

="229E10"

For reasons that I do not understand, Excel will strip the extra formatting off the string and treat the contents as plain text. This also works for arbitrary strings, such as ="ABC"

Installing OpenCV Ruby gem on mac OS X

2012-08-14T15:50:00.001-07:00

In my previous post I showed how to Install the OpenCV computer vision library on Mac OS X Lion.

I want to use the Ruby interface to the library and of the various project forks I went with https://github.com/ruby-opencv/ruby-opencv

The github page has several alternate ways to install the interface, but installing it as a gem is the most mainstream. Even so, you need to do a bit more work than usual. First create an install directory for the 'raw' gem and clone it from github:

$ mkdir opencv_install
$ cd opencv_install/
$ git clone git://github.com/ruby-opencv/ruby-opencv.git
$ cd ruby-opencv/
$ git checkout master
$ bundle install
$ rake gem
$ gem install pkg/opencv-*.gem -- --with-opencv-dir=/usr/local/Cellar/opencv/2.4.2

Your opencv directory may not be identical to mine but this is where homebrew should place it. For me, that was all I needed to do before copying the examples from the github page and trying things out ! Here is my version of the basic image display program:

#!/usr/bin/env ruby
require "opencv"
abort "Usage: #{$0} " unless ARGV.length == 1
image = OpenCV::IplImage.load(ARGV[0])
window = OpenCV::GUI::Window.new("preview")
window.show(image)
OpenCV::GUI::wait_key

When you run the script with an image file, you can quit by typing 'q' in the resulting image window. Now for the fun stuff. Here is my version of the github page face detection script. Face detection uses 'haarcascade' classifier files which live in your OpenCV library directory. With my Homebrew installation these are found in /usr/local/Cellar/opencv/2.4.2/share/OpenCV/haarcascades

Get yourself an image file with a few faces (looking directly at the camera) to test it on.

#!/usr/bin/env ruby
require "opencv"
abort "Usage: #{$0} source_image_file  dest_image_file" unless ARGV.length == 2
classifiers_dir = "/usr/local/Cellar/opencv/2.4.2/share/OpenCV/haarcascades"
data = File.join(classifiers_dir, "haarcascade_frontalface_alt.xml")
detector = OpenCV::CvHaarClassifierCascade::load(data)
image = OpenCV::IplImage.load(ARGV[0])
detector.detect_objects(image).each do |region|
  color = OpenCV::CvColor::Red
  image.rectangle! region.top_left, region.bottom_right, :color => color
end
image.save_image(ARGV[1])

Make the file executable, run the script and then view the output file

$ chmod a+x findface.rb
$ ./findface.rb smiths.jpg output.jpg
$ open output.jpg

And you should something along the lines of this:

Pretty amazing, if you ask me - Not perfect but three out of four ain't bad !
Now on to some real experiments with OpenCV - I can't wait...

Installing the OpenCV library on Mac OS X Lion using Homebrew

2012-08-14T14:57:00.000-07:00

OpenCV (Open Source Computer Vision) is a remarkable library of functions for real time computer vision.

I'm interested in using it on to recognize buildings in satellite/aerial photographs - I'll explain more if I get the idea to work.

The first step is to get OpenCV up and running on my Mac (OS X Lion 10.7.4). The library has a lot of dependencies that need to be installed but thankfully there is a Homebrew recipe for doing all of the heavy lifting... almost... I ran into a few problems but here is what I needed to do.

In principle all you need is:

$ brew update
$ brew install opencv

But I got two errors:
1: brew wanted me to update my version of Xcode from 4.2 to 4.3
On Lion you update Xcode via the App store and it turns out that 4.4 is available. So I installed that and reran 'brew install opencv'
2: brew then complained that the Python package 'numby' (Numerical Python) was not installed. Brew does not install Python dependencies but suggested that I do this with:

$ easy_install numby

I did that but I still got the error.

I've got two versions of python installed (via homebrew)

$ python --version
Python 2.7.3
$ python3 --version
Python 3.2.3

For our purposes you only want to deal with Python 2 - version 3 should work but the homebrew package seems to want version 2

Given the problem with easy_install, I went with a manual installation
1: Download the tar file for the latest stable version of numby
http://sourceforge.net/projects/numpy/files/NumPy/1.6.2/
2: Unpack the file into a temporary directory and cd into it
3: Run the python setup script

$ python setup.py install

Sit back and relax as this takes a while. Everything went smoothly for me and hopefully for you too.

All being well you can now run

$ brew install opencv

Again this takes a while - about 9 minutes on my box. You will get a message at the end of the output telling you to add the following to your .bash_profile

export PYTHONPATH="/usr/local/lib/python2.7/site-packages:$PYTHONPATH"

OpenCV has an excellent interface to Python, but I work primarily in Ruby so the next step for me is to get an OpenCV ruby gem set up and start to do some image processing. The next post on this blog will cover that.