Common Sense Software

What to do when ActiveRecord thinks an Oracle key is a decimal

2010-01-27T09:10:00.005-05:00

I recently created a model for an existing database table using the legacy_data gem and was confused when my primary key showed up in scientific notation. It turned out the issue was due to sloppiness in the table definition and could be easily fixed once I understood what ActiveRecord was doing.

I created a Person model connected to the people table


class Person < ActiveRecord::Base
end

but when I went into script/console the primary key showed up as a BigDecimal when I expected an integer.


$ script/console 
Loading development environment (Rails 2.3.4)
>> Person.first.id
=> #

This wasn't what I wanted and would cause problems in my app when it tried to build a url with that id like http://localhost:3000/people/10024844425.0. The rails routing engine would see the .0, treat it as a format (like .xml or .json) and get confused. Let's look at why this is happening.


>> Person.columns_hash['id']
=> #
>> Person.columns_hash['id'].type
=> :decimal
>> Person.columns_hash['id'].sql_type
=> "NUMBER"

We see that ActiveRecord is treating this column as a :decimal because it's sql_type is NUMBER. It turns out this is correct because an Oracle number is a decimal unless you specify it to have 0 digits after the decimal point (scale of 0). Here's the documentation from Oracle (the last sentence is my bold)


NUMBER Datatype

The NUMBER datatype stores fixed and floating-point numbers. Numbers of virtually any magnitude can be stored and are guaranteed portable among different systems operating Oracle, up to 38 digits of precision.

The following numbers can be stored in a NUMBER column:

Positive numbers in the range 1 x 10-130 to 9.99...9 x 10125 with up to 38 significant digits

Negative numbers from -1 x 10-130 to 9.99...99 x 10125 with up to 38 significant digits

Zero

Positive and negative infinity (generated only by importing from an Oracle Version 5 database)

For numeric columns, you can specify the column as:

column_name NUMBER 

Optionally, you can also specify a precision (total number of digits) and scale (number of digits to the right of the decimal point):

column_name NUMBER (precision, scale) 

If a precision is not specified, the column stores values as given. If no scale is specified, the scale is zero.

Oracle guarantees portability of numbers with a precision equal to or less than 38 digits. You can specify a scale and no precision:

column_name NUMBER (*, scale) 

In this case, the precision is 38, and the specified scale is maintained.

When you specify numeric fields, it is a good idea to specify the precision and scale. This provides extra integrity checking on input.

Let's look in my database and sure enough the ID is a number


$ sqlplus myusername/mypassword@localhost:1521/mydatabase.world

SQL*Plus: Release 10.2.0.4.0 - Production on Wed Jan 27 09:15:09 2010

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.


Connected to:
Oracle Database 10g Release 10.2.0.4.0 - Production

SQL> desc people;
 Name        Null?    Type
 ----------------------------------------- -------- ----------------------------
 ID          NUMBER
 NAME          VARCHAR2(10)

If you are allowed to change your database you can create a migration like


$ script/generate migration change_person_id_to_integer
STUBBING MckinseyLDAP
      exists  db/migrate
      create  db/migrate/20100127145747_change_person_id_to_integer.rb

now edit the migration


class ChangePersonIdToInteger < ActiveRecord::Migration
  def self.up
    change_column(:people, :id, :integer)  
  end
  def self.down
    change_column(:people, :id, :decimal)  
  end
end

In my case there were other applications using this table and I was not allowed to change it so I implemented a fix in Ruby to tell my model to treat this column as an integer even though it was defined as a decimal in the database.


#config/initializers/legacy_data_type_coercion.rb
module LegacyDataTypeCoercion
  def set_integer_columns *col_names
    col_names.each do |col_name|
      columns_hash[col_name.to_s].instance_eval do
        @type = :integer
      end
    end
  end
end
ActiveRecord::Base.extend(LegacyDataTypeCoercion)


#app/models/person.rb
class Person < ActiveRecord::Base
  set_integer_columns :id
end

We defined a method set_integer_columns that will force ActiveRecord to treat the columns we specify as integers. In our Person model we declare :id is an integer column. Let's test it out!


$ script/console 
Loading development environment (Rails 2.3.4)
>> Person.first.id
=> 10024844425
>> Person.columns_hash['id'].type
=> :integer

Just as expected id is now an integer and we can go ahead building the rest of our application.

This is not an issue with all Oracle tables as if the column was defined as NUMBER(10) (with a precision and implicit scale of 0) then ActiveRecord will interpret it as an integer automatically based on the parentheses in the data type - i.e NUMBER(10) ActiveRecord or Oracle Enhanced).

Freezing a gem that has native extensions

2010-01-13T16:30:00.000-05:00

I like to freeze all the gems I use as we run in a shared hosting environment and need to our apps isolated from each other. Deployments are also handled by an operational team that does not intimately understand our applications so keeping our deployments to a single capistrano command cap deploy:migrations has been a big win for us. Freezing most gems is pretty straightforward and has been built in since Rails 2.1. When dealing with a gem that requires native extensions to be built there's only one additional step to add to your Capfil.

Let's say we want to localize hpricot which does include native C extensions.

First tell Rails about your gem by adding a config.gem line to your environment.rb


Rails::Initializer.run do |config|
  ...
  config.gem 'hpricot'
  ...
end

Now we can ask rails about its configured gems


  $ rake gems
  (in /Users/alexrothenberg/ruby/my_project)
   - [I] hpricot 

   I = Installed
   F = Frozen
   R = Framework (loaded before rails starts)

The 'I' means its hpricot is installed on my system but not frozen in the application. If you see '[]' instead you need to run sudo gem install hpricot (add '--source http://gemcutter.org' if necessary). At this point you could write some code to use hpricot and your application will work. But if hpricot (or the version you're expecting) is not installed on your production server you'll be in trouble.

To freeze the gem into your vendor directory run rake gems:unpack (optionally you can add 'GEM=hpricot' if you just want to unpack one gem).


  $ rake gems:unpack 
  (in /Users/alexrothenberg/ruby/my_project)
  Unpacked gem: '/Users/alexrothenberg/ruby/my_project/vendor/gems/hpricot-0.8.2'

We can ask rails again...


  $ rake gems
  (in /Users/alexrothenberg/ruby/my_project)
  The following gems have native components that need to be built
    hpricot  

  You're running:
    ruby 1.8.6.287 at /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
    rubygems 1.3.2 at /Users/alexrothenberg/.gem/ruby/1.8, /Library/Ruby/Gems/1.8, /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8

  Run `rake gems:build` to build the unbuilt gems.

Oops our vendored gem is missing hasn't built the native extensions. Not to worry the message tells us what to do and we run rake gems:build


  $ rake gems:build
  (in /Users/alexrothenberg/ruby/my_project)
  Built gem: '/Users/alexrothenberg/ruby/mars-admin/vendor/gems/hpricot-0.8.2'
  alex-rothenbergs:mars-admin alexrothenberg$ rake gems
  (in /Users/alexrothenberg/ruby/my_project)
   - [F] hpricot 

  I = Installed
  F = Frozen
  R = Framework (loaded before rails starts)

We can ask rails again to see that the gem is now frozen and also look in our vendor folder


  $ rake gems
  (in /Users/alexrothenberg/ruby/my_project)
   - [F] hpricot 

  I = Installed
  F = Frozen
  R = Framework (loaded before rails starts)
  
  $ ls vendor/gems/hpricot-0.8.2/
  total 72
  -rw-r--r--   1 alexrothenberg  staff  4672 Jan 13 12:33 CHANGELOG
  -rw-r--r--   1 alexrothenberg  staff  1048 Jan 13 12:33 COPYING
  -rw-r--r--   1 alexrothenberg  staff  9216 Jan 13 12:33 README
  -rw-r--r--   1 alexrothenberg  staff  8242 Jan 13 12:33 Rakefile
  drwxr-xr-x   4 alexrothenberg  staff   136 Jan 13 12:33 ext/
  drwxr-xr-x   3 alexrothenberg  staff   102 Jan 13 12:33 extras/
  drwxr-xr-x   6 alexrothenberg  staff   204 Jan 13 12:39 lib/
  drwxr-xr-x  11 alexrothenberg  staff   374 Jan 13 12:33 test/

Everything looks good and you can check this into git and now have a frozen version of the hpricot gem stored with your application.
But if we stop here, when we deploy to our production server we'd be using the native extensions we built on your laptop which may not work on the server if you have one is 32bit and the other 64bit or you have different OS libraries installed or any number of other reasons.

To be safe, we need to rebuild the native extensions on the server when we deploy. This is not as hard as it sounds as rails gave us the rake task rake gems:build. We can ask capistrano to run that command on the server by adding the following to your Capfile.


after "deploy:finalize_update" do
  # build the native extensions for hpricot gem
  run "cd #{release_path} && #{rake} RAILS_ENV=#{rails_env} gems:build GEM=hpricot"
end

Now when capistrano deploys in with all the other messages you'll see something like


  ...
  * executing "cd /opt/apps/my_project/releases/20100108185109 && rake RAILS_ENV=production gems:build"
     servers: ["your.server.com"]
     [your.server.com] executing command
  ** [out :: your.server.com] (in /opt/apps/my_project/releases/20100108185109)
  ** [out :: your.server.com] Built gem: '/opt/apps/my_project/releases/20100108185109/vendor/gems/hpricot-0.8.2'
     command finished
  ...

So rails give us a few simple patterns to follow to freeze our gems in the vendor folder and with a few lines in you Capfile you can use this pattern to vendor a gem with native extensions.

Generate Models from Tables - Legacy Data Gem

2009-11-05T15:48:00.005-05:00

Today I'd like to announce the release of a gem I've been working on Legacy Data

Getting started on a Rails project with a large existing database can be daunting. How to you extract all the information that's
encoded in the database? Do you have to understand the entire data model before you get started? The models_from_tables generator
in the legacy_data gem can help! This generator looks into your existing database and generates ActiveRecord models based on the
information encoded in it.

How to use it

To generate an ActiveRecord model for each table in the database just type
script/generate models_from_tables

If you don't want all tables in the database tell it which table to model
script/generate models_from_tables --table-name comments
This uses any foreign_key constraints in the database to spider the database and model the comments table and all associated tables.

If you really only want the comments table tell it not to follow any foreign_keys
script/generate models_from_tables --table-name comments --skip-associated

If you use factory girl (and everyone should) it will generate a simple factory for each model it generates
script/generate models_from_tables --table-name comments --with-factories

(You do need to install the plugin gem install legacy_data as long as http://gemcutter.org is one of your gem sources)

Examples

Several examples come with the gem source in the examples folder on github. These include

A simple blog database tested with MySQL and Sqlite3

The Drupal 6.14 database tested with MySQL

The J2EE Petstore example tested with MySQL, Sqlite3 and Oracle

What kind of information can it extract from the database?

Associations

If the database contains foreign_key constraints it uses them to build has_many or belongs_to associations
in your ActiveRecord models

Validation constraints

It will generate the following types of validation constraints in your models

validates_uniqueness_of - For columns where the database has an index that enforces uniqueness

validates_presence_of - When the database column is non-nullable

validates_inclusion_of - For non-nullable boolean columns and custom constraints with a SQL rule flag IN ('Y', 'N')

validates_numericality_of - For integer columns (nullable and non-nullable)

custom validation - For custom SQL validation rules in the database it puts a placeholder in your model with the original SQL for you to translate into Ruby

Non-Rails naming conventions

Since the database is existing it's likely that it doesn't follow Rails naming conventions. Not to worry as the generator will
put the non-standard name into the generated models if it needs to.

What kinds of non-standard names can it generate?

Let's look at a sample output


class Post < ActiveRecord::Base

  set_table_name  :tbpost
  set_primary_key :postid
  
  # Relationships
  has_many :comments, :foreign_key => :postid

  # Constraints
  validates_presence_of :title, :body
  
end

Class Names - It named the model Post instead of the Rails convention Tbpost. The generator could not do this itself but knowing the conventions will often not apply to legacy databases it pauses after spidering the database giving you a chance to override the table to class name mapping. It generates a yaml file app/models/table_mappings.yml where you can verify or change any class name before proceeding to generate the models.

Table Names - It overrode the table name since the actual name tbpost does not match the Rails naming convention posts

Primary Keys - It overrode the primary key since the actual column postid does not match the Rails naming convention id

Foreign Keys - It overrode the foreign key on the comment table to be postid instead of the Rails naming convention id

PL/SQL debugging information now a part of Oracle Enhanced Adapter v1.2.2

2009-10-02T15:39:00.002-04:00

Raimonds Simanovskis has just published a version 1.2.2 of the Oracle Enhanced Adapter that includes the ability to capture dbms_output debug statements from your pl/sql code in the Rails log file. This is a bit of code that I wrote and blogged about a few months ago so not only do I think its useful but am very excited to have contributed to something many others use.

Using the Whenever gem to manage scheduled cron jobs without installing it on the server

2009-09-30T13:02:00.004-04:00

I've been using Javan's Whenever gem to manage scheduled jobs in my project and its fantastic!! There are many existing resources where you can learn more (readme, railscast google group) but I'd like to describe the specific way I'm using it

When the gem is not installed on my server

How administrators can use Capistrano to both schedule and unschedule your jobs

How to use a library such as the Oracle client that requires certain environment variables

At the end of the day we want to have 2 capistrano tasks we can run to have cron call a rake task of ours on the schedule we want.


  cap schedule_jobs
  cap unschedule_jobs

If we want to get fancy and pass a custom configuration argument that will be passed into the rake task.


  cap schedule_jobs SOME_CONFIGURATION=false

When the gem is not installed on my server

Let's start with the Capfile


  #Capfile
  desc "Schedule the jobs"
  task :schedule_job, :roles => :app, :only => { :primary => true } do
    some_configuration = ENV['SOME_CONFIGURATION'] || true #default to true
    arguments = ["RAILS_ENV=#{rails_env}",
                 "APP_PATH=#{current_path}",
                 "SOME_CONFIGURATION=#{some_configuration}"].join(' ')
    run "cd #{current_path} && #{rake} whenever:update_crontab #{arguments}"
  end

  desc "Unschedule the jobs"
  task :unschedule_job, :roles => :app, :only => { :primary => true } do
    run "cd #{current_path} && #{rake} whenever:update_crontab  UNSCHEDULE=true"
  end

How does this differ from the example on the whenever site? Since the gem is not installed on the server we cannot call whenever from the command line so invoke the whenever:update_crontab rake task instead and to allow administrators to easily disable the scheduled jobs we define the unschedule_jobs capistrano task. Let's take a look at the whenever:update_crontab Rake task that gets this all done.


  #lib/tasks/whenever.rake
  namespace :whenever do
    desc "updates crontab with our scheduled jobs"  
    task :update_crontab => :load_whenever_gem do     
      Whenever::CommandLine.execute({:update=>true, :identifier=>'YOUR_APP_NAME'})
    end

    task :load_whenever_gem do     
      begin
        gem_dir_root = "#{RAILS_ROOT}/vendor/gems/"
        chronic_gem_dir = Dir["#{RAILS_ROOT}/vendor/gems/*"].detect do |subdir|
          subdir.gsub(gem_dir_root,"") =~ /^(\w+-)?chronic-(\d+)/ && File.exist?("#{subdir}/lib/chronic.rb")
        end
        require "#{chronic_gem_dir}/lib/chronic"

        whenever_gem_dir = Dir["#{RAILS_ROOT}/vendor/gems/*"].detect do |subdir|
          subdir.gsub(gem_dir_root,"") =~ /^(\w+-)?whenever-(\d+)/ && File.exist?("#{subdir}/lib/whenever.rb")
        end
        require "#{whenever_gem_dir}/lib/whenever"

      rescue MissingSourceFile => e
        raise "Cannot find Whenever or Chronic : #{e}"
      end
    end
  end

The actual whenever:update_crontab task just does the same the command line does but unless you add config.gem 'whenever' to your environment (which I don't since whenever is not needed by my app at runtime) we also have the other task that loads whenever and chronic from the vendor/gems directory.

At this point we've gotten Capistrano calling a rake task to invoke whenever even though the gem is localized in my application but not installed on the server.

How administrators can use capistrano to both schedule and unschedule your jobs

The whenever gem does not have any support for unscheduling but it will schedule whatever is included in your schedule.rb file so if that file tells it to schedule nothing that's the same as unscheduling. Its easy to do that by wrapping the entire file with unless ENV['UNSCHEDULE'] (remember the UNSCHEDULE parameter we passed in the Capfile?)


  #config/schedule.rb
  unless ENV['UNSCHEDULE']

    module MyApp
      module Job
        class CronRakeTask < Whenever::Job::Default
          def output
            path_required
            "cd #{@path} && /usr/bin/env /usr/local/bin/cron_rake #{task} SOME_CONFIGURATION=#{ENV['SOME_CONFIGURATION']} RAILS_ENV=#{@environment}"
          end      
        end
      end
    end

    set :path,        ENV['APP_PATH'] || RAILS_ROOT 
    set :environment, RAILS_ENV || 'production'
    every 1.day, :at => '10:00pm' do
      command 'do_something', :class       => MyApp::Job::CronRakeTask, 
                              :environment => @environment, 
                              :path        => @path
    end
  end

What's going on with the weird CronRakeTask? This brings us to the final point

How to use a library such as the Oracle client that requires certain environment variables

Cron loads its environment settings differently than an interactive shell and typically does not have all the environment variables you may have in your profile file. You could solve this by adding them to the top of your crontab file but I prefer to leave that file as simple as possible and create a wrapper script to call instead of rake. Basically the CronRakeTask does the same as Whenever's built in RakeTask except it calls /usr/local/bin/cron_rake instead of rake.

The cron_rake file just sets the environment variables I need then calls rake.


  #!/bin/bash

  export PATH=/usr/local/lib/ruby-enterprise/bin:$PATH
  export ORACLE_HOME=/opt/oracle
  export LD_LIBRARY_PATH=/opt/oracle:$LD_LIBRARY_PATH

  rake $@

As I said in the beginning since I've been using the Whenever gem I no longer need to manually edit my crontab files ever and I can enable my jobs as part of my normal deployment process. Its wonderful and I think everyone should use it!

How to capture Oracle's dbms_output in your Rails log file

2009-08-19T09:09:00.006-04:00

I have been writing a rails application on top of a large existing Oracle database where each table has 5+ triggers that each call several stored procedures and each of those PL/SQL stored procedures is hundreds of lines long. Often a simple update statement fails with an ORA-xxxx exception coming from deep in the PL/SQL code and it can be tough to figure out what's gone wrong.

The usual way Oracle database folks figure out what's going on is to put print statements in their code. In oracle this looks like

  dbms_output.put_line('hi i hit this line of pl/sql');

When you're using an Oracle editor like TOAD or SQLDeveloper you have to turn output on and then will see anything that's printed.

  set serveroutput on;

This is great if you divide the application between Rails
and Database developers and assume each group can work independently to write perfect code but what about the real world!
Today I want to show you how I monkey patched the Oracle Enhanced Adapter to stick the dbms_output into
the rails log file.

Let's start with an simple example of a simple PL/SQL function that tells you if a string is more than 5 characters long (with some simple debugging print statements).

  CREATE OR REPLACE FUNCTION
  MORE_THAN_FIVE_CHARACTERS_LONG (some_text VARCHAR2) RETURN INTEGER
  AS
  longer_than_five INTEGER;
  BEGIN
    dbms_output.put_line('before the if -' || some_text || '-');
    IF length(some_text) > 5 THEN
      dbms_output.put_line('it is longer than 5');
      longer_than_five := 1;
    ELSE
      dbms_output.put_line('it is 5 or shorter');
      longer_than_five := 0;
    END IF;
    dbms_output.put_line('about to return: ' || longer_than_five);
    RETURN longer_than_five;
  END;

Now we can run the following in TOAD

  set serveroutput on;
  select MORE_THAN_FIVE_CHARACTERS_LONG('a long string') from dual;
  select MORE_THAN_FIVE_CHARACTERS_LONG('short') from dual;

And we get this output

  MORE_THAN_FIVE_CHARACTERS_LONG('ALONGSTRING') 
  --------------------------------------------- 
  1                                             

  1 rows selected

  before the if -a long string-
  it is longer than 5
  about to return: 1

  MORE_THAN_FIVE_CHARACTERS_LONG('SHORT') 
  --------------------------------------- 
  0                                       

  1 rows selected

  before the if -short-
  it is 5 or shorter
  about to return: 0

To get the same in our Rails app we just need to monkey patch the OracleEnhancedAdapter by copying what's below into your project as config/initializers/oracle_enhanced_adapter.rb

  module ActiveRecord
    module ConnectionAdapters
      class OracleEnhancedAdapter < AbstractAdapter
        DBMS_OUTPUT_BUFFER_SIZE = 10000  #can be 1-1000000
        DBMS_LINE_MAX_SIZE      = 1000  
        def enable_dbms_output
          @enable_dbms_output = true
          execute "BEGIN dbms_output.enable(#{DBMS_OUTPUT_BUFFER_SIZE}); END;"
        end
        def disable_dbms_output
          @enable_dbms_output = false
          execute "BEGIN dbms_output.disable(); END;"
        end
        def dbms_output_enabled?
          @enable_dbms_output
        end

        protected
          def log(sql, name)
            super(sql, name)
          ensure
            log_all_dbms_output if dbms_output_enabled?
          end

        private  
          def log_next_line_of_dbms_output
            dbms_output_text, status = @connection.exec "BEGIN dbms_output.get_line(:return, :status); END;", ' '*DBMS_LINE_MAX_SIZE, 1
            got_text = (status == 0)
            @logger.debug "DBMS_OUTPUT: #{dbms_output_text}" if got_text
            got_text
          end
        
          def log_all_dbms_output
            while log_next_line_of_dbms_output do
            end
          end
      end
    end
  end

To use it let's show a quick script/console session

  >> ActiveRecord::Base.connection.enable_dbms_output
  => []
  >> ActiveRecord::Base.connection.select_all("select MORE_THAN_FIVE_CHARACTERS_LONG('a long string') from dual")
  => [{"more_than_five_characters_long('alongstring')"=>1}]

And what's in log/development.log


    SQL (27.0ms)   BEGIN dbms_output.enable(10000); END;
    SQL (25.9ms)   select MORE_THAN_FIVE_CHARACTERS_LONG('a long string') from dual
  DBMS_OUTPUT: before the if -a long string-
  DBMS_OUTPUT: it is longer than 5
  DBMS_OUTPUT: about to return: 1

This is admittedly a very simple example but I have used this in a real application where I am updating several related ActiveRecord objects and seeing the DBMS_OUTPUT inline with the various SQL update statements has been extremely useful in tracking down a real bug in the PL/SQL procedure that has been in our system for over a year!

I've submitted this as a patch to the Oracle Enhanced Adapter so perhaps it will be included at some point so you wont have to do the monkey patching yourself. I was somewhat surprised not to find anything similar out there so if you know of something please leave a comment.

One year of blogging

2009-07-22T10:26:00.001-04:00

I just noticed that its been a year since my first post and thought I'd take a moment to reflect on the experience....

I set out with a goal of writing 2-3 posts a month and have managed to write 28 articles over 12 months so I feel pretty good about that. Most of the time I've gotten excited about something I accomplished or learned during the week. The process of writing it up turned out to be as much of a learning experience for myself as the original discovery. As they say "Teaching is the best way to learn"!

I've been most surprised how easy it is to get to the top of a Google Search results list (for very specific searches of course). With my Google Analytics tracking I can see more than 1/2 of my hits coming from search engines and not all of them are me searching for my own name :) Its been great when I've gotten comments from people I don't know (especially when the comments are positive).

Writing this blog has also helped turn my focus outward so I'm not only working on internal projects at my company but feel (in a small way) a part of the broader community.

I hope to continue in the coming year as I'm having a lot of fun so far ...

Using Github Through Draconian Proxies (Windows And Unix)

2009-06-25T09:54:00.003-04:00

I came across this great set of instructions on how to tunnel through a proxy to use github - http://returnbooleantrue.blogspot.com/2009/06/using-github-through-draconian-proxies.html

The proxy where I work is not quite so draconian that I need to follow these steps but its nice to have these instructions just in case :)

Metric_fu 1.1.0 released with a patch from me

2009-06-24T16:12:00.003-04:00

Metric_fu just released version 1.1.0 of their gem which I'm pleased to say includes a patch submitted by me. This is exciting as its the first time I've had my code included by someone I don't know in one of the open source projects I admire.

I was not originally going to make the patch as I thought I'd just hack around to fix the problem locally just enough to get it working but my friend Matt encouraged me to fix the root cause which turned out to be not too hard and got me into the metric_fu source. I always learn something when I read source from others I admire.

The process of submitting the patch was pretty easy and I plan to make it a habit when I run into issues with other gems. I,

Experienced a problem and decided not to live with it
Created a fork of metric_fu on github
Cloned locally and iteratively made my changes until I had my fix complete
Rebased my changes into a single commit (I wrote an article on this a few weeks ago)
Pushed my patch back to github on a branch
Submitted my patch back to Jake's repository as an issue

Solving the problem of gem dependencies with github names

I am using relevance-rcov instead of the rubyforge version of rcov as it has been maintained more recently and specifically fixed a segmentation fault problem I previously faced. The problem is that I couldn't install metric_fu as it put a gem dependency on 'rcov' in its gemspec. What seems broken is that there's no way to put a dependency on 'rcov' but have rubygems realize that 'relevance-rcov' is a fork of rcov so should satisfy the dependency.

I've seen a lot of talk recently about rubygems (including Why Using require ‘rubygems’ Is Wrong, RubyGems: Problems and (proposed) Solutions and Rip: A Next Generation Ruby Packaging System - Watch Out RubyGems!) but so far I haven't found a solution to the forking and naming problem emerge.

For Metric_fu I decided to change the install-time dependency to a runtime one. This solves my basic problem of not finding relevance-rcov but also allows for more flexibile use of metric_fu. If someone does not want to generate reek or rcov metrics why should they be forced to install those gems in order to use metric_fu? By defering the dependency until runtime we will never hit the dependency for those metrics we are not using.

You can look through the commit to see exactly how I did this.

Deleted the install time dependencies from metric_fu.gemspec
Added the runtime dependencies when a specific type of metric is instantiated by

Inserting a verify_dependencies! strategy step to the initialization in lib/base/generator.rb
Implementing verify_dependencies! in each of the subclasses for different metrics in lib/generators/*

Did it with TDD writing tests around everything before implementing my changes

Should gems use install time dependencies? Should a gem author decide which fork of a gem is required? What do you think?

Changing history with git rebase: How to combine several commits into one

2009-06-10T04:53:00.005-04:00

When I look at commits on github I'm always impressed at how concise they are. When I read a commit I can understand the intent of the change without getting distracted by the author's journey to get there. In contrast when I look at my commits they tend to be smaller and more incremental and meandering as I work my way down some false starts until I get to the solution I want. I'm guessing that I'm not alone in the way I work and recently discovered git rebase and an helpful tutorial showing how I can continue to work in my meandering style but package my changes to hide the journey before publishing to the world on github.

Let me show you what I mean with some changes I recently made to metric_fu. Over the course of a few days I made 6 small commits as you can see below.


> git log

commit d4b18b16e982ac57741f7a0a12cb085bb9b0e840
Author: Alex Rothenberg 
Date:   Mon Jun 1 09:55:37 2009 -0400

    reverted rakefile

commit ebda1cb67a2f0f0a85e51469d70919fa7c27d318
Author: Alex Rothenberg 
Date:   Mon Jun 1 09:49:05 2009 -0400

    refactoring of verify_dependencies

commit 3d45903b64a772fd09ec07bf69880bdb29ae4944
Author: Alex Rothenberg 
Date:   Fri May 29 20:40:49 2009 -0400

    runtime dependency check for all gems

commit 36e269bf8f0edccfb39cc767182406cdaa16a559
Author: Alex Rothenberg 
Date:   Fri May 29 20:24:16 2009 -0400

    logic for checking dependencies in generator base

commit bb2ee9a983c6adf54a8c95ee5851c6f8d3bffaba
Author: Alex Rothenberg 
Date:   Fri May 29 19:38:21 2009 -0400

    made rcov dependency figure itself out when generating rcov metrics

commit 750b000e5563e917f21eb1b7837e8001fb53f688
Author: Alex Rothenberg 
Date:   Fri May 29 16:42:12 2009 -0400

    removed gem dependency on rcov - to allow use of relevance-rcov or other github forks

commit d6af5089adce9eeed4916a155c3bdaeb4be6771a
Author: Randy Souza 
Date:   Sat May 16 08:47:37 2009 +0800

    Added a simple fix for cases where Saikuro results with nested information
    cause metrics:all to crash
    
    Signed-off-by: Jake Scruggs 

...

I am going to combine all of these into a single commit using the incredible power of git rebase. I find the last commit I do not want to change (the one made by Randy Souza on May 16th) and issue a git rebase command with that id.


> git rebase --interactive d6af5089adce9eeed4916a155c3bdaeb4be6771a

Now my editor comes up showing the 6 changes since then giving me options of what to do. Its very powerful, I can reorder commits, combine commits, eliminate commits. I'm going back in time to change the past!


pick 750b000 removed gem dependency on rcov - to allow use of relevance-rcov or other github forks
pick bb2ee9a made rcov dependency figure itself out when generating rcov metrics
pick 36e269b logic for checking dependencies in generator base
pick 3d45903 runtime dependency check for all gems
pick ebda1cb refactoring of verify_dependencies
pick d4b18b1 reverted rakefile

# Rebase d6af508..d4b18b1 onto d6af508
#
# Commands:
#  p, pick = use commit
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

In my case I want to combine these into a single commit so I change all but the first pick command to a squash and save.


pick 750b000 removed gem dependency on rcov - to allow use of relevance-rcov or other github forks
squash bb2ee9a made rcov dependency figure itself out when generating rcov metrics
squash 36e269b logic for checking dependencies in generator base
squash 3d45903 runtime dependency check for all gems
squash ebda1cb refactoring of verify_dependencies
squash d4b18b1 reverted rakefile

# Rebase d6af508..d4b18b1 onto d6af508
#
# Commands:
#  p, pick = use commit
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

It now gives me a chance to edit the new commit message which it defaults to the original messages concatenated together. This really is a single commit so if one of the original commits changed a file and a subsequent one undid the change that file will no longer appear in the list of modified files.



# This is a combination of 6 commits.
# The first commit's message is:
removed gem dependency on rcov - to allow use of relevance-rcov or other github forks

# This is the 2nd commit message:

made rcov dependency figure itself out when generating rcov metrics

# This is the 3rd commit message:

logic for checking dependencies in generator base

# This is the 4th commit message:

runtime dependency check for all gems

# This is the 5th commit message:

refactoring of verify_dependencies

# This is the 6th commit message:

reverted rakefile

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# Explicit paths specified without -i nor -o; assuming --only paths...
# Not currently on any branch.
# Changes to be committed:
#   (use "git reset HEAD ..." to unstage)
#
#       modified:   Rakefile
#       modified:   lib/base/generator.rb
#       modified:   lib/generators/flay.rb
#       modified:   lib/generators/flog.rb
#       modified:   lib/generators/rcov.rb
#       modified:   lib/generators/reek.rb
#       modified:   lib/generators/roodi.rb
#       modified:   lib/generators/saikuro.rb
#       modified:   metric_fu.gemspec
#       modified:   spec/base/generator_spec.rb
#       modified:   spec/generators/flay_spec.rb
#       modified:   spec/generators/flog_spec.rb
#       modified:   spec/generators/reek_spec.rb
#

Let's edit this commit message to read something like


Changed gem dependencies from install-time in gemspec to runtime when each
of the generators is loaded.  This allows use of github gems (i.e.
relevance-rcov instead of rcov) and also allows you to install only the
gems for the metrics you plan on using.

                                     
# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# Explicit paths specified without -i nor -o; assuming --only paths...
# Not currently on any branch.
# Changes to be committed:
#   (use "git reset HEAD ..." to unstage)
#
#       modified:   Rakefile
#       modified:   lib/base/generator.rb
#       modified:   lib/generators/flay.rb
#       modified:   lib/generators/flog.rb
#       modified:   lib/generators/rcov.rb
#       modified:   lib/generators/reek.rb
#       modified:   lib/generators/roodi.rb
#       modified:   lib/generators/saikuro.rb
#       modified:   metric_fu.gemspec
#       modified:   spec/base/generator_spec.rb
#       modified:   spec/generators/flay_spec.rb
#       modified:   spec/generators/flog_spec.rb
#       modified:   spec/generators/reek_spec.rb
#

And that's it! Now when we check our history we see.


> git log

commit 6f389364eb972871867d3a71677b8eb7046541a2
Author: Alex Rothenberg 
Date:   Fri May 29 16:42:12 2009 -0400

    Changed gem dependencies from install-time in gemspec to runtime when each
    of the generators is loaded.  This allows use of github gems (i.e.
    relevance-rcov instead of rcov) and also allows you to install only the
    gems for the metrics you plan on using.

commit d6af5089adce9eeed4916a155c3bdaeb4be6771a
Author: Randy Souza 
Date:   Sat May 16 08:47:37 2009 +0800

    Added a simple fix for cases where Saikuro results with nested information
    cause metrics:all to crash
    
    Signed-off-by: Jake Scruggs 

...

And this commit is how I pushed this patch to github for all the world to see.

How to use dates in Rails when your database stores a string

2009-05-21T16:55:00.018-04:00

When working with Rails there's a lot of magic that happens behind the scenes to make it easy to do complex things. Most of the time you don't need to know how that "magic" works but there are times when things don't work as expected and its helpful to dig in and understand what Rails is doing under the covers so you can change how it works. Did I just say "change how Rails works"?!? I did! Rails is opinionated software that seeks to lead you down the golden path but there are legitimate times when you have to veer off that path and Rails lets you do so. I find this most often happens to me when I'm dealing with an existing legacy database which is not setup as Rails would like.

Today I'm going to go through an example that happened to me recently when I had an existing database that stored some dates in a text column but I needed to treat them as dates in my UI. I couldn't change the type of that column as there was another legacy application that expected it to be text.

Using Dates the Rails Way

First let's look at how easy it is to work with dates when you can follow the Rails Way. Let's create a new project and add a scaffolded Person object with a date attribute called birthday.


rails date_select_example

cd date_select_example

script/generate scaffold person name:string birthday:date

rake db:migrate

Now if we hit the site and try to create a new person we see a screen like this

And when you click "Create"

Exactly what you want with almost no code in the view and none in the model. I did make a minor edit to the view so we would get 1960 in the year select by adding :start_year=>1900 (a full documentation of date_select options are available here)


#app/views/people/new.html.erb

<h1>New person

<% form_for(@person) do |f| %>

  <%= f.error_messages %>

  <p>

    <%= f.label :name %><br />

    <%= f.text_field :name %>

  </p>

  <p>

    <%= f.label :birthday %><br />

    <%= f.date_select :birthday, :start_year=>1900 %>

  </p>

  <p>

    <%= f.submit 'Create' %>

  </p>

<% end %>



<%= link_to 'Back', people_path %>

and


#app/models/person.rb

class Person < ActiveRecord::Base

end

Errors when storing as text in the database

Now what happens when you run into a case where the date is stored as a string in the database. Let's say we have an "anniversary" attribute stored as a string that we want to treat the same way as we did birthday. We create the migration.


script/generate migration AddAnniversaryToPerson anniversary:string

rake db:migrate

Then add the anniversary to our view.


#app/views/people/edit.html.erb

<h1>Editing person

<% form_for(@person) do |f| %>

  <%= f.error_messages %>

  <p>

    <%= f.label :name %><br />

    <%= f.text_field :name %>

  </p>

  <p>

    <%= f.label :birthday %><br />

    <%= f.date_select :birthday, :start_year=>1900  %>

  </p>

  <p>

    <%= f.label :anniversary %><br />

    <%= f.date_select :anniversary, :start_year=>1900  %>

  </p>

  <p>

    <%= f.submit 'Update' %>

  </p>

<% end %>



<%= link_to 'Show', @person %> |

<%= link_to 'Back', people_path %>

It looks like we're done so we click "Update" and .. Oops. It doesn't work! We get the error 1 error(s) on assignment of multiparameter attributes. Now we need to figure out what multiparameter attributes are and whey they're not working for us.


ActiveRecord::MultiparameterAssignmentErrors in PeopleController#update



1 error(s) on assignment of multiparameter attributes

RAILS_ROOT: /Users/alexrothenberg/date_select_example



Application Trace | Framework Trace | Full Trace

/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:3061:in `execute_callstack_for_multiparameter_attributes'

/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:3022:in `assign_multiparameter_attributes'

/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:2749:in `attributes='

/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:2627:in `update_attributes'

/Users/alexrothenberg/date_select_example/app/controllers/people_controller.rb:63:in `update'

...more stack trace...



Request



Parameters:



{"commit"=>"Update",

 "_method"=>"put",

 "authenticity_token"=>"qezkVq+MNzFuXxFBJ/GaSoh2BNdxM6oF3H7JP5beFFE=",

 "id"=>"1",

 "person"=>{"name"=>"Barack Obama",

 "birthday(2i)"=>"8",

 "birthday(3i)"=>"4",

 "anniversary(1i)"=>"2009",

 "anniversary(2i)"=>"5",

 "anniversary(3i)"=>"22",

 "birthday(1i)"=>"1960"}}

What happened? There are two keys to figuring out what's going on

The date_select helper actually sends 3 http parameters to our application anniversary(1i), anniversary(2i) and anniversary(3i). ActiveRecord must combine those into a single Date before updating the row in the database.
Looking at assign_multiparameter_attributes in active_record/base.rb we see this this comment that talks about combining 3 http parameters into a date type by calling new on the column type

Instantiates objects for all attribute classes that needs more than one constructor parameter. This is done by calling new on the column type or aggregation type (through composed_of) object with these parameters.
So having the pairs written_on(1) = "2004", written_on(2) = "6", written_on(3) = "24", will instantiate written_on (a date type) with Date.new("2004", "6", "24"). You can also specify a typecast character in the parentheses to have the parameters typecasted before they're used in the constructor. Use i for Fixnum, f for Float, s for String, and a for Array. If all the values for a given attribute are empty, the attribute will be set to nil.

Now we know we're close to the right place and can check the column's return type using script/console


$ script/console

Loading development environment (Rails 2.3.2)

>> Person.columns_hash['birthday']

=> #

>> Person.columns_hash['birthday'].klass

=> Date

>> Person.columns_hash['anniversary']

=> #

>> Person.columns_hash['anniversary'].klass

=> String

This makes sense. ActiveRecord is treating the anniversary column as a string because that's what it is in the database so is not combining the 3 multiparameter attributes into a Date. What we need to do is override the default ActiveRecord logic and tell it to treat this column as a date. We can write a failing spec


require File.dirname(__FILE__) + '/../spec_helper'



describe Person do

  it "should treat anniversary as a Date column" do

    Person.columns_hash['anniversary'].klass.should == Date

  end

end



# 'Person should treat anniversary as a Date column' FAILED

# expected: Date,

#      got: String (using ==)

Then take advantage of a trick of Ruby that allows you to extend an object without affecting other instances of its class (see PickAxe book's explanation of this technique). If we lookup the definition of klass for a column It is actually very simple to implement our fix with the 5 lines below.


class Person < ActiveRecord::Base



  class << columns_hash['anniversary']

    def type

      :date

    end

  end

end

Now our tests pass and when we go back to our site and click the Update button and it works.

Testing AJAX without a browser with Cucumber and Webrat

2009-05-06T13:43:00.012-04:00

I have lately fallen in love with using Cucumber and Webrat for my integration/acceptance testing. Cucumber because it allows non-technical people to write or at least read the test scenarios and Webrat because it matches content and encourages you to write integration tests without relying on xpath to find html elements. The way I like to use these tools is to run Rails integration tests which means its fast since I don’t need to start a mongrel or fire up a browser and can use Rails’ transactional fixtures to rollback all my database changes at the end of each test scenario. The only downside is that you can’t test javascript.

Today I am going to talk about how to get around this and test a form with an ajax autocomplete field. I've built a sample application with all the code examples here and you can download it from http://github.com/alexrothenberg/testing-ajax-example if you like. The application I'm building is just some simple app created with scaffolding that just has a User resource with a name and address. I modified the /users page to not display all users but include the auto_complete typeahead to let you pick a user (imagining there may be a lot) so the page looks something like this.

My first test scenario will ignore the ajax and just test the form which is super easy and can be done by writing a single feature file.


#features/find_a_user.feature
Feature: Allow anyone to find a user and see their details
  In order to handle a large set of users
  I want search with autocomplete
 
  Scenario: View a candidate detail page without testing ajax
    Given "Mickey Mouse" is a user living at "123 Main Street"
    When I am on the homepage
      And I fill in "Which user" with "Mickey Mouse"
      And I press "Find"
    Then I should see "Mickey Mouse"
      And I should see "123 Main Street"

I run it it all passes and I get


$ rake features
(in /Users/alexrothenberg/ruby/testing-ajax-example)
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby -I "/Library/Ruby/Gems/1.8/gems/cucumber-0.3.2/lib:lib" "/Library/Ruby/Gems/1.8/gems/cucumber-0.3.2/bin/cucumber" --format pretty --require features/step_definitions/user_steps.rb --require features/step_definitions/webrat_steps.rb --require features/support/env.rb --require features/support/paths.rb features/find_a_user.feature
Feature: Allow anyone to find a user and see their details
  In order to handle a large set of users 
  I want search with autocomplete

  Scenario: View a candidate detail page without testing ajax     # features/find_a_user.feature:5
    Given "Mickey Mouse" is a user living at "123 Main Street"    # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                     # features/step_definitions/webrat_steps.rb:6
    And I fill in "Which user" with "Mickey Mouse"                # features/step_definitions/webrat_steps.rb:22
    And I press "Find"                                            # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                              # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                            # features/step_definitions/webrat_steps.rb:93

1 scenario (1 passed)
6 steps (6 passed)

I makes use of the default webrat steps that cucumber gives you for free in features/steps/webrat_steps.rb so I don’t even have to write any code to get it to pass but I still haven’t tested any of my code that responds to the autocomplete request. I’d like my test to verify that my routes, controller and model will all work together. So, I write another scenario and run it and this time it fails because I haven't defined the typeahead steps for the typeahead lines.


<first scenario omitted>

Scenario: View a candidate detail page testing (most of) the ajax # features/find_a_user.feature:13
    Given "Mickey Mouse" is a user living at "123 Main Street"      # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                       # features/step_definitions/webrat_steps.rb:6
    And I typeahead in "Which user" with "Mickey Mouse"             # features/find_a_user.feature:16
    And I fill in "Which candidate" with the first typeahead result # features/find_a_user.feature:17
    And I press "Find"                                              # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                                # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                              # features/step_definitions/webrat_steps.rb:93

2 scenarios (1 undefined, 1 passed)
13 steps (3 skipped, 2 undefined, 8 passed)

You can implement step definitions for undefined steps with these snippets:

When /^I typeahead in "([^\"]*)" with "([^\"]*)"$/ do |arg1, arg2|
  pending
end

When /^I fill in "([^\"]*)" with the first typeahead result$/ do |arg1|
  pending
end

Now I take the hints cucumber has given me and write my autocomplete steps. The interesting thing here is that I need to leave the response object unchanged so I can fill in the form field after running the typeahead step so I can't use the existing webrat steps as they work on a single pair of request and response objects. So I knew I'd be creating a new class with its own request and response that could be used without affecting the one used by my other cucumber steps. Using good outside-in development practices I deferred thinking about how to do that and first wrote my steps file to look something like this. One interesting thing to notice here is that you can invoke a step from inside another step just by omitting the block as I do in the second step.


When /^I typeahead in "(.*)" with "(.*)"$/ do |field, value|
  field = field_labeled field
  @typeahead = AutoCompleteStepHelper.new(request)
  @typeahead.type(field, value)
end

When /^I fill in "(.*)" with the first typeahead result$/ do |field|
  When %Q[I fill in "#{field}" with "#{@typeahead.items.first}"]
end

Now when run the feature again it fails telling me I haven’t yet built the AutoCompleteStepHelper.


<first scenario omitted>

Scenario: View a candidate detail page testing (most of) the ajax # features/find_a_user.feature:13
    Given "Mickey Mouse" is a user living at "123 Main Street"      # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                       # features/step_definitions/webrat_steps.rb:6
    And I typeahead in "Which user" with "Mickey Mouse"             # features/step_definitions/autocomplete_steps.rb:1
      uninitialized constant AutoCompleteStepHelper (NameError)
      ./features/step_definitions/autocomplete_steps.rb:3:in `/^I typeahead in "(.*)" with "(.*)"$/'
      features/find_a_user.feature:16:in `And I typeahead in "Which user" with "Mickey Mouse"'
    And I fill in "Which candidate" with the first typeahead result # features/step_definitions/autocomplete_steps.rb:7
    And I press "Find"                                              # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                                # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                              # features/step_definitions/webrat_steps.rb:93

2 scenarios (1 failed, 1 passed)
13 steps (1 failed, 4 skipped, 8 passed)

So the next step is to write the AutoCompleteStepHelper class. This class’ job is to have its own request and response that will not affect the ones used by cucumber for the main page requests. It turns out that I can do this by having my class extend ActionController::IntegrationTest and I can even use webrat methods in it because webrat adds its methods to IntegrationTest. In this example I'm calling visit and current_dom and using nokogiri to parse the dom. It is a little weird that I'm subclassing IntegrationTest but this class is not a TestUnit class itself but I decided that was okay.


class AutoCompleteStepsHelper < ActionController::IntegrationTest
  def initialize(existing_request)
    @controller_name = existing_request.parameters[:controller]
    @controller_class = "#{@controller_name.to_s.camelize}Controller".constantize
    raise "Can't determine controller class for #{@controller_class_name}" if @controller_class.nil?

    @controller = @controller_class.new
    @request = ActionController::TestRequest.new
    @response = ActionController::TestResponse.new
    @response.session = @request.session
  end
 
  def type(field, value)
    visit url_for(:controller=>@controller_name, :action=>"auto_complete_for_#{field.id}", field.send(:name)=>value)
  end
 
  def items
    current_dom.search('//ul/li').map(&:inner_html)
  end
end

Now when I run the feature it all passes.


<first scenario omitted>

  Scenario: View a candidate detail page testing (most of) the ajax # features/find_a_user.feature:13
    Given "Mickey Mouse" is a user living at "123 Main Street"      # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                       # features/step_definitions/webrat_steps.rb:6
    And I typeahead in "Which user" with "Mickey Mouse"             # features/step_definitions/autocomplete_steps.rb:1
    And I fill in "Which user" with the first typeahead result      # features/step_definitions/autocomplete_steps.rb:7
    And I press "Find"                                              # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                                # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                              # features/step_definitions/webrat_steps.rb:93

2 scenarios (2 passed)
13 steps (13 passed)

The last step I took was to test that the list returned in the typeahead was correct. I created another scenario in my feature.


  Scenario: Typeahead should return 2 users that match but not a third
    Given "Mickey Mouse" is a user living at "123 Main Street"
      And "Donald Duck" is a user living at "123 Pond Lane"
      And "Minnie Mouse" is a user living at "123 Disney Avenue"
    When I am on the homepage
      And I typeahead in "Which user" with "Mi"
    Then I should see in my typeahead "Mickey Mouse"
      And I should see in my typeahead "Minnie Mouse"
      And I should not see in my typeahead "Donald Duck"

and I added two new steps


Then /^I should see in my typeahead "(.*)"$/ do |text|
  @typeahead.response_body.should =~ /#{text}/m
end

Then /^I should not see in my typeahead "(.*)"$/ do |text|
  @typeahead.response_body.should_not =~ /#{text}/m
end

Now when I run my features all 3 scenarios are passing


$ rake features
(in /Users/alexrothenberg/ruby/testing-ajax-example)
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby -I "/Library/Ruby/Gems/1.8/gems/cucumber-0.3.2/lib:lib" "/Library/Ruby/Gems/1.8/gems/cucumber-0.3.2/bin/cucumber" --format pretty --require features/step_definitions/autocomplete_steps.rb --require features/step_definitions/user_steps.rb --require features/step_definitions/webrat_steps.rb --require features/support/autocomplete_steps_helper.rb --require features/support/env.rb --require features/support/paths.rb features/find_a_user.feature
Feature: Allow anyone to find a user and see their details
  In order to handle a large set of users 
  I want search with autocomplete

  Scenario: View a candidate detail page without testing ajax  # features/find_a_user.feature:5
    Given "Mickey Mouse" is a user living at "123 Main Street" # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                  # features/step_definitions/webrat_steps.rb:6
    And I fill in "Which user" with "Mickey Mouse"             # features/step_definitions/webrat_steps.rb:22
    And I press "Find"                                         # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                           # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                         # features/step_definitions/webrat_steps.rb:93

  Scenario: View a candidate detail page testing (most of) the ajax # features/find_a_user.feature:13
    Given "Mickey Mouse" is a user living at "123 Main Street"      # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                       # features/step_definitions/webrat_steps.rb:6
    And I typeahead in "Which user" with "Mickey Mouse"             # features/step_definitions/autocomplete_steps.rb:1
    And I fill in "Which user" with the first typeahead result      # features/step_definitions/autocomplete_steps.rb:7
    And I press "Find"                                              # features/step_definitions/webrat_steps.rb:14
    Then I should see "Mickey Mouse"                                # features/step_definitions/webrat_steps.rb:93
    And I should see "123 Main Street"                              # features/step_definitions/webrat_steps.rb:93

  Scenario: Typeahead should return 2 users that match but not a third # features/find_a_user.feature:22
    Given "Mickey Mouse" is a user living at "123 Main Street"         # features/step_definitions/user_steps.rb:1
    And "Donald Duck" is a user living at "123 Pond Lane"              # features/step_definitions/user_steps.rb:1
    And "Minnie Mouse" is a user living at "123 Disney Avenue"         # features/step_definitions/user_steps.rb:1
    When I am on the homepage                                          # features/step_definitions/webrat_steps.rb:6
    And I typeahead in "Which user" with "Mi"                          # features/step_definitions/autocomplete_steps.rb:1
    Then I should see in my typeahead "Mickey Mouse"                   # features/step_definitions/autocomplete_steps.rb:16
    And I should see in my typeahead "Minnie Mouse"                    # features/step_definitions/autocomplete_steps.rb:16
    And I should not see in my typeahead "Donald Duck"                 # features/step_definitions/autocomplete_steps.rb:20

3 scenarios (3 passed)
21 steps (21 passed)

What I've done is not full javascript testing (for that I'm planning to look into Blue-Ridge from Relevance). This technique does allow you to test ajax (skipping the "J") without a browser.

Managing the draw of a single-elimination tournament

2009-03-30T11:02:00.004-04:00

My company has been running a GO tournament and since its around NCAA March Madness time I thought it'd be easy to find a site to manage the draw. I looked but couldn't find anything that did what I wanted so I wrote my own. Caveat: I just spent a few days on this and it could stand to be enhanced with more features and a rewrite of the ui implementation but I'm open to suggestions if anyone finds this useful.

To use it

Edit the file lib/names.txt with a the players in your tournament

Run 'rake create_tournament

Now when you view the site you'll see a draw that looks something like this.

For games with 2 players - all first round games and games where the previous round winners have been determined - you'll be able to click the link and set which player won.

This was all very quickly thrown together but if you're interested in hosting any sort of single-elimination tournament check it out on github tournament-draw.

Let me know if you use it and how you'd like to enhance it from the quick application it is today.

Link: Search Engine in 200 lines of Ruby Code

2009-03-22T21:47:00.002-04:00

I read a very interesting article by someone who works for Yahoo about how he wrote a basic search engine in Ruby in just a few files. I wouldn't use this for a production system for as an example of how search engines work its fascinating and I plan to keep following his site to see how he enhances it

Description: http://blog.saush.com/2009/03/write-an-internet-search-engine-with-200-lines-of-ruby-code/

Source: http://github.com/sausheong/saushengine

... and he's got a really cool css/javascript formatter for the source in his blog. I've gotta learn more about SyntaxHighlighter

Java as a Legacy Language

2009-03-18T15:13:00.004-04:00

I came across this article titled Java as Legacy Language today. As an ex-Java guy who is now committed to Ruby I was amused by the title but also think he makes a good point.

One thing is for sure: If you're in the software development business, don't cling to old ways of doing development. And also, don't get too carried away thinking that something like Scrum is going to be the Bandaid that fixes your agility problems, because it may turn out that your main problem is Java itself. Keep an open mind. Try new things. Be ready when the next disruption arrives, or you may find yourself without a chair when the music stops.

I don't think Java is "bad" but I do think it encourages you to write big, complicated applications to solve big, complicated problems. In contrast I find Rails encourages you to realize most of the apps we write are in fact fairly simple and do the same CRUD steps over and over. Having so much support for that built into the framework helps you think about what's "interesting" about your appplication which may be unique but is probably not big.

Add to this how easy it is to test in Rails - tools like RSpec & Shoulda, mocking tools, integration testing and more recently cucumber + webrat all make it easier to practice TDD than not to. While there are tools in Java, the fact that they're harder to use, I believe, means fewer developers will use them and these practices will remain less ingrained in that community.

Using scopes in auto_complete plugin

2009-03-18T09:05:00.004-04:00

My colleague Pat Shaughnessy has spent a lot of time recently enhancing the auto_complete plugin. I suggest you read his blog posts and check out his fork of auto_complete on github to see the details.

I was reading his latest change to allow filtering of auto complete picklists and really like what he did but thought there was one thing that didn't quite feel right - the fact that you have to mix the application logic to filter the list with the plugin logic to find the list in the block in your controller.

Here's the code Pat wrote in his controller and what I'd like to avoid is having to re-implement the "LOWER(tasks.name) LIKE ?" portion that's already implemented in the filtered_auto_complete_for method of autocomplete.rb in the plugin.


# For task name auto complete, only display tasks
# that belong to the given project: 
filtered_auto_complete_for :task, :name do | find_options, params|
  find_options.merge!(
    {
      :include => :project,
      :conditions => [ "LOWER(tasks.name) LIKE ? AND projects.name = ?",
                       '%' + params['task']['name'].downcase + '%',
                       params['project'] ],
      :order => "tasks.name ASC"
    }
  )
end

I'd like to propose a slight modification so the block you write as part of your application can focus just on the filtering and leave the responsibility for the search with plugin. I'm also proposing we use scopes (which I don't think were around when the original auto_complete plugin was written) to filter and sort the list the plugin. Here is the code I'd like to write in my application.


class ProjectController < ApplicationController
  # For task name auto complete, only display tasks
  # that belong to the given project: 
  filtered_auto_complete_for :task, :name do | list, params |
    list.by_project(params['project']['id'])
  end
end

class Project < ActiveRecord::Base
  named_scope :by_project, 
              lambda {|project_id| {:conditions => {:project_id => project_id} } }
end

We can do this with a simple modification of the plugin implementation of filtered_auto_complete_for


def filtered_auto_complete_for(object, method)
  define_method("auto_complete_for_#{object}_#{method}") do
    find_options = { 
      :conditions => [ "LOWER(#{method}) LIKE ?", '%' +
        params[object][method].downcase + '%' ], 
      :order => "#{method} ASC",
      :limit => 10 }
    @items = object.to_s.camelize.constantize.scoped(find_options)
    @items = yield(@items, params) if block_given?

    render :inline => "<%= auto_complete_result @items, '#{method}' %>"
  end
end

Originally, it used passed the find_options hash to the block and then executed the search in one step as "@items = object.to_s.camelize.constantize.find(:all, find_options)". My change is to rely on ActiveRecord proxy objects to chain criteria together leaving the block independent of the criteria here in the plugin. The best part is you don't have to worry about performance as ActiveRecord will intelligently combine the criteria into a single SQL request. For example searching for tasks that start with 'tas' within project #7


 User Load (1.2ms)   SELECT * FROM `tasks` WHERE ((`tasks`.`project_id` = '7') AND (LOWER(name) LIKE '%tas%'))  ORDER BY name ASC LIMIT 10

Microsoft Office links causing InvalidAuthenticityToken in Rails

2009-03-04T10:00:00.005-05:00

I started receiving a lot of error notifications recently from my ExceptionNotfier plugin for an error with ActionController::InvalidAuthenticityToken. It turned out the error was occurring because one of my users was pasting a link to my app in an MS Office document and when Office sees the link it makes a request that Rails could not handle. Here I'll show you a simple fix you can use to avoid these errors with much credit going to an article at
Dealing with Microsoft Office Protocol Discovery in Rails
.

My execptions looked something like this (lots of boring details omitted)


A ActionController::InvalidAuthenticityToken occurred in events#1164:

  ActionController::InvalidAuthenticityToken
  [RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/request_forgery_protection.rb:86:in `verify_authenticity_token'

-------------------------------
Environment:
-------------------------------
  * HTTP_USER_AGENT        : Microsoft Data Access Internet Publishing Provider Protocol Discovery
  * REQUEST_METHOD         : OPTIONS

The problem is that Rails doesn't understand the method 'OPTIONS' (see rails/actionpack/lib/action_controller/routing.rb


#around line 270 of routing.rb
module ActionController
  module Routing
    HTTP_METHODS = [:get, :head, :post, :put, :delete]
  end
end

Fixing the problem is fairly simple. You insert a before_filter into your application controller to intercept and handle requests with the option method before the rails code realizes it can't handle the request.


class ApplicationController < ActionController::Base
  before_filter :options_for_microsoft_office_protocol_discovery

  ...

  def options_for_microsoft_office_protocol_discovery
    render :nothing => true, :status => 200 if request.method == :options
  end
end

Its also easy to write a simple spec in rspec to verify the behavior. There is one trick which is that rails/actionpack/lib/action_controller/test_process.rb defines helper methods for get, post, put, delete & head that we can't use so we need to call the underlying method directly but the signature for that underlying method changed with Rails 2.3 (commit) so depending what version you're using you'll need one of 2 flavors.


  # Rails 2.3 and above 
  it 'should not throw an exception on OPTIONS request (from ms office protocol discovery)' do
    process '/any/goofy/path', nil, nil, nil, 'OPTIONS'
    response.should be_success
  end 

  # Rails < 2.3 version
  it 'should not throw an exception on OPTIONS request (from ms office protocol discovery)' do
    @request.env['REQUEST_METHOD'] = 'OPTIONS'
    process '/any/goofy/path'
    response.should be_success
  end

You can also test from the command line using curl


curl -X OPTIONS http://localhost:3000/

Update: It turns out that Rails stores the acceptable methods in 2 different places actionpack/lib/action_controller/request.rb which does include all of get head put post delete options and also (see rails/actionpack/lib/action_controller/routing.rb which only includes :get, :head, :post, :put, :delete (options is missing). This means this fix will only work for OPTIONS requests and not any other type as ActionController::Request request_method will throw an exception before getting to the filter code above.

Investigating how Symbol to_proc works

2009-02-07T11:39:00.007-05:00

One of the things I love about Ruby is how expressive it is and how with open classes it can be optimized to become even more expressive. Since I started using Ruby I don't think I've written a single for or while loop - something I couldn't have imagined saying with any other language! Of course I do this by using iterators and writing code like


user_names = User.all.collect {|user| user.name}

I recently started discovered I could write the same thing even more concisely (as long as I'm using Rails or Ruby 1.9)


user_names = User.all.collect(&:name)

I decided to investigate how this works.

First I found some good posts by Prag Dave and Ryan Bates and at InfoQ. This helped but I still didn't understand it all so decided to dig further.

First I took a look at how Rails extends Symbol


unless :to_proc.respond_to?(:to_proc)
  class Symbol
    # Turns the symbol into a simple proc, which is especially useful for enumerations. Examples:
    #
    # # The same as people.collect { |p| p.name }
    # people.collect(&:name)
    #
    # # The same as people.select { |p| p.manager? }.collect { |p| p.salary }
    # people.select(&:manager?).collect(&:salary)
    def to_proc
      Proc.new { |*args| args.shift.__send__(self, *args) }
    end
  end
end

So they defined the to_proc method on symbol and that means the new code will be called when we write &:name because it magically gets transformed into :name.to_proc. I learned something but still needed to learn more to understand how it all works.

Why does the & cause Ruby to call to_proc? I knew that & in the last parameter declaration will pass a provided block as a parameter but this seems to be doing the reverse. Calling a method as an argument but having it interpreted as a block. I tried a couple of experiments in irb


def was_block_given?
  block_given?
end

# As expected
was_block_given? {}
=> true 

# Passing a proc is not the same as having a block
was_block_given? Proc.new{}
ArgumentError: wrong number of arguments (1 for 0)
 from (irb):195:in `was_block_given?'
 from (irb):195

# Prefixing the proc with an & makes it like a block
was_block_given? &Proc.new{}
=> true

It was not all as I expected but some reading through the PickAxe book led me to a better understanding. I found this paragraph in the Calling A Method section (page 115 in my copy)

If the last argument to a method is preceded by an ampersand, Ruby assumes that it is a Proc object.
It removes it from the parameter list, converts the Proc object into a block, and associates it with the method.

Ok so now I know why when Ruby sees User.all.collect(&:name) it invokes the collect method with name.to_proc as a block. Next, it was time to figure out why the code Rails put in the to_proc method worked. I took a look at the Rubinius implementation Enumerable


  def collect
    ary = []
    if block_given?
      each { |o| ary << yield(o) }
    else
      each { |o| ary << o }
    end
    ary
  end

Again I decided to experiment with irb to see what each part of the to_proc implementation was doing. First I redefined the Symbol to_proc again with a puts so I could confirm what was going on.


class Symbol
  # Turns the symbol into a simple proc, which is especially useful for enumerations. Examples:
  #
  #   # The same as people.collect { |p| p.name }
  #   people.collect(&:name)
  #
  #   # The same as people.select { |p| p.manager? }.collect { |p| p.salary }
  #   people.select(&:manager?).collect(&:salary)
  def to_proc
    Proc.new do |*args| 
      puts "to_proc args: #{args.inspect}"
      args_shift = args.shift
      puts "to_proc: #{args_shift.inspect}.__send__(#{self.inspect}, *#{args.inspect})"
      result = args_shift.__send__(self, *args) 
      puts "to_proc result: #{result.inspect}"
      result
    end
  end
end

# Make the call and see what happens
[1].collect( &:to_s)
# to_proc args: [1]
# to_proc: 1.__send__(:to_s, *[])
# to_proc result: "1"
# => ["1"]

Its brute force but tells us everything we need to know. As expected collect yields to our proc/block with the element in a variable length argument [1], it extracts the 1 and sends it the to_s method with no arguments returning the string "1". At this point I think I understand how it all works and decide to confirm by running a few more (more complicated) tests in irb


[1, 'hi', :b].collect( &:to_s)
# to_proc args: [1]
# to_proc: 1.__send__(:to_s, *[])
# to_proc result: "1"
# to_proc args: ["hi"]
# to_proc: "hi".__send__(:to_s, *[])
# to_proc result: "hi"
# to_proc args: [:b]
# to_proc: :b.__send__(:to_s, *[])
# to_proc result: "b"
# => ["1", "hi", "b"]

[1,2,3].inject(&:+)
# to_proc args: [1, 2]
# to_proc: 1.__send__(:+, *[2])
# to_proc result: 3
# to_proc args: [3, 3]
# to_proc: 3.__send__(:+, *[3])
# to_proc result: 6
# => 6

1.__send__(:+, *[2])
# => 3

:b.__send__(:to_s, *[])
# => "b"

It all works as expected and I decide I know as much as I need to about this and call it a day.

So why did I bother figuring all this out and then writing it up? Mostly because I didn't know how it worked and thought there was some 'magic' going on. I could have continued using this feature without understanding how it worked but now that I understand how it works if some need ever arises for me to do some similar magic I know how to go about it. As for writing it up I hope someone else may read this and find it useful but by I increased my own understanding through the act of writing.

Maintaining your technical chops is a full time job

2009-02-05T13:41:00.002-05:00

Great quote from Uncle Bob (near the bottom of the post)

I think that maintaining your technical chops is a full time job. For that reason I have avoided becoming a business wonk. I hire people to do that for me so I can keep my technical skills as sharp as possible and remain relevant to my profession. I don’t believe I can offer technical advice unless I am living that technical advice.

Never sacrifice quality for speed!

2009-02-04T15:42:00.002-05:00

I just read two really good articles by Ron Jeffries and Uncle Bob about why sacrificing quality to go faster is always a bad idea.

This is very relevant to me now as I'm working with a sponsor now who thinks that by pushing harder and 'doing it in parallel' he can get everything he wants by the date he wants avoiding any hard decisions involving tradeoffs between scope and date. I have worked on many teams over the years that could speed up without sacrificing quality by focusing on the right things basically by emphasizing working software over high ceremony (pretty much straight from the Agile Manifesto). However the idea that you can get something out the door quickly without "wasting your time on quality" to me means that the person you're talking to doesn't understand quality. They think quality means those silly engineering things developers spend their time on rather than software that works the way you want and can be reliably enhanced later.

Somehow we as technologists need to do a better job of explaining to our business sponsors that "quality" is not a technical term!

How I refine a test spec while writing getting it to green

2009-01-09T15:10:00.002-05:00

In most of the examples I've read on TDD they show the Red-Green-Refactor cycle as

Write a failing test

Write just enough code to make the test pass

Refactor

Repeat

I absolutely do believe in this cycle and live it every day however I think there's a small detail that differs from what I do. When I move from step 1 to step 2, I keep my test window open and will switch back and forth between the test and code windows and refining the test while writing the code that makes the test pass. Its only as I write the code that I realize what calls I need to mock out in the test which also adds additional expectations to my test. I'm sure this is how most people work but I've never seen it written up so I'm going to try going through an example here.

Let's say I have a social networking application with a page showing friend requests where you can accept or reject each request. Something like this

Friend Requests:

Now I'm ready to implement an action in the Friend Controller that receives a posted form with a list of friend requests to accept and reject. I might first go into my friends_controller_spec.rb and write something like


describe FriendController do
  it 'should approve one friend requests' do
    post :update_requests, :approved_requests => ['123']
    response.should be_redirect
    response.should redirect_to(:back)
    flash[:notice].should  == "Friend requests approved."
  end
end

Now I have a failing test so I start implementing the code


class FriendController < ApplicationController
  def update_requests
    FriendRequest.approve(params[:approve_requests])
    flash[:notice] = "User access changes saved."
    redirect_to :back
  end
end

I'm done but my test is still failing...why? I haven't added the approve method to my FriendRequest model yet. I could go ahead and implement that method to get this test to pass but I don't want to do that because then this test would be testing more than just one unit. This is a unit test for the update_requests method of my FriendController and should be isolated from bugs in the rest of my application. Time to go back to the spec and add a mock to isolate this spec from the model and impose another expectation on my spec (that the method be called with the correct argument).


describe FriendController do
  it 'should approve one friend requests' do
    FriendRequest.expects(:approve).with(['123'])
    request.env['HTTP_REFERER'] = "http://some.site.com"
  
    post :update_requests, :approved_requests => ['123']
    response.should be_redirect
    response.should redirect_to(:back)
    flash[:notice].should  == "Friend requests approved."
  end
end

Now my spec passes and I can do my refactoring then go back to the beginning and write my next expectation.

The point of this example is that when I first wrote the spec I hadn't yet thought about how I would implement update_requests so did not yet know that I would need to mock FriendRequest.approve. I do this all the time and typically find myself go back and forth between the code and spec many times with the spec telling me what code I need to write and the code telling me how to refine the spec.

One last point (and a teaser for a future post). You have probably noticed that although my spec passes if I try to run the application it will not work because I still haven't written the FriendRequest.approve method. In a future post I hope to discuss how to interleave integration testing into this process to ensure that the classes we're writing in isolation also integrate so that the application satisfies its business function.

2008-12-11T13:56:00.002-05:00

I read an interesting post today by David Chelimsky who wrote RSpec A case against a case against mocking and stubbing. Its about mocking in testing and isolated vs integrated tests. I liked it all but what I particularly liked is how it describes the process of outside-in development.

To quote:

Write scenarios in plain text with cucumber (driven by user stories, organized in features).

Write the code for a step (or part of a step), run the feature, and observe the failure.

Optionally (yes, it depends – and why is the topic for another blog) drive out a view with a view spec. When I say “drive out,” I mean a very granular Red/Green/Refactor cycle that only involves this view, and only enough of this view to support satisfaction of the failing step in the cucumber feature.

Drive out a controller action using the same, granular Red/Green/Refactor cycle. And it may not be the entire controller action I think I want, covering all the cases I think I want. Just enough to support satisfaction of the failing step.

Drive out the parts of the model that I need to satisfy the failing step, using the same granular R/G/R process.

Run the cucumber feature and assess where I am.

In my projects I think I'm pretty good at following 3,4 &5 but I've struggled with steps 1 & 6. With the advent of cucumber I think now is the time to work on those other steps. I'll let you know how it goes in the New Year.

Bug/patch with rspec-rails and helper instance variables

2008-12-06T19:39:00.002-05:00

I finally got around to upgrading my version of rspec-rails from one that's almost a year old and came across an issue with the way implicit module inclusion is handled.

If you have a handler that uses memoization to cache some information in instance variables such as this (I'm not sure if this is a smell but my project has some examples of it)


module UsersHelper
 def all_users
   @users ||= User.find(:all)
 end
end

You would expect this spec to work


describe UsersHelper do
 it "should find all users" do
   User.expects(:find).with(:all).returns(result=mock)
   helper.all_users.should == result
 end
end

Sometimes it does but if any other spec has called helper.all_users previously it will fail as the memoized @users variable is not nil so the collection is reused.

I have submitted a lighthouse patch to rspec-rails that fixes the problem so hopefully they will agree to fix it soon. Until then patch is available on github in my fork or rspec-rails if you want to use it.

Knows which specs are slowing down my spec suite

2008-12-06T18:41:00.004-05:00

I often watch my specs run with the ........'s marching across my terminal and sometimes it seems to pause as it hits a particularly slow one. I figure there's some poorly written spec that's slowing my entire suite down and I should figure out what it is and fix it. But I rarely (if ever) take the time to track down the offender so never fix it. I just discovered that the clever folks writing rspec have taken away my excuse for laziness. I can tell it to report on the slowest specs by putting a line in my spec.opts file telling it to use the ProfileFormatter (see the first line below)


--format profile
--colour
--loadby mtime
--reverse

For example running the specs on my current project I get this output.


Profiling enabled.
........(a bunch more dots)

Top 10 slowest examples:
2.4908950 WikiContent should find all activity for a user
0.8061750 WikiContent should find recent activity for a user in last 15 days
0.2438460 WikiContent::Version should allow limit on results from finding recent activity
0.1821710 User should have a list of owned content
0.1700670 WikiContent should find results if access type is not same as role of the user but role of user is 'Leader'
0.1593900 ContentsController allows PUT request to 'update' action for 'member' role
0.1514150 Search::SearchResponse gets the wiki content id of an asset from the xml response
0.1470310 Content should always allow leaders to see all pages
0.1394920 Page should delete all comments when destroyed
0.1370330 ContentsController should save content and create the uploaded asset on successful POST to the 'create' action

Over the next few days I will find time to take a look at 'WikiContent find all activity' and 'WikiContent find recent activity' specs as they are the two clear outliers.

I now have no excuse to not optimize my slowest specs because I didn't know which they were!

Applications usually use their own data

2008-11-25T09:31:00.003-05:00

I just read a very interesting post by Martin Fowler called Database Thaw. He talks about various database technologies and what the future might hold for object or relational databases but what caught my interest was his discussion of application and database integration patterns.

He says "For many organizations today, the primary pattern for integration is Shared Database Integration - where multiple applications are integrated by all using a common database." This is certainly true where I work. I've even heard it said that we have one gigantic application since all (or most) share the same underlying database so cannot be updated independently.

He also talks about Integration Databases which store data for multiple applications and Application Databases which are controlled and accessed by a single application.

Historically my company has moved toward the integration database pattern often building complex service layers to enable shared access to the data on the assumption that if you build it they will come the data would be reused if it was easy enough. However I believe that its only a very small subset of our data that is truly common and many applications want to use. For that small subset a shared database with a service layer is probably a good design. The vast majority is private to an application and by designing it into a separate application database can evolve to best meet the needs of the application without concern for a more general service that in our case is more hindrance than benefit.

One important thing to remember about an application database is that the data exists to meet the needs of the application so a design that allows the database to best meet the application's needs will be to everyone's benefit. Finally database technologies are very complex and do a lot so it is probably wise to have a good database developer on your team to make sure you're using the database appropriately but they should be a part of your team not on a separate database team as I've seen in the past - particularly on Java/Oracle projects I've been a part of.

Thoughts?