If you are using cucumber to test a multilingual app, you should check out this post for a step definition that you can use to test multilingual strings. However, most likely you would be using variables in your translation strings. Assuming you have pickle installed (if you don’t you should), you can use the step and step definition below to test strings with variable interpolation

Just like pickle, you can add any number of variables.

aka .. can’t throw hardware at it

Just when I decided to focus fulltime on my new startup SupportBee, our first product Muziboo decided to grow (no complaints though). We went from about 10k visits a day to about 30k visits a day in a month’s time. Muziboo was my first web programming experience and in the sprit of a real startup, I always postponed worrying about scaling. However, the cracks showed through this time. Here is what we observed

• Slow page load
• High CPU load and disk I/O
• MySQL consuming very high CPU consistently

Below is a log of what we did to make the site faster

Better Server Monitoring and Logging

You can’t fix what you can’t measure. We researched a bit and installed munin. Apart from the standard plugins (disk, cpu, network) we setup plugins for beanstalk, rails and nginx. This gave us some pretty charts to understand how the system was behaving. Here is an example of what munin charts looks like

Not only can you see the current state of the system, you can also see data for past weeks/months/years. If you are not excited by installing munin, you can look at hosted solution like Server Density

MySQL slow query log for missing indexes

Rails does not (cannot) add indexes to your database. Rails does not (cannot) add indexes to your database automatically. You will have to figure out what indexes to add based on your queries. This is not a problem for small tables (hundreds or thousands of rows) but its a problem with bigger tables. Fortunately, its easy to find out queries that are not using the index. Apart from listing slow queries, MySQL can list all the queries that don’t use the index in slow query log. You can enable it by having this directive in your mysql configuration file

log-queries-not-using-indexes

We enabled this option and found out that we had a bunch of index less queries. As we added these indexes, mysql’s cpu usage started going down bringing the system to a much better shape. You can use mysql’s explain statement to understand your queries better and add indexes. Its a good idea to have explain’s output in your rails’ log during development. I found this snippet that does it for you

unless RAILS_ENV == 'production'
      module ActiveRecord
        module ConnectionAdapters
          class MysqlAdapter < AbstractAdapter             def select_with_explain(sql, name = nil)               explanation = execute_with_disable_logging('EXPLAIN ' + sql)               e = explanation.all_hashes.first               exp = e.collect{|k,v| " | #{k}: #{v} "}.join               log(exp, 'Explain')               select_without_explain(sql, name)             end             def execute_with_disable_logging(sql, name = nil) #:nodoc:               #Run a query without logging               @connection.query(sql)             rescue ActiveRecord::StatementInvalid => exception
              if exception.message.split(":").first =~ /Packets out of order/
                raise ActiveRecord::StatementInvalid, "'Packets out of order' error was received from the database. Please update your mysql bindings (gem install mysql) and read http://dev.mysql.com/doc/mysql/en/password-hashing.html for more information.  If you're on Windows, use the Instant Rails installer to get the updated mysql bindings."
              else
                raise
              end
            end
            alias_method_chain :select, :explain
          end
        end
      end
    end

Put it in your config/initializers directory as mysql_explain.rb and restart your server. Now you should see output from explain in your logs before your sql queries. One point to note is that throwing memcache is not a substitute for having faster queries.

Beware of MySQL’s ORDER BY rand()

Even after adding indexes, one of our queries was not using them. On investigating further I found out that MySQL’s order by rand() is evil. I won’t go into the specifics now but you can read up more on stackoverflow. In short, order by rand() will always do a full table scan. Avoid it.

MySQLDump haz all the locks

Moving from mongrel to passenger

Muziboo was started when the world was just moving over from fastcgi to mongrel. Hence we were using a bunch of  mongrels monitored by monit. However, every time you want to add more mongrels, you have to change your monit configuration file to start and monitor more mongrels. Also there is no easy way to reduce the number of mongrels when under less load. Hence we decided to move over to passenger. In passenger, you just have to change the number of max instances in the nginx configuration files and more app instances will be started under load. When the load is lesser, these instances are shut down. All this is automatically done for you.

Taming I/O during rsync

We use rsync to backup uploaded files uploaded by Muziboo’s user. We have about 1 TB of data and even though the backup is incremental, there is still considerable I/O to check which files have to be moved. To make sure that rsync does not overwhelm everything else, we used the following options

nice -20 ionice -c2 -n7 /usr/bin/rsync -avz -e ssh --times --size-only --stats ......

Nice and IONice help you keep the CPU and disk usage low. You can also use it for other cron/background jobs that you run.

With these changes in place, we have been able to accomodate all the growth on a single server (Quadcore + 12 GB RAM). I think with more growth we may have slowers mysql writes and at that point we may need to move to a separate db server.

Read some interesting comments on this post on Hacker News

Hosting a git repo yourself is a very simple exercise. If you don’t need (or don’t want to pay for) features offered by github, you can setup your server to host your git repo. All you need is a server with ssh access. Here are the steps that you need to follow

Login to your server


ssh tom@example.com
tom@example.com$ mkdir my_repo.git
tom@example.com$ cd my_repo.git
tom@example.com$ git init


On you local machine, where you have your code


$ cd my_code # cd to your code
$ git init
$ git add . # Adds everything. Use .gitignore to ignore some files
$ git remote add origin tom@example.com:my_repo.git
$ git push origin master


Now you can clone tom@example.com:my_repo.git on any number of machines. To not share password, put your public keys on tom@example.com

This approach gives you a basic git setup with ssh access (no http access). These steps should work on shared hosting too (as long as you have git available)

If you are using flexmock for testing your rails code and you are mocking an active record model using flexmock(:model, Model), you may get an error like

undefined method `destroyed?'

The problem is that flexmock defines a lot of methods/attributes like id and new_instance? etc but does not define a destroyed? method that rails now expects. Fortunately, this is easy to fix. Create a file ‘config/initializers/flexmock_extensions.rb’ and put this in

class FlexMock
  class MockContainerHelper
    def add_model_methods(mock, model_class, id)
      container = mock.flexmock_container
 
      mock_errors = container.flexmock("errors")
      mock_errors.should_receive(:count).and_return(0).by_default
      mock_errors.should_receive(:full_messages).and_return([]).by_default
 
      mock.should_receive(:id).and_return(id).by_default
      mock.should_receive(:to_params).and_return(id.to_s).by_default
      mock.should_receive(:new_record?).and_return(false).by_default
      mock.should_receive(:class).and_return(model_class).by_default
      mock.should_receive(:errors).and_return(mock_errors).by_default
      mock.should_receive(:destroyed?).and_return(false).by_default
 
      # HACK: Ruby 1.9 needs the following lambda so that model_class
      # is correctly bound below.
      lambda { }
      mock.should_receive(:is_a?).with(any).and_return { |other|
        other == model_class
      }.by_default
      mock.should_receive(:instance_of?).with(any).and_return { |other|
        other == model_class
      }.by_default
      mock.should_receive(:kind_of?).with(any).and_return { |other|
        model_class.ancestors.include?(other)
      }.by_default
    end
  end
end

Basically, we are defining the method destroyed? and returning false everytime. Now your tests should work

Pickle is a great add on to cucumber as it provides steps for often used scenario. There is a step provided for verifying the association size of an active record instance. Therefore you can write something like

Then the user: “Joe” should have 1 friends

and it will check if joe.friends.size == 0. However this does not check the counter cache (if you have one). I modified it a bit to verify counter cache (changes in file features/step_definitions/pickle_steps.rb)

# assert size of association
Then /^#{capture_model} should have (\d+) (\w+)$/ do |name, size, association|
  if(model!(name).respond_to?("#{association}_count"))
    model!(name).send("#{association}_count").should == size.to_i
  else
    model!(name).send(association).size.should == size.to_i
  end
end

This assumes that you counter cache is named association_cache which is the convention in rails’ world.

YSlow is a popular tool for measuring and tweaking site performance. The tool recommends that Javascript be put at the bottom of the page (since javascript blocks the page until its loaded). However one of the challenges of doing this is that you cannot call any javascript function (that you have defined in your js files) before the files are loaded. This can be a serious limitation if you are doing a dynamic website where the header and footer (which is where the script tags are inserted) is static and all the dynamic content is in the middle of the page. If you have used any Ruby on Rails or similar framework, you know what I mean. Basically, its hard to put different code in the footer for different pages.

Lets take an example from Muziboo. I have a function in javascript to initialize the flash player Muziboo.Player.embedSingle(). If my script tags (for including the js) are in the header, I can make this call anywhere on the page and it will work. However if the javascript files are loaded last, this call has to be made after loading the javascript.

However you can overcome this limitation by using a javascript queue that holds the function calls till the external javascript files are loaded. Once the javascript is loaded, it can look at the queue and process the messages. So something like this would work

This idea is not new and is used by Google Analytics’ async tracking code. Lets see how to implement such a queue in javascript (using the prototype.js javascript library)

The queue is nothing but a simple javascript array. The process message simply pops out each message, executes it and then changes the push method on _muzq object to execute the message instead of storing it (since now the js is loaded and there is no need to wait). I am still trying to find out a better way than to use eval but this implementation works.

I just pushed some changes to my fork of the blackbook address book importer gem. Here is a list of changes

• The gem now works with the latest version of mechanize and nokogiri
• Myspace and Gmail importer working perfectly. Myspace had changed its url scheme and also address book rendering stuff so the gem was broken for a while

Sadly the yahoo address book importer is not working as yahoo as added a captcha before csv download. I will try to use an alternative view and fix the issue sometime soon.  I have not tried aol and hotmail import.

If you need any help, please leave a comment here.

Yes … as unrealistic as it may sound, yesterday, I ran into an issue where something worked in Internet explorer but not in Firefox. I was trying to call an actionscript functions on a swf file from javascript. This is a fairly standard thing and is well documented on the web. I followed all the steps (registering a callback and embedding the swf using swfobject library) and it worked perfectly in IE. I could call my actionscript functions using javascript in IE but in firefox it would occasionally fail and I would see the following two messages in firebug

uncaught exception: [Exception... "Failure" nsresult: "0x80004005 (NS_ERROR_FAILURE)" location: "JS frame :: http://localhost:9001/javascripts/swfobject.js?1259133537 :: anonymous :: line 4" data: no]

$(’movieName’).asFunc() undefined method

where movieName is the id of the swf file and asFunc is the function defined in actionscript using ExternalInterface.addCallback. After a lot of googling, I ran into this thread which explained the basic problem. Since my swf file was hidden in the footer of the page, the callbacks worked after I had once scrolled down and seen the footer. Yes as crazy as it sounds, that was really the problem.  Firefox wants you to see the swf once before you make the function call.  To make sure that the 1×1 sized swf file was always visible no matter where you are in the page, I added the following css rule

#movieName {
position: fixed;
top: 0;
left: 0;
}

This rule makes sure that no matter how much you scroll up or scroll down, the swf file is always at the same place (in this case, just at the starting of the page, relative to the browser window). You can read more about fixed positioning here. I am sure there must be a better explanation/solution for this problem but I have tried my approach and it atleast works. If you find a better solution, please post as a comment and I will update the post.

File column is a great plugin by Sebastian Kanthak for uploading images (and other files) in a rails’ site. The plugin has rmagick integration and can create thumbnails automatically for you. We use it for Muziboo and have been pretty happy with it (yes despite the rmagick dependency). We recently had to update the thumbnail dimensions for a few of our models. Its fairly simple to change the thumbnail sizes in the model and all new uploads are resized to new dimensions. However, the existing images are not automatically changed for you . Thankfully, on digging deeper into the code I found that there is pretty simple way to accomplish that. Lets say you have a user model (with a file_column field photo) and you want to update the thumbnails for all existing images. You just need to simply run the following commands

User.find(:all,:conditions => "photo IS NOT NULL").each do |u|
  u.photo = File.open(u.photo)
  u.save
end

Ofcourse if you have a lot of users, you may wanna fetch them in batches. Also image resizing is a pretty CPU intensive operation so you may wanna watch out for that. Once the operation is over, you will have updated thumbnails for all your previously uploaded images.

Blackbook gem can be used to fetch contacts from gmail, yahoo and hotmail address books. However I recently found out that it would throw up a ‘Must be authenticated to access contacts’ error everytime I tried to import gmail contacts. On digging deeper, I found out that google has changed the contents of its cookie and blackbook uses this cookie to make sure that the user is authenticated. To fix the issue, you just need to make a one line change in the function scrape_contacts in the file lib/blackbook/importer/gmail.rb.

Change

unless agent.cookies.find{|c| c.name == 'GAUSR'  &&
  c.value == "mail:#{options[:username]}"}
raise( Blackbook::BadCredentialsError, "Must be authenticated to access contacts." )

to

unless agent.cookies.find{|c| c.name == 'GAUSR' &&
  (c.value =~ /mail.*:#{options[:username]}/)!=nil}
raise( Blackbook::BadCredentialsError, "Must be authenticated to access contacts." )

If you want to install the updated gem instead of making the change, you can find it in my blackbook fork. My fork also includes the myspace address book import