1. Migrations with no schema specifics

Your data model is the core of your application. Without schema constraints, your data will slowly corrode due to bugs in your codebase until you can’t depend on any fields being populated. Here’s a Contact schema:

  create_table "contacts" do |t|
    t.integer  "user_id"
    t.string   "name"
    t.string   "phone"
    t.string   "email"
  end

What is required? Presumably a Contact must belong_to a User and have a name — use database constraints to guarantee this. By adding :null => false, we ensure that the model is always consistent even if we have bugs in our validation because the database will not allow a model to be saved if it fails those constraints.

  create_table "contacts" do |t|
    t.integer  "user_id", :null => false
    t.string   "name", :null => false
    t.string   "phone"
    t.string   "email"
  end

Bonus points: use :limit => N to size your string columns appropriately. Strings default to 255 characters and phone probably doesn’t need to be that big, does it?

2. Object-Oriented Programming

Most Rails developers do not write object-oriented Ruby code. They write MVC-oriented Ruby code by putting models and controllers in the expected locations. Most will add utility modules with class-methods in lib/, but that’s it. It takes 2-3 years before developers realize: “Rails is just Ruby. I can create simple objects and compose them in ways that Rails does not explicitly endorse!”

Bonus points: introduce facades for any 3rd-party services you call. Provide a mock facade for use in your tests so that you don’t actually call the 3rd party service in your test suite.

3. Concatenating HTML in helpers

If you are creating helper methods, kudos, at least you trying to keep your view layer clean. But developers often don’t know the basics of creating tags within helpers, leading to messy string concatenation or interpolation:

str = "<li class='vehicle_list'> "
str += link_to("#{vehicle.title.upcase} Sale", show_all_styles_path(vehicle.id, vehicle.url_title))
str += " </li>"
str.html_safe

Yikes, it’s ugly and can easily lead to XSS security holes! content_tag is your friend.

content_tag :li, :class => 'vehicle_list' do
  link_to("#{vehicle.title.upcase} Sale", show_all_styles_path(vehicle.id, vehicle.url_title))
end

Bonus points: start introducing helper methods that take blocks. Nested blocks are a natural fit when generating nested HTML.

4. Giant queries loading everything into memory

You need to fix some data so you’ll just iterate through it all and fix it, right?

User.has_purchased(true).each do |customer|
  customer.grant_role(:customer)
end

You have an ecommerce site with a million customers. Let’s say each User object takes 500 bytes. This code will take 500MB of memory at runtime! Better:

User.has_purchased(true).find_each do |customer|
  customer.grant_role(:customer)
end

find_each uses find_in_batches to pull in 1000 records at a time, dramatically lowering the runtime memory requirements.

Bonus points: use update_all or raw SQL to perform the mass update. SQL takes time to learn well but the benefits are even more tremendous: you’ll see a 100x improvement in the performance.

5. Code review!

I’m guessing you are using GitHub and I’m also guessing you aren’t using pull requests. If you spend a day or two building a feature, do it on a branch and send a pull request. Your team will be able to review your code, offer suggestions for improvement and possible edge cases that you didn’t consider. I guarantee your code will be higher quality for it. We’ve switched to using pull requests for 90% of our changes at TheClymb and it’s been a 100% positive experience.

Bonus points: Don’t merge pull requests without tests for at least the happy path. Testing is invaluable to keep your application stable and your sleep peaceful.

Did I miss any common issues? Leave a comment and let me know!

Update: use find_each rather than find_in_batches, thanks readers!

• 668 GitHub watchers
• 171 issues, 11 open
• 7 commercial licenses purchased!
• 54 pull requests with 30+ different developers!
• 1 EngineYard podcast interview
• 1 RailsConf lightning talk by @jwo
• 1 South African Ruby group talk on Sidekiq!
• 1 new license (LGPL rather than GPL)
• 0 locks in the multithreaded codebase

I consider that a success; I’ve never had a project grow this fast with just my own promotion and community word of mouth. TheClymb switched to Sidekiq last week and our biggest problem so far has been that Sidekiq can be too parallel and crush servers with traffic; we’ve had to rewrite some jobs to be serial!

My goals remain the same:

• Provide the easiest and best supported queueing system for Ruby.
• Be the first Rubygem people mention or consider when choosing a queueing system
• Improve Ruby’s overall efficiency and perceived performance through multi-threading.
• Evangelize multi-threaded infrastructure written with actor abstractions as relatively straightforward for knowledgable developers to build. Celluloid continues to be a huge asset to Sidekiq’s ease of development and stability.

Thank you all for your support so far! Thank you especially to EngineYard, Tony Arcieri and the early adopters that have made hacking on Sidekiq an exciting adventure rather than a lonely chore.

• You have no idea what your coworkers are working on right now.
• You don’t know what you will be working on next week.

Does this sound like your team? The problem is almost always the same: lack of communication. The point of development process is simple: ensure team communication. The simplest effective development process for your team ensures that everyone is in sync about the present and near-future goals for the team.

Process gets a bad reputation because often meetings are used as a substitute for process. Since lack of communication is the problem, we should just throw everyone in a room together to talk, right? Good, high-bandwidth meetings can be very useful but just as important are tools which allow the team to communicate on their own schedule (think issue trackers, code review tools, group chat, a pinboard with index cards, etc). Your process should be a mixture of synchronous, high-bandwidth meetings and asynchronous, low-bandwidth tools. There’s no right mixture but your process must have two mandatory properties:

1. Everyone must sign off on it as “the way”.
2. Everyone must actually follow it.

I generally try for two team meetings per week: a Monday morning sync about upcoming work for this week and next week and Friday afternoon demo of work accomplished that week (the demo almost always finds bugs and issues to be fixed next week). Monday morning kicks the work week into high gear and Friday afternoon winds down the work so everyone can enjoy their weekend.

Great, which process should you use? Agile, XP, Scrum, Kanban? I treat them like web frameworks: pick one, learn it well and modify it as necessary for your own needs. The one booklet that I keep going back to year after year is this booklet on Scrum and XP practices offered by InfoQ. It’s a very practical, quick read and low tech enough that you can implement it in 2007 or 2017.

A suggestion was to just set this in an initializer:

Mysql2::Client::CHARSET_MAP['latin1'] = Encoding::UTF_8


This will solve your problems in Ruby but will not solve your problems in the database: MySQL will still sort and compare strings thinking they are latin1 and thus do so incorrectly. Here’s an example:

CREATE TABLE names_latin1 (name varchar(32) character SET latin1);
INSERT INTO names_latin1 VALUES ('Martin Strauße');
INSERT INTO names_latin1 VALUES ('Martin Straure');
INSERT INTO names_latin1 VALUES ('Martin Strausse');
INSERT INTO names_latin1 VALUES ('Martin Straute');
 
CREATE TABLE names_utf8 (name varchar(32) character SET utf8);
INSERT INTO names_utf8 VALUES ('Martin Strauße');
INSERT INTO names_utf8 VALUES ('Martin Straure');
INSERT INTO names_utf8 VALUES ('Martin Strausse');
INSERT INTO names_utf8 VALUES ('Martin Straute');

I’m not a linguist but to the best of my knowledge the German ß is essentially “ss”. When we ask MySQL to sort our names, you can see that the UTF-8 results put the ß character between “r” and “ss” but the Latin1 results don’t. If a German were to see this, they would be enraged due to your culturally insensitive code!

> select * from names_latin1 order by name;
+-----------------+
| name |
+-----------------+
| Martin Straure |
| Martin Strausse |
| Martin Straute |
| Martin Strauße |
+-----------------+


> select * from names_utf8 order by name;
+-----------------+
| name |
+-----------------+
| Martin Straure |
| Martin Strauße |
| Martin Strausse |
| Martin Straute |
+-----------------+


We need to update the CHARACTER SET without doing any conversion of the data. This is simple to do: you convert the columns to a blob format and then convert them back to a string format with the proper encoding declared; MySQL will not do any conversion of raw binary data. For example:

ALTER TABLE categories CHARACTER SET utf8 COLLATE utf8_unicode_ci, CHANGE title title VARBINARY(255)
ALTER TABLE categories CHANGE title title VARCHAR(255) CHARACTER SET utf8 COLLATE utf8_unicode_ci

With this in mind, I wrote a rake task to convert our application’s database. Here’s the full script in a Github Gist. You’ll need to run it with a DOIT parameter to actually make the changes otherwise it will just print the SQL it will run to the terminal. The script will take a long time for large databases since it has to ALTER TABLE, which means MySQL will write out the table to disk in full; the script does minimize the number of ALTER TABLEs it runs to two per table.

rake convert_to_utf8 DOIT=1


So please think of the Germans and the rest of our international friends: converting the character set of your database to the proper value is important to get correct sorting of results.

  begin
    do_something
  rescue => ex
    # do something with the error
  end

This will catch all StandardErrors and its subclasses. A frequent issue I see in Ruby code is this:

  begin
    do_something
  rescue Exception => ex
    # Apparently I REALLY want to make sure this block 
    # won't interfere will the outside code.
  end

This is a BAD, BAD idea, dear reader, and here’s why. Ruby uses Exceptions for other things than your application code errors. One example is the Interrupt class which is a SignalException. Ruby sends this Exception to all threads so that when the process gets an INT/Ctrl-C signal, all the threads will unwind and the process will shutdown. If you rescue Exception, you will potentially catch this exception and ignore it, making your thread and process an unkillable computing zombie. Your only choice will be to pull out your kill -9 shotgun and aim for the head.

Here’s an example of a Ruby script you cannot shutdown gracefully. Run it and you’ll see exactly the behavior I’ve described.

while true
  begin
    sleep 5
    puts 'ping'
  rescue Exception => ex
    puts "Mmmmm, brains"
  end
end

So remember, your application errors should be subclasses of StandardError and if you want to catch everything, just stick will plain old “rescue => ex”. Your application will behave better for it.

Here’s two easy ways to clean out that table quickly.

1) Use ALTER IGNORE on MySQL 5.1+

MySQL will allow you to create a unique index on a table with duplicate records with its IGNORE SQL extension:

ALTER IGNORE TABLE 'SHIPMENTS' ADD UNIQUE INDEX (CART_ID, TRACKING_NUMBER)

Duplicates will be deleted.

2) Recreate the table with GROUP BY

    execute 'CREATE TABLE shipments_deduped like shipments;'
    execute 'INSERT shipments_deduped SELECT * FROM shipments GROUP BY cart_id, tracking_number;'
    execute 'RENAME TABLE shipments TO shipments_with_dupes;'
    execute 'RENAME TABLE shipments_deduped TO shipments;'
    add_index :shipments, [:cart_id, :tracking_number], :unique => true
    execute 'DROP TABLE shipments_with_dupes;'

Recreating the table is much, much faster than trying to delete the records in the existing table and doesn’t lock the existing table, making your application downtime minimal.

• 462 watchers
• 66 issues opened so far.
• Three contributors earned commit rights.
• One commercial license purchased!
• Approximately one pull request per day, far better than any of my previous projects.

I’ve chatted with two different people running Sidekiq in production and their experience has been the same:

• “I’m expecting to save $2000/month in worker boxes moving from delayed_job to Sidekiq.”
• “I’ve gone from 160 worker dynos to 10 dynos, saving over $5000/month.”

Astounding, all in the first month for a one man OSS project. If you use Sidekiq in production, please email me (mperham AT gmail) and tell me your success story!

So yeah, messaging and I have a history but I think that’s because Ruby still doesn’t have a simple, scalable option for message processing. Sidekiq is my attempt to solve that need.

Simple

Sidekiq integrates tightly with Rails 3 so your application’s worker code is nearby and easy for Sidekiq to find and load. I’d like to support plain Ruby applications but I need a replacement for the nice integration Rails 3 gives me.

Efficient

Sidekiq uses multiple threads so you can process hundreds of messages in parallel if you desire without the overhead of hundreds of processes. Internally Sidekiq uses the Celluloid actor library to make threading easier and safer to manage.

Developer Friendly

I’ve tried to make Sidekiq developer friendly in two ways:

• Resque-compatible – I love conventions, why reinvent the wheel when you can stick closely to what someone else has done and leverage that work? I aim to make Sidekiq easy to integrate into an existing Resque project and monitorable via resque-ui.
• Middleware – Sidekiq pulls in a middleware API similar to Rack for running code before/after/around a processed message and ships with middleware that helps with ActiveRecord and Airbrake.

Licensing

Sidekiq is an experiment in one regard: licensing. I’ve decided to license Sidekiq under the GPLv3 for free or a commercial license for a $50 pledge at my Pledgie page. During my time as a Ruby consultant, $50 would get you less than 20 minutes of my time. I hope you think this is a reasonable price.

Please let me know how you like Sidekiq – I love hearing stories about how people are using it.

*/5 * * * * osascript -e 'tell application "System Events" to if (processes whose name is "iChat") exists then tell application "iChat" to log in'

This uses AppleScript to tell iChat to log in every 5 minutes. Now if the network drops, you’ll only be disconnected a few minutes.