Dmytro Shteflyuk's Home

Advanced Capistrano usage

Dmytro Shteflyuk — Wed, 23 Jun 2010 19:32:06 +0000

One of the most important parts of a development process is an application deployment. There are many tools developed to make this process easy and painless: from the simple inploy to a complex all-in-one chef-based solutions. My tool of choice is Capistrano, simple and incredibly flexible piece of software. Today I’m going to talk about some advanced Capistrano usage scenarios.

1. Graceful Passenger restarts

Passenger user guide contains a simple Capistrano recipe for application server restarts. It works pretty well in almost all the cases, but there is a huge problem when you use a multi-server setup: it restarts all Passengers at the same time, so all client requests will hang (or even drop) during the time needed to start your application. The simplest solution is to restart Passengers one by one with some shift in time (for example, 15 seconds — choose this value based on how long it take to get your application up and running), so at any given moment only one of your application servers will be unavailable. In this case Haproxy (you use it, don’t you?) won’t send any requests to the restarting server, and most of your users will continue their work without any troubles.

Let me show you how we could achieve this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

namespace :deploy do
desc <<-EOF
Graceful passengers restarts. By default, it restarts \
passengers on servers with a 15 interval, but \
this delay could be changed with the smart_restart_delay \
variable (in seconds). If you specify 0, the restart will be \
performed on all your servers immediately.

cap production deploy:smart_restart

Yet another way to restart passenger immediately everywhere is \
to specify NOW environment variable:

NOW=1 cap production deploy:smart_restart
EOF
task :smart_restart, :roles => :app do
delay = fetch(:smart_restart_delay, 15).to_i
delay = 0 if ENV['NOW']

if delay <= 0
logger.debug "Restarting passenger"
run "touch #{shared_path}/restart.txt"
else
logger.debug "Greaseful passengers restart with #{delay} seconds delay"
parallel(:roles => :app, :pty => true, :shell => false) do |session|
find_servers(:roles => :app).each_with_index do |server, idx|
# Calculating restart delay for this server
sleep_time = idx * delay
time_window = sleep_time > 0 ? "after #{sleep_time} seconds delay" : 'immediately'

# Restart command sleeps a given number of seconds and the touches the restart.txt file
touch_cmd = sleep_time > 0 ? "sleep #{sleep_time} && " : ''
touch_cmd << "touch #{shared_path}/restart.txt && echo [`date`] Restarted Passenger #{time_window}"
restart_cmd = "nohup sh -c '(#{touch_cmd}) &' 2>&1 >> #{current_release}/log/restart.log"

# Run restart command on a given server
session.when "server.host == '#{server.host}'", restart_cmd
end
end
end
end
end

The trickiest part is at the lines 25-26. There we use the parallel method to run all our commands in parallel, but it has a great limitation: there is no way to substitute command parts on the fly based on server where the command is going to be executed. So instead we are building a condition for each server in the :app role, and calculate time shift based on its index.

Sometimes it’s necessary to perform an immediate restart (for example, a database migration breaks old code). We use an environment variable to do this: cap production deploy:restart NOW=1

2. Generating deployment stages on the fly in multi-stage environments

In Scribd we use a single QA box for testing, with multiple configured applications on it. The only difference between corresponding deployment scripts is an application path (e.g. /var/www/apps/qa/01, /var/www/apps/qa/02, etc.) So how do we keep them DRY? First we have created a single deployment stage called qa, and deployed with cap qa deploy QAID=1. Works, but smells bad. Today’s version is much more elegant, but it took some effort to implement:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

(1..10).each do |idx|
qid = '%02d' % idx
name = "qa#{qid}"
stages << name

desc "Set the target stage to `#{name}'."
task(name) do
location = fetch(:stage_dir, "config/deploy")
set :stage, :qa
set :qa_id, qid
load "#{location}/qa"
end
end
# This is a tricky part. We need to re-define [cci]multistage:ensure[/cci] callback
# (which is simply raises an exception), so it will not be executed for our newly
# defined stages.
if callbacks[:start]
idx = callbacks[:start].index { |callback| callback.source == 'multistage:ensure' }
callbacks[:start].delete_at(idx)
on :start, 'multistage:ensure', :except => stages + ['multistage:prepare']
end

In the qa stage script we set the :deploy_to variable from :qa_id. Now we can deploy using cap qa01 deploy. I leave the implementation of cap qa deploy, which selects a free QA box and then performs deploy there, up to you (check the Hint 4: Deploy locks explaining how to prevent stealing QA boxes by overwriting deployments using a simple locks technique).

3. Campfire notifications

This is the most straightforward and easy to implement feature:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

begin
gem 'tinder', '>= 1.4.0'
require 'tinder'
rescue Gem::LoadError => e
puts "Load error: #{e}"
abort "Please update tinder, your version is out of date: 'gem install tinder -v 1.4.0'"
end

namespace :campfire do
desc "Send a message to the campfire chat room"
task :snitch do
campfire = Tinder::Campfire.new 'SUBDOMAIN', :ssl => true, :token => 'YOUR_TOKEN'
room = campfire.find_room_by_name 'YOUR ROOM'
snitch_message = fetch(:snitch_message) { ENV['MESSAGE'] || abort('Capfire snitch message is missing. Use set :snitch_message, "Your message"') }
room.speak(snitch_message)
end

desc "Send a message to the campfire chat room about the deploy start"
task :snitch_begin do
set :snitch_message, "BEGIN DEPLOY [#{stage.upcase}]: #{ENV['USER']}, #{branch}/#{real_revision[0, 7]} to #{deploy_to}"
snitch
end

desc "Send a message to the campfire chat room about the deploy end"
task :snitch_end do
set :snitch_message, "END DEPLOY [#{stage.upcase}]: #{ENV['USER']}, #{branch}/#{real_revision[0, 7]} to #{deploy_to}"
snitch
end

desc "Send a message to the campfire chat roob about the rollback"
task :snitch_rollback do
set :snitch_message, "ROLLBACK [#{stage.upcase}]: #{ENV['USER']}, #{latest_revision[0, 7]} to #{previous_revision[0, 7]} on #{deploy_to}"
snitch
end
end

#############################################################
# Hooks
#############################################################

before :deploy do
campfire.snitch_begin unless ENV['QUIET'].to_i > 0
end

after :deploy do
campfire.snitch_end unless ENV['QUIET'].to_i > 0
end

before 'deploy:rollback', 'campfire:snitch_rollback'

To deploy without notifications use cap production deploy QUIET=1 (but be careful, usually it’s not a good idea).

4. Deploy locks

Sometimes it’s useful to lock deploys to a specific stage. The most common reason is that you pushed a heavy migration to the master and want to run it yourself, before the actual deploy, or performing some production servers maintenance and want to be sure nobody will interfere with your work.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

namespace :deploy do
desc "Prevent other people from deploying to this environment"
task :lock, :roles => :web do
check_lock
msg = ENV['MESSAGE'] || ENV['MSG'] ||
fetch(:lock_message, 'Default lock message. Use MSG=msg to customize it')
timestamp = Time.now.strftime("%m/%d/%Y %H:%M:%S %Z")
lock_message = "Deploys locked by #{ENV['USER']} at #{timestamp}: #{msg}"
put lock_message, "#{shared_path}/system/lock.txt", :mode => 0644
end

desc "Check if deploys are OK here or if someone has locked down deploys"
task :check_lock, :roles => :web do
# We use echo in the end to reset exit code when lock file is missing
# (without it deployment will fail on this command — not exactly what we expected)
data = capture("cat #{shared_path}/system/lock.txt 2>/dev/null;echo").to_s.strip

if data != '' and !(data =~ /^Deploys locked by #{ENV['USER']}/)
logger.info "\e[0;31;1mATTENTION:\e[0m #{data}"
if ENV['FORCE']
logger.info "\e[0;33;1mWARNING:\e[0m You have forced the deploy"
else
abort 'Deploys are locked on this machine'
end
end
end

desc "Remove the deploy lock"
task :unlock, :roles => :web do
run "rm -f #{shared_path}/system/lock.txt"
end
end

before :deploy, :roles => :web do
deploy.check_lock
end

Now use can use cap production deploy:lock MSG="Running heavy migrations".

5. Generating servers list on the fly

Another interesting and sometimes pretty useful task is to fetch the list of servers for a deploy from some external service. For example, you have an application cloud, and do not want to change your deployment script every time you add, remove, or disable a node. Well, I have a good news for you: it’s easy!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

namespace :deploy do
task :set_nodes_from_remote_resource do
# Here you will fetch the list of servers from somewhere
nodes = %w(app01 app02 app03)

# Clear servers lists of :app and :db roles
roles[:app].clear
roles[:db].clear

# Fill :app role servers lists
nodes.each do |node|
parent.role :app, node
end

# First server in list is a primary node and db node (to run migrations)
primary = roles[:app].first
primary.options[:primary] = true
roles[:db].push(primary)

# Show information in log about where we are going to deploy to
nodes_to_deploy = roles[:app].servers.map do |server|
opts = server.options[:primary] ? ' (primary, db)' : ''
"#{server.host}#{opts}"
end.join(', ')

logger.info "Deploying to #{nodes_to_deploy}"
end
end

on :start, 'deploy:set_nodes_from_remote_resource'

When you run cap production deploy, something like this will be printed to your console:

1
2
3

triggering start callbacks for `deploy'
* executing `deploy:set_nodes_from_remote_resource'
** Deploying to app01 (primary, db), app02, app03

That’s all for today. Deployment automation could be a really tricky task, but with a right tool it turns out to be a pleasure. Do you have any questions, suggestions, or some other example deployment recipes? Do me a favor, put them in a comment! Also I have (surprise!) a Twitter account @kpumuk, and you simply must follow me there. No excuses!

Submitting a patch to the Open Source project: composite_primary_keys

Dmytro Shteflyuk — Thu, 17 Dec 2009 10:28:12 +0000

Not so far ago I have found a weird bug in the Open Source Ruby gem called composite_primary_keys, occurred when you specify :primary_key option for has_one or has_many association. There are two ways to get it fixed: submit an issue and wait till someone will work out this problem or fix it by yourself and then pull request to get the patch merged into the core. This is a great library and I use it in almost all my project, so I decided to help the author and fix this bug by myself. Here I will show you how to do that.

I have discovered, that composite_primary_keys breaks my SQL queries when :primary_key option specified both for has_many and has_one associations.

Step 0. Reproducing the bug

First of all we need to reproduce a bug. Please note: if you know where the problem is, you can skip this step. Let’s start from a simple example:

1	rails cpk_bug && cd cpk_bug

Now we will add dependencies to the config/environment.rb:

1	config.gem 'composite_primary_keys'

and to the config/environments/test.rb:

1
2
3

config.gem 'rspec', :lib => 'spec'
config.gem 'rspec-rails', :lib => false
config.gem 'factory_girl'

Okay. Now we are ready to start. Let’s generate some migrations:

1
2
3
4
5

script/generate rspec_model document upload_request_id:integer title:string description:text
script/generate rspec_model upload_request filename:string state:integer
script/generate rspec_model copyright_request upload_request_id:integer explanation:text
rm -rf spec/fixtures
rake db:migrate && rake db:test:clone

Let’s create our factories:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

Factory.define :copyright_request do |cr|
cr.explanation "This document is copyrighted by O'Reilly"
end

Factory.define :document do |d|
d.sequence(:title) { |n| "Document #{n}" }
d.description { |a| "The perfect description for the document '#{a.title}'" }
d.association :upload_request
end

Factory.define :upload_request do |ur|
ur.sequence(:filename) { |n| "file#{'%03d' % n}.pdf" }
ur.state 0
end

And our spec which should fail:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

require 'spec/spec_helper'

describe Document do
context 'when has copyright requests' do
before :each do
# Fake upload request used to desynchronise document and
# upload request IDs
Factory(:upload_request)
end

it 'should not have any copyright requests when just created' do
@document = Factory(:document)
@document.copyright_requests.should == []
end

it 'should return a list of copyright requests from #copyright_requests' do
@document = Factory(:document)
@request1 = Factory(:copyright_request, :upload_request => @document.upload_request)

@document.copyright_requests.should == [@request1]
end
end
end

Ok, it fails because we haven’t defined our associations on models. Let’s do it:

1
2
3
4
5
6
7
8
9
10
11
12

class Document < ActiveRecord::Base
belongs_to :upload_request

has_many :copyright_requests, :primary_key => :upload_request_id, :foreign_key => :upload_request_id
end

class UploadRequest < ActiveRecord::Base
end

class CopyrightRequest < ActiveRecord::Base
belongs_to :upload_request
end

Woohoo! Specs are failing with an error we’re trying to reproduce:

1
2
3
4
5
6
7
8
9
10
11
12

~/cpk_bug$ spec spec
..F.

1)
'Document when has copyright requests should return a list of copyright requests from #copyright_requests' FAILED
expected: [#],
got: [] (using ==)
/Users/kpumuk/cpk_bug/spec/models/document_spec.rb:20:

Finished in 0.155708 seconds

4 examples, 1 failure

Step 1. Setting up an environment for composite_primary_keys gem

In general to submit a patch to the Open Source project hosted by GitHub you have to perform the following steps: fork the repository on GitHub, write tests which fail, write a patch, ensure it works, push your changes to your fork repository, and submit a pull request. Let’s do just that!

To fork the repository I will use a perfect github gem by Dr Nic (BTW, he is the author of composite_primary_keys!)

1
2
3
4

sudo gem install github
gh clone drnic/composite_primary_keys
cd composite_primary_keys
gh fork

Now let’s configure our test environment and run tests:

1
2
3

rake local:setup
rake mysql:build_databases
rake test_mysql

You should see something like this:

1
2
3
4
5
6

Using native MySQL
Started
...................................................................................
Finished in 0.487679 seconds.

83 tests, 262 assertions, 0 failures, 0 errors

If you have any problems, check the test/README_tests.txt file for help.

Step 2. Reproducing failing tests inside composite_primary_keys test suite

We are doing TDD, right? So before any fixes we have to write a failing test first. Gem we’re hacking has a powerful test suite with many database tables created, so all we need is just to add associations to one of models, which will cover our issue.

First, add this to the test/fixtures/membership.rb model:

1 2	has_many :readings, :primary_key => :user_id, :foreign_key => :user_id has_one :reading, :primary_key => :user_id, :foreign_key => :user_id, :order => 'id DESC'

And this tests set goes to the test/test_associations.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

def test_has_many_with_primary_key
@membership = Membership.find([1, 1])

assert_equal 2, @membership.readings.size
end

def test_has_one_with_primary_key
@membership = Membership.find([1, 1])

assert_equal 2, @membership.reading.id
end

def test_joins_has_many_with_primary_key
@membership = Membership.find(:first, :joins => :readings, :conditions => { :readings => { :id => 1 } })

assert_equal [1, 1], @membership.id
end

def test_joins_has_one_with_primary_key
@membership = Membership.find(:first, :joins => :reading, :conditions => { :readings => { :id => 2 } })

assert_equal [1, 1], @membership.id
end

Now rake test_mysql produces following error (there are 4 of them, I will show only the first one):

1
2
3
4

1) Error:
test_has_many_with_primary_key(TestAssociations):
ActiveRecord::StatementInvalid: Mysql::Error: Operand should contain 1 column(s): SELECT * FROM `readings` WHERE (`readings`.`user_id` = 1,1)
...

Well, that are the errors we are working on. Time to fix them!

Step 3. Fixing the bug

I will not explain how I fixed that, you can check my commit for details. Here is the diff:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

diff --git a/lib/composite_primary_keys/associations.rb b/lib/composite_primary_keys/associations.rb
index 6b63664..9a9e173 100644
--- a/lib/composite_primary_keys/associations.rb
+++ b/lib/composite_primary_keys/associations.rb
@@ -180,11 +180,12 @@ module ActiveRecord::Associations::ClassMethods
raise AssociationNotSupported, "Polymorphic joins not supported for composite keys"
else
foreign_key = options[:foreign_key] || reflection.active_record.name.foreign_key
+ primary_key = options[:primary_key] || parent.primary_key
" LEFT OUTER JOIN %s ON %s " % [
table_name_and_alias,
composite_join_clause(
full_keys(aliased_table_name, foreign_key),
- full_keys(parent.aliased_table_name, parent.primary_key)),
+ full_keys(parent.aliased_table_name, primary_key)),
]
end
when :belongs_to
@@ -338,7 +339,7 @@ module ActiveRecord::Associations
@finder_sql << " AND (#{conditions})" if conditions

else
- @finder_sql = full_columns_equals(@reflection.klass.table_name, @reflection.primary_key_name, @owner.quoted_id)
+ @finder_sql = full_columns_equals(@reflection.klass.table_name, @reflection.primary_key_name, owner_quoted_id)
@finder_sql << " AND (#{conditions})" if conditions
end

@@ -386,7 +387,7 @@ module ActiveRecord::Associations
"#{@reflection.klass.quoted_table_name}.#{@reflection.options[:as]}_id = #{@owner.quoted_id} AND " +
"#{@reflection.klass.quoted_table_name}.#{@reflection.options[:as]}_type = #{@owner.class.quote_value(@owner.class.base_class.name.to_s)}"
else
- @finder_sql = full_columns_equals(@reflection.klass.table_name, @reflection.primary_key_name, @owner.quoted_id)
+ @finder_sql = full_columns_equals(@reflection.klass.table_name, @reflection.primary_key_name, owner_quoted_id)
end

@finder_sql << " AND (#{conditions})" if conditions

Run tests to get the following output:

1
2
3
4
5
6
7

~/cpk_bug/composite_primary_keys (master)$ rake test_mysql
Using native MySQL
Started
.......................................................................................
Finished in 0.511129 seconds.

87 tests, 266 assertions, 0 failures, 0 errors

We are done for now!

Step 4. Committing changes and pulling request

It’s time to commit our changes now:

1
2
3
4
5
6
7
8
9
10
11
12
13

~/cpk_bug/composite_primary_keys (master)$ git add .
~/cpk_bug/composite_primary_keys (master)$ git commit -m 'Fixed several bugs in has_one and has_many associations when :primary_key specified'
[master 3e29891] Fixed several bugs in has_one and has_many associations when :primary_key specified
3 files changed, 31 insertions(+), 3 deletions(-)
~/cpk_bug/composite_primary_keys (master)$ git push
Counting objects: 16, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (9/9), done.
Writing objects: 100% (9/9), 2.14 KiB, done.
Total 9 (delta 7), reused 0 (delta 0)
To git@github.com:kpumuk/composite_primary_keys.git
050d832..3e29891 HEAD -> master
~/cpk_bug/composite_primary_keys (master)$ gh home

Okay, the next step is a pull request. Last command opened a browser window with your fork. Navigate to the latest commit and press the “Pull Request” button (at the time of writing this article gh pull-request didn’t worked, and you can try to fix by yourself to understand the workflow):

That’s all, we have contributed to the community! Today Darrin Holst merged my commit into the core, and you can find it here. Not all things went smooth (I forgot to add a fixture, so tests were failing on first run), but he helped me a lot to get it working. That’s how Open Source works: we help each other to develop high quality software.

Credits

First of all, thanks to Dr Nic for the great plugin, one of the best piece of functionality I can’t imagine life without. Thanks to Darrin Holst for his patience and great help in debugging tests problem, and also for merging my commit into the composite_primary_keys core. Thanks to GitHub for the great Open Source code hosting solution, which makes working on Open Source projects so exciting.

Do you have comments or suggestions? You are welcome! Also, I will be happy if you follow me in Twitter.

My top 7 RSpec best practices

Dmytro Shteflyuk — Wed, 25 Nov 2009 16:22:31 +0000

I use RSpec in all my projects. It’s really hard to overemphasize how helpful it is and how much easier becomes your life if you have good specs coverage. But its outstanding flexibility enables many ways to make your specs awful: horribly slow, over-bloated, even non-readable sometimes. I do not want to teach you BDD and RSpec here, but instead I will give you some ideas how to improve your specs quality and increase efficiency of your BDD workflow.

1. Use `before :all` block carefully

Sometimes it looks like a good idea to create a test data in before :all block. But be careful — these blocks are not wrapped in a transaction, so the data will not be rolled back after the test. In this case you should clear your data in the after :all block manually.

1
2
3
4
5
6
7
8
9
10
11
12
13

describe Friendship do
before :all do
@users = (1..5).collect { Factory(:user) }
end

after :all do
@users.each { |user| user.destroy! }
end

it 'should do something' do
# Something interesting with @users
end
end

Another option is to move your before :all blocks to before :each to make them rolled back automatically.

2. For each test create exactly what it needs

Fixtures are cool when you start working on a project. But they quickly become painful while project grows: you add a new field to a fixture and break a half of your tests. There are tons of plugins which could simplify test data creation, I personally recommend factory_girl: it’s pretty slick and easy to use.

1
2
3
4
5
6
7
8

Factory.define :user do |f|
f.sequence(:login) { |n| "user#{n}" }
f.email { |a| "#{a.login}@example.com" }
f.description "Ruby on Rails Developer"
end

# Somewhere in specs
@user = Factory(:user, :admin => true)

3. Do not create hundreds of records for a particular spec

Sometimes you want to test a method which operates on large set of records (filtering, trimming, etc). For example, this method returns 50 most popular videos (and no more). The straight approach is to create 51 record and make sure, that the size of the returned array is 50. When I saw a code snippet like this in our project first time, I was surprised. There was a few more pieces sharing this behavior, so here is my advice: add a parameter to the method, which will limit the number of records to return. In this case you can create 3 records, and pass 2 as a parameter.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

describe User do
it 'should return top users in User.top method' do
@users = (1..3).collect { Factory(:user) }
top_users = User.top(2).all
top_users.should have(2).entries
end
end

class User < ActiveRecord::Base
# Select N top users. Returns 10 entries when called without arguments.
# User.top.all.size # => 10
# User.top(2).all.size # => 2
#
named_scope :top, lambda { |*args| { :limit => (args.size > 0 ? args[0] : 10) } }
end

4. Do not over-mock

Mocking is interesting and sometimes very useful technology. You may mock just everything so you spec will not hit the database. But there is a catch: your model code may be changed some day causing callers to break. Since you mock everything, you will never get failing specs. So now you should update all your mocks to fit a new interface. Also you would not be able to find SQL queries errors if you have mocked them. Instead of this I use integration approach: controller should talk to models, which have to hit the database. Real database with real data (OK, not so real). The practice 2 can help you in test data creation.

Bad:

1
2
3
4
5
6
7
8
9
10
11
12

describe VideosController do
describe '.create action' do
it 'should assign top videos' do
params = { :title => 'new video', :description => 'video description' }
@video = mock_model(Video)
Video.should_receive(:new).and_return(@video)
@video.should_receive(:update_attributes).with(params).and_return(true)
get :index, :video => params
assigns[:video].should be(@video)
end
end
end

Good:

1
2
3
4
5
6
7
8
9
10
11

describe VideosController do
describe '.create action' do
it 'should assign top videos' do
params = { :title => 'new video', :description => 'video description' }
get :index, :video => params
assigns[:video].should_not be_new_record
assigns[:video].title.should == params[:title]
assigns[:video].description.should == params[:description]
end
end
end

But you can use mocks to skip records retrieving from the database (make sure you have specs covering corresponding model code). Let me explain this. For example, you need to render 20 entries in an RSS feed. You could create 21 record in the database using a factory, and then ensure only 20 of them were retrieved, or you could mock your finder method and check its parameter. You may not like magic numbers like 20 in this particular case, and this is a good point. Just move this magic number to the config and ensure it was used to do the retrieval.

1
2
3
4
5
6
7
8
9
10

describe VideosController do
describe '.index action' do
it 'should assign top videos' do
@videos = [mock_model(Video), mock_model(Video)]
Video.should_receive(:top).with(50).and_return(@videos)
get :index
assigns[:top_videos].should be(@videos)
end
end
end

5. Use contexts

RSpec spec specifies how particular code should work. Usually, in the beginning you tell what you are going to describe in this spec, and inside describe block you specify what the code should do:

1
2
3
4
5

describe Video do
it 'should return 5 records in Video.top method' do
Video.top.should have(5).items
end
end

Usually you have more than one it block for each method. To group related specs I recommend to use nested describe blocks. Since describe is aliased to context when placed inside another describe, I think it’s a good idea to use it for specs grouping. Each context may have its own before and after blocks (in this case parent blocks will be called right before child ones).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

describe Video do
describe '.top' do
it 'should return 5 records' do
Video.top.should have(5).items
end
end

context 'when just created' do
before :each do
@video = Video.new
end

# ...
end
end

6. Create several test suites to speed up your workflow

There are many things you can do to make your BDD more efficient. We will take a look at two of them: creating a separate test suites and running recently modified specs.

There are several standard test suites configured in RSpec by default: spec:controllers, spec:views, spec:helpers, spec:lib. Check the rake -T spec output to get a list of available RSpec tasks. Let’s create a simple Rake tasks generator for spec suites:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

SPEC_SUITES = [
{ :id => :acl, :title => 'access control', :files => %w(spec/controllers/**/acl_spec.rb) },
{ :id => :amazon, :title => 'Amazon libraries', :dirs => %w(spec/lib/amazon) }
]

namespace :spec do
namespace :suite do
SPEC_SUITES.each do |suite|
desc "Run all specs in #{suite[:title]} spec suite"
Spec::Rake::SpecTask.new(suite[:id]) do |t|
spec_files = []
if suite[:files]
suite[:files].each { |glob| spec_files += Dir[glob] }
end

if suite[:dirs]
suite[:dirs].each { |glob| spec_files += Dir["#{glob}/**/*_spec.rb"] }
end

t.spec_opts = ['--options', "\"#{Rails.root}/spec/spec.opts\""]
t.spec_files = spec_files
end
end
end
end

Check what tasks are available now:

1
2
3
4
5

~/test$ rake -T spec:suite

(in /Users/kpumuk/test)
rake spec:suite:acl # Run all specs in access control spec suite
rake spec:suite:amazon # Run all specs in Amazon libraries spec suite

It was easy! And now let’s take a look at the Rake task for running the recently touched specs (last 10 minutes).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Grab recently touched specs
def recent_specs(touched_since)
recent_specs = Dir['app/**/*'].map do |path|
if File.mtime(path) > touched_since
spec = File.join('spec', File.dirname(path).split("/")[1..-1].join('/'),
"#{File.basename(path, ".*")}_spec.rb")
spec if File.exists?(spec)
end
end.compact

recent_specs += Dir['spec/**/*_spec.rb'].select do |path|
File.mtime(path) > touched_since
end.uniq
end

namespace :spec do
desc 'Run all recent specs in spec directory touched in last 10 minutes'
Spec::Rake::SpecTask.new(:recent) do |t|
t.spec_opts = ['--options', "\"#{RAILS_ROOT}/spec/spec.opts\""]
t.spec_files = recent_specs(Time.now - 10.minutes)
end
end

And don’t forget to check autospec and watchr gems.

7. Stop `spec_helper` from being loaded multiple times

Just don’t do that. If you got a big project, there is a chance that the spec_helper will be required in a many different ways: File.expand_path, File.join, etc.,— which results in it being loaded several times and it slows down your test suite!

To avoid this, add the following code at the top of your spec_helper.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# figure out where we are being loaded from
if $LOADED_FEATURES.grep(/spec\/spec_helper\.rb/).any?
begin
raise "foo"
rescue => e
puts <<-MSG
===================================================
It looks like spec_helper.rb has been loaded
multiple times. Normalize the require to:

require "spec/spec_helper"

Things like File.join and File.expand_path will
cause it to be loaded multiple times.

Loaded this time from:

#{e.backtrace.join("\n ")}
===================================================
MSG
end
end

It will show where you’ve tried to load spec_helper from so you will be able to fix it immediately. Also there is an interesting snippet of code here, which will find and replace all wrong includes.

Conclusion

RSpec is not a silver bullet. You can have 100% coverage and fine-grained specs, but it does not mean your application is completely bug-free. Refactor your specs, increase your programming level, and refactor again. Write specs for any issue that you, your QAs or users have faced. And remember: do not over-mock.

Credits

This is my first article written completely in Google Wave in collaboration with several good Russian rubyists. Thank you all, guys. I want to acknowledge the editorial help of Roman Dmytrenko and Alexey Kovyrin. Robby Russell created a great picture illustrating how sexy is RSpec. And thank you all my readers for your attention.

Did you like this article? You should follow me in Twitter here.

Weekly Link Dump #2

Dmytro Shteflyuk — Tue, 22 Sep 2009 13:51:04 +0000

Time to post some interesting stuff I’ve found in Internet last week. Today we going to talk about CSS font stacks, Ruby structs, extract_options! method came from Active Support, bash scripting, software version control visualization.

Guide to CSS Font Stacks: Techniques and Resources

by Cameron Chapman

CSS Font stacks are one of those things that elude a lot of designers. Many stick to the basic stacks Dreamweaver auto-recommends or go even more basic by just specifying a single web-safe font.

But doing either of those things means you’re missing out on some great typography options. Font stacks can make it possible to show at least some of your visitors your site’s typography exactly the way you intend without showing everyone else a default font. Read on for more information on using and creating effective font stacks with CSS.

When I started posting links, I decided not to mention articles posted by Smashing Magazine, thoughtbot and other sites which everyone reads. But this is totally awesome: a complete guide on font faces, list of font tools and articles about typography. I promise, I will not publish links from Smashing Magazine ever again.

Structs inside out

by Robert Klemme

Today we’re back to normal blog mode, where each article stands for itself. Muppet Labs are closed and we will be continuing our journey across the Ruby universe starting with an indepth look at Ruby’s Struct class — Ruby’s Swiss army knife for structured data.

Struct can be used without any additional require statement — it’s just there. This means it comes with zero additional overhead during initial interpreter startup — one of the many advantage of using Struct. But first let’s look at the basics.

Great Ruby Struct class usage examples.

Inside Ruby on Rails: extract_options! from Arrays

by Simone Carletti

How many times did you see a method call like the following one in your Rails application: my_method :arg1, :foo => true?

What makes my_method quite special is the ability to pass an arbitrary number of parameters (:arg1, :arg2…) followed by a list of keyword/value options.

This is made possible by a really helpful method provided by ActiveSupport called extract_options!. What this core extension does is to extract the options from the given set of arguments. When no options are available, the method returns a blank Hash.

Nice Ruby on Rails method I didn’t know.

Advanced Bash-Scripting Guide

by Mendel Cooper

The shell is a command interpreter. More than just the insulating layer between the operating system kernel and the user, it’s also a fairly powerful programming language. A shell program, called a script, is an easy-to-use tool for building applications by “gluing together” system calls, tools, utilities, and compiled binaries. Virtually the entire repertoire of UNIX commands, utilities, and tools is available for invocation by a shell script. If that were not enough, internal shell commands, such as testing and loop constructs, lend additional power and flexibility to scripts. Shell scripts are especially well suited for administrative system tasks and other routine repetitive tasks not requiring the bells and whistles of a full-blown tightly structured programming language.

Old good guide for the bash scripting. Must read for any developer.

gource — software version control visualization

by Andrew Caudwell

Gource is a software version control visualization tool for Git and CVS.

Software projects are displayed by Gource as an animated tree with the root directory of the project at its centre. Directories appear as branches with files as leaves. Developers can be seen working on the tree at the times they contributed to the project.

This freaking awesome tool produces animated visualization of source code history based on Git or CVS reporsitory. Also take a look at the code_swarm project, which does very similar things.

PS. Did you notice a new threaded comments structure in this blog? Also I have rewritten theme layout a little, so there is HTML5 here now.

ЗЗЫ. Журналисты WebStream.com.ua взяли у меня интервью. Читаем здесь.

Weekly Link Dump #1

Dmytro Shteflyuk — Mon, 14 Sep 2009 09:48:01 +0000

This is a first link dump in this blog, where I will list all interesting links that I have found in Internet. I plan to post link dumps once a week, so stay tuned to read most useful stuff with no effort. Todays topics are: organizing your CSS, top mistakes made by WordPress plugins authors, ways MySQL uses indexes, and the git version control advantages over the Subversion.

Beautiful CSS: Organizing Your Stylesheets

by Matthew Griffin

When I first took the plunge into CSS several years ago, one of my biggest frustrations was stylesheet organization. I scoured source code from popular sites trying to figure how they accomplished various layout effects. But tracking back and forth from stylesheets to HTML proved to be a difficult task. Unfortunately , that separation of style and content that makes CSS so awesome can also make it difficult to understand. Adding to that difficulty is the fact that each designer may have a different way of organizing stylesheets. If you inherit someone else’s site, this can cause some problems. In a perfect world everyone’s CSS would be well-organized, easy to scale, and easy to understand. We may not be able to attain such CSS Nirvana but we can at least make it easier on ourselves and those we work with by following this set of guidelines.

A nice article about organization of your CSS code. Author recommends to split your CSS code to several sections (universal styles, library styles, template layouts, individual page styles.) Also he proposes to indent styles related to nested HTML elements.

Top 10 Most Common Coding Mistakes in WordPress Plugins

by Ozh

As promised, I’m going to share a list of the most common mistakes, errors, misunderstandings, bad habits or wrong design decisions I’ve encountered while reviewing all these 43 plugins. Some are highly critical stuff (I’ve contacted 3 plugins authors after finding serious security holes in their plugin), some are more potential annoyances than real bugs, or are just causing a waste of server resources that could be avoided, but all have something in common: they’re trivial to fix. I’ve classified them in two parts: 10 bad code signs, plus a bonus with design decisions that suck. If you consider yourself a semi experienced coder or better, be sure to skip this article, you’re not going to learn a thing.

A comprehensive list of common pitfalls of WordPress plugins authors. If you maintain a plugin, please read this article carefully, most of mistakes are quite common (yes, I found several issues with my ones

3 ways MySQL uses indexes

by Peter Zaitsev

I often see people confuse different ways MySQL can use indexing, getting wrong ideas on what query performance they should expect. There are 3 main ways how MySQL can use the indexes for query execution, which are not mutually exclusive, in fact some queries will use indexes for all 3 purposes listed here.

You should read it. Period! Article covers all ways how MySQL uses indexes: filtering rows, sorting data, and reading data.

Why You Should Switch from Subversion to Git

by Scott Chacon

You may have heard some hubbub over distributed version control systems recently. You may dismiss it as the next hot thing, the newest flavor of kool-aid currently quenching the collective thirst of the bandwagon jumpers. You, however, have been using Subversion quite happily for some time now. It has treated you pretty well, you know it just fine and you are comfortable with it – I mean, it’s just version control, right?

Yet another article about git and what features make it so wonderful tool for any developer. Author covers main features of this version control system, and explains how it could improve your work and boost your productivity. Hey Subversion (SourceSafe, CVS, etc) users, I do not accept any excuses, you definitely should read it. Yes, right now!

In reply to latest articles in Thoughtbot blog

Dmytro Shteflyuk — Thu, 10 Sep 2009 10:33:51 +0000

Last few weeks Thoughtbot publish lots of really stupid (gsub syntax manual; are you serious?) articles (WTF is Best practice: index every boolean column) in their blog. And yesterdays article about tailing your Rails log is the absolute leader, it’s freaking awesome. Hey guys, I’m waiting for other articles in this series: “@ in attribute names, what does it mean?”, “if-then-else statement usage best practices for Ruby on Rails senior developers”, “how to install a gem”.

Hey robots, we have a reply to your outstanding articles: A wonderful way to list your project files. Please read it carefully, you definitely will find something useful for you! Thanks to @labria for his great exploration.

Simplifying your Ruby on Rails code: Presenter pattern, cells plugin

Dmytro Shteflyuk — Wed, 09 Sep 2009 05:41:14 +0000

Today we will talk about code organization in Ruby on Rails projects. As everybody knows, Ruby on Rails is a conventional framework, which means you should follow framework architects’ decisions (put your controllers inside app/controllers, move all your logic into models, etc.) But there are many open questions around those conventions. In this write-up I will try to summarize my personal experience and show how I usually solve these problems.

Here is the list of questions we will talk about:

You have some logic in your view, which uses your models extensively. There are no places in other views with such logic. The classic recommendation is to move this code into a model, but after a short time your models become bloated with stupid one-off helper methods. The solution: pattern Presenter.
Your constructor contains a lot of code to retrieve some values for your views from the database or another storage. You have a lot of fragment_exist? calls to ensure no of your data is loaded when corresponding fragment is already in cache. It’s really hard to test a particular action because of it’s size. The solution: pattern Presenter.
You have a partial, used everywhere on the site. It accepts a lot of parameters to configure how rendered code should look like. The header of this partial, which initializes default values of parameters, becomes larger and larger. The solution: cells plugin.

Please note: sample application is available on GitHub.

Presenter Pattern

Okay, you have an idea when to use this patterns. Let’s look at the example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

class HomeController < ApplicationController
def show
unless fragment_exist?('home/top_videos')
@top_videos = Video.top.all(:limit => 10)
end

unless fragment_exist?('home/categories')
@categories = Category.all(:order => 'name DESC')
end

unless fragment_exist?('home/featured_videos')
@featured_videos = Video.featured.all(:limit => 5)
end

unless fragment_exist?('home/latest_videos')
@latest_videos = Video.latest.all(:limit => 5)
end
end
end

And the view:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

<h1>Home page</h1>

<div id="top_videos">
<h2>Top videos</h2>
<% cache('home/top_videos') do %>
<%= render 'videos', :videos => @top_videos, :hide_description => true %>
<% end %>
</div>

<div class="tabs">
<ul id="taxonomy">
<li><a href="#" id="categories" class="current">Categories</a></li>
</ul>
<div class="categories_panel">
<h2>Categories</h2>
<% cache('home/categories') do %>
<%= render 'categories' %>
<% end %>
</div>
</div>

<div class="box">
<div id="latest">
<h2>Latest videos</h2>
<% cache('home/latest_videos') do %>
<%= render 'videos', :videos => @latest_videos, :hide_thumbnail => true %>
<% end %>
</div>
<div id="featured">
<h2>Featured videos</h2>
<% cache('home/featured_videos') do %>
<%= render 'videos', :videos => @featured_videos, :hide_thumbnail => true %>
<% end %>
</div>
</div>

Note: this code is available in the first commit of my presenter example project.

Scary code, isn’t it? So let’s refactor it using Presenter pattern. I prefer to put presenters into a separate folder app/presenters, so first we should add it to Rails load path. Add this line to your config/environment.rb:

1
2
3

config.load_paths += %W(
#{Rails.root}/app/presenters
)

Now we are ready to write our presenter (app/presenters/home_presenters/show_presenter.rb):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

module HomePresenters
class ShowPresenter
def top_videos
@top_videos ||= Video.top.all(:limit => 10)
end

def categories
@categories ||= Category.all(:order => 'name DESC')
end

def featured_videos
@featured_videos ||= Video.featured.all(:limit => 5)
end

def latest_videos
@latest_videos ||= Video.latest.all(:limit => 5)
end
end
end

Sometimes presenters depend on parameters, so feel free to add an initialize method. It could accept particular params or whole params collection:

1
2
3

def initialize(video_id)
@video_id = video_id
end

Now let’s refactor our controller:

1
2
3
4
5

class HomeController < ApplicationController
def show
@presenter = HomePresenters::ShowPresenter.new
end
end

Whoa, that’s nice! View now is little different:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

<h1>Home page</h1>

<div id="top_videos">
<h2>Top videos</h2>
<% cache('home/top_videos') do %>
<%= render 'videos', :videos => @presenter.top_videos, :hide_description => true %>
<% end %>
</div>

<div class="tabs">
<ul id="taxonomy">
<li><a href="#" id="categories" class="current">Categories</a></li>
</ul>
<div class="categories_panel">
<h2>Categories</h2>
<% cache('home/categories') do %>
<%= render 'categories' %>
<% end %>
</div>
</div>

<div class="box">
<div id="latest">
<h2>Latest videos</h2>
<% cache('home/latest_videos') do %>
<%= render 'videos', :videos => @presenter.latest_videos, :hide_thumbnail => true %>
<% end %>
</div>
<div id="featured">
<h2>Featured videos</h2>
<% cache('home/featured_videos') do %>
<%= render 'videos', :videos => @presenter.featured_videos, :hide_thumbnail => true %>
<% end %>
</div>
</div>

Presenters testing is much easier than testing of bloated controllers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

describe HomePresenters::ShowPresenter do
before :each do
@presenter = HomePresenters::ShowPresenter.new
end

it 'should respond to :top_videos' do
expect { @presenter.top_videos }.to_not raise_error
end

it 'should respond to :categories' do
expect { @presenter.categories }.to_not raise_error
end

it 'should respond to :featured_videos' do
expect { @presenter.featured_videos }.to_not raise_error
end

it 'should respond to :latest_videos' do
expect { @presenter.latest_videos }.to_not raise_error
end
end

Please note: this code is available in the second commit of my presenter example project.

Please note: you should not do any manipulations on models in presenters. They only decorate models with helper methods to be used inside controllers or views, nothing else. There are several articles describing a Conductor pattern as a presenter, do not repeat their mistakes. See the first link in the list below to get an idea about the differences.

Cells Plugin

Okay, now we have a clean controller. But what about views? Let’s take a look at the videos partial:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

<%
hide_thumbnail = hide_thumbnail === true;
hide_description = hide_description === true;
css_class ||= 'videos'
style ||= :div
case style.to_sym
when :section
parent_tag = 'section'
child_tag = 'div'
when :list
parent_tag = 'ul'
child_tag = 'li'
else
parent_tag = 'div'
child_tag = 'div'
end
%>

<% content_tag parent_tag, :class => css_class do %>
<% videos.each do |video| %>
<% content_tag child_tag do %>
<h3><%= h video.title %></h3>
<%= image_tag(video.thumbnail_url, :class => 'thumb') unless hide_thumbnail %>
<%= '

%s</p>' % h(video.description) unless hide_description %>
<% end %>
<% end %>
<% end %>

So, what the heck? Is this a view or a controller? Remember old PHP days, with all this spaghetti code? That is it. It’s hard to test, it looks scary, it bad. So here cells plugin comes to the stage.

First, we need to install the plugin:

1	script/plugin install git://github.com/apotonick/cells.git

Now let’s generate a cell:

1	script/generate cell Video videos

And write some code (app/cells/video.rb):

1
2
3
4
5
6
7
8
9
10
11
12
13

class VideoCell < Cell::Base
def videos
@videos = @opts[:videos]
@hide_thumbnail = @opts[:hide_thumbnail] === true;
@hide_description = @opts[:hide_description] === true;
@css_class = @opts[:css_class] || 'videos'

view = (@opts[:style] || :div).to_sym
view = :div unless [:section, :list].include?(view)

render :view => "videos_#{view}"
end
end

app/cells/video/videos_section.html.erb:

1
2
3
4
5
6
7

class="<%= @css_class %>">
<% @videos.each do |video| %>
<div>
<%= render :partial => 'video', :locals => { :video => video } %>
</div>
<% end %>
</section>

app/cells/video/videos_list.html.erb:

1
2
3
4
5
6
7

<ul class="<%= @css_class %>">
<% @videos.each do |video| %>
<li>
<%= render :partial => 'video', :locals => { :video => video } %>
</li>
<% end %>
</ul>

app/cells/video/videos_div.html.erb:

1
2
3
4
5
6
7

<div class="<%= @css_class %>">
<% @videos.each do |video| %>
<div>
<%= render :partial => 'video', :locals => { :video => video } %>
</div>
<% end %>
</div>

app/cells/video/_video.html.erb:

1
2
3

<h3><%= h video.title %></h3>
<%= image_tag(video.thumbnail_url, :class => 'thumb') unless @hide_thumbnail %>
<%= '

%s</p>' % h(video.description) unless @hide_description %>

And the view:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

<h1>Home page</h1>

<div id="top_videos">
<h2>Top videos</h2>
<% cache('home/top_videos') do %>
<%= render_cell :video, :videos, :videos => @presenter.top_videos, :hide_description => true %>
<% end %>
</div>

<div class="tabs">
<ul id="taxonomy">
<li><a href="#" id="categories" class="current">Categories</a></li>
</ul>
<div class="categories_panel">
<h2>Categories</h2>
<% cache('home/categories') do %>
<%= render 'categories' %>
<% end %>
</div>
</div>

<div class="box">
<div id="latest">
<h2>Latest videos</h2>
<% cache('home/latest_videos') do %>
<%= render_cell :video, :videos, :videos => @presenter.latest_videos, :hide_thumbnail => true %>
<% end %>
</div>
<div id="featured">
<h2>Featured videos</h2>
<% cache('home/featured_videos') do %>
<%= render_cell :video, :videos, :videos => @presenter.featured_videos, :hide_thumbnail => true %>
<% end %>
</div>
</div>

Wow! That’s pretty easy to read and modify. All the logic is in the code now, all the views are easy to read, and moreover: it’s more than easy to test now! I have a little plugin called rspec-cells, and I have committed a patch yesterday to get it working with the latest RSpec. Here is how you spec could look like:

1
2
3
4
5
6
7
8
9

describe VideoCell do
context '.videos' do
it 'should initialize :videos variable' do
videos = mock('Videos')
render_cell :videos, :videos => videos
assigns[:videos].should be(videos)
end
end
end

So it looks almost like a classic Ruby on Rails controller spec. I hope to review the code in nearest feature and will send a pull request to the cells plugin author. Of course, if you
found a bug, feel free to contact me.

Please note: this code is available in the third commit of my presenter example project.

Scribd open source projects

Dmytro Shteflyuk — Tue, 08 Sep 2009 02:06:05 +0000

It’s time to summarize what we have done for the Open Source community. Scribd is pretty open company, we release a lot of code into the public after a time (sometimes it is short, sometimes it is not). Here I want to mention all the code we have opensourced. Please take into account that time is moving on, so we are publishing more and more code. I will update this post periodically, so stay tuned. Follow me on Twitter to get instant updates.

Here is the list of our projects in alphabetical order:

bounces-handler — Email Bounces Processing System with Rails plugin to prevent Rails mailers from sending any messages to a blocked addresses.
db-charmer — ActiveRecord Connections Magic (slaves, multiple connections, etc).
easy-prof — Simple and easy to use Ruby code profiler, which could be used as a Rails plugin.
Fast Sessions — Sessions class for ActiveRecord sessions store created to work fast (really fast).
loops — Simple background loops framework for Ruby on Rails and Merb.
magic-enum — Method used to define ENUM-like attributes in your model (int fields actually).
rlibsphinxclient — A Ruby wrapper for pure C searchd client API library.
rscribd — Ruby client library for the Scribd API.
Rspec Cells — A library for testing applications that are using Cells in RSpec.
Scribd Desktop Uploader — A fully native Cocoa Macintosh uploader app for the Scribd.com website.

bounces-handler

Bounces-handler package is a simple set of scripts to automatically process email bounces and ISP’s feedback loops emails, maintain your mailing blacklists and a Ruby on Rails plugin to use those blacklists in your RoR applications.

This piece of software has been developed as a part of more global work on mailing quality improvement in Scribd.com, but it was one of the most critical steps after setting up reverse DNS records, DKIM and SPF.

Links: Project Home Page on GitHub | Introduction Blog Post | RDoc Documentation.

db-charmer

DbCharmer is a simple yet powerful plugin for ActiveRecord that does a few things:

Allows you to easily manage AR models’ connections (switch_connection_to method)
Allows you to switch AR models’ default connections to a separate servers/databases
Allows you to easily choose where your query should go (Model.on_db methods)
Allows you to automatically send read queries to your slaves while masters would handle all the updates.
Adds multiple databases migrations to ActiveRecord

It requires Ruby on Rails version 2.3 or later. The main purpose of this plugin is to put all the databases-related code we have been using in Scribd for a while into a single easy-to use package.

Links: Project Home Page on GitHub | Test Rails Application on GitHub | RDoc Documentation.

easy-prof

Simple and easy to use Ruby code profiler, which could be used as a Rails plugin. The main idea behind the easy-prof is creating check points and your code and measuring time needed to execute code blocks. Here is the example of easy-prof output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

[home#index] Benchmark results:
[home#index] debug: Logged in user home page
[home#index] progress: 0.7002 s [find top videos]
[home#index] progress: 0.0452 s [build categories list]
[home#index] progress: 0.0019 s [build tag cloud]
[home#index] progress: 0.0032 s [find featured videos]
[home#index] progress: 0.0324 s [find latest videos]
[home#index] debug: VIEW STARTED
[home#index] progress: 0.0649 s [top videos render]
[home#index] progress: 0.0014 s [categories render]
[home#index] progress: 2.5887 s [tag cloud render]
[home#index] progress: 0.0488 s [latest videos render]
[home#index] progress: 0.1053 s [featured video render]
[home#index] results: 3.592 s

From this output you can see what checkpoints takes longer to reach, and what code fragments are pretty fast.

Links: Project Home Page on GitHub | Introduction Blog Post | RDoc Documentation.

Fast Sessions

FastSessions is a sessions class for ActiveRecord sessions store created to work fast (really fast). It uses some techniques which are not so widely known in developers’ community and only when they cause huge problems, performance consultants are trying to help with them.

FastSessions plugin was born as a hack created for Scribd.com (large RoR-based web project), which was suffering from InnoDB auto-increment table-level locks on sessions table.

So, first of all, we removed id field from the table. Next step was to make lookups faster and we’ve used a following technique: instead of using (session_id) as a lookup key, we started using (CRC32(session_id), session_id) — two-columns key which really helps MySQL to find sessions faster because key cardinality is higher (so, mysql is able to find a record earlier w/o checking a lots of index rows). We’ve benchmarked this approach and it shows 10–15% performance gain on large sessions tables.

And last, but most powerful change we’ve tried to make was to not create database records for empty sessions and to not save sessions data back to database if this data has not been changed during current request processing. With this change we basically reduce inserts number by 50-90% (depends 0n application).

All of these changes were implemented and you can use them automatically after a simple plugin installation.

There is a fork patched by mudge for full compatibility with Ruby on Rails version 2.3 or later.

Links: Project Home Page on Google Code | Introduction Blog Post | Fork on GitHub Compatible with Rails 2.3 and Later.

loops

loops is a small and lightweight library for Ruby on Rails, Merb and other frameworks created to support simple background loops in your application which are usually used to do some background data processing on your servers (queue workers, batch tasks processors, etc).

Originally loops plugin was created to make our own loops code more organized. We used to have tens of different modules with methods that were called with script/runner and then used with nohup and other not so convenient backgrounding techniques. When you have such a number of loops/workers to run in background it becomes a nightmare to manage them on a regular basis (restarts, code upgrades, status/health checking, etc).

After a short time of writing our loops in more organized ways we were able to generalize most of the loops code so now our loops look like a classes with a single mandatory public method called run. Everything else (spawning many workers, managing them, logging, backgrounding, pid-files management, etc) is handled by the plugin itself.

Links: Project Home Page on GitHub | Introduction Blog Post | RDoc Documentation.

magic-enum

Method used to define ENUM-like attributes in your model (int fields actually). It’s easier to show what it does in code rather than to explain in plain English:

1
2
3
4
5
6
7

Statuses = {
:unknown => 0,
:draft => 1,
:published => 2,
:approved => 3
}
define_enum :status, :default => 1, :raise_on_invalid => true, :simple_accessors => true

is identical to

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

Statuses = {
:unknown => 0,
:draft => 1,
:published => 2,
:approved => 3
}
StatusesInverted = Statuses.invert

def status
StatusesInverted[self[:status].to_i] || StatusesInverted[1]
end

def status=(value)
raise ArgumentError, "Invalid value \"#{value}\" for :status attribute of the #{self.class} model" if
Statuses[value].nil?
self[:status] = Statuses[value]
end

def unknown?
status == :unknown
end

def draft?
status == :draft
end

def published?
status == :published
end

def approved?
status == :approved
end

This plugin was originally developed for Best Tech Videosand later was cleaned up in Scribd repository and released to the public.

Links: Project Home Page on GitHub | RDoc Documentation.

rlibsphinxclient

A Ruby wrapper for pure C searchd client API library. It works much faster than any Ruby client for Sphinx, so you can check it to ensure you application works as fast as possible.

Please note: this is *highly experimental* library so use it at your own risk.

Links: Project Home Page on GitHub | RDoc Documentation.

rscribd

Ruby client library for the Scribd API. This gem provides a simple and powerful library for the Scribd API, allowing you to write Ruby applications or Ruby on Rails websites that upload, convert, display, search, and control documents in many formats. For more information on the Scribd platform, visit the Scribd Platform Documentation page.

The main features are:

Upload your documents to Scribd’s servers and access them using the gem
Upload local files or from remote web sites
Search, tag, and organize documents
Associate documents with your users’ accounts

Links: Project Home Page on GitHub | Scribd Platform Documentation | RDoc Documentation.

Rspec Cells

This plugin allows you to test your cells easily using RSpec. Basically, it adds an example group especially for cells, with several helpers to perform cells rendering.

If you are not sure what is cells, please visit its home page.

Spec for a regular cell could look like:

1
2
3
4
5
6
7
8
9
10
11
12
13

describe VideoCell do
integrate_views

context '.videos' do
it 'should initialize :videos variable' do
params[:id] = 10
session[:user_id] = 20
opts[:opt] = 'value'
result = render_cell :videos, { :videos => [] }, :slug => 'hello'
result.should have_tag('div', :class => :videos)
end
end
end

Links: Project Home Page on GitHub | Cells Home Page | Cells Home Page on GitHub.

Scribd Desktop Uploader

A fully native Cocoa Macintosh uploader app for the Scribd.com website. Supports following features:

Upload many files at once from your desktop.
Edit titles, tags, and other metadata before uploading.
Quickly and easily manage bulk uploads, straight from your desktop.
Right-click to start uploading files directly to Scribd (Windows only).

Links: Project Home Page on GitHub | Home Page on Scribd.com.

Changelog

September 9, 2009
- Fixed a link to GitHub homepage of the rspec-cells plugin.
September 8, 2009
- Added rspec-cells plugin.

Creating a simple but powerful profiler for Ruby on Rails

Dmytro Shteflyuk — Wed, 26 Aug 2009 21:23:15 +0000

You are developing a large Web application. Controllers are full of complex data retrieving logic, views contain tons of blocks, partials, loops. One day you will receive an email with user complaints about some of your pages slowness. There are many profiling tools, some of them are easy (ruby-prof), others are large and complex (newrelic), but regardless of this it’s really hard to find the particular place where you have a real bottleneck. So we created really simple, but über-useful tool for ruby code profiling.

First of all, we need to decide what features we need from this tool. Don’t know about you, but all I need is to measure execution time of particular ruby code block. Here is what I mean:

1
2
3
4
5
6
7
8
9
10
11
12
13

[home#index] debug: Logged in user home page
[home#index] progress: 0.7002 s [find top videos]
[home#index] progress: 0.0452 s [build categories list]
[home#index] progress: 0.0019 s [build tag cloud]
[home#index] progress: 0.0032 s [find featured videos]
[home#index] progress: 0.0324 s [find latest videos]
[home#index] debug: VIEW STARTED
[home#index] progress: 0.0649 s [top videos render]
[home#index] progress: 0.0014 s [categories render]
[home#index] progress: 2.5887 s [tag cloud render]
[home#index] progress: 0.0488 s [latest videos render]
[home#index] progress: 0.1053 s [featured video render]
[home#index] results: 3.592 seconds

So what do we see from this output? There are two slow blocks: top videos retrieving and tag cloud rendering. Now we just know what to do to make this page faster.

Let’s write the code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91

module EasyProfiler
class Profile
@@profile_results = {}

cattr_accessor :enable_profiling
@@enable_profiling = false

cattr_accessor :print_limit
@@print_limit = 0.01

def self.start(name, options = {})
options[:enabled] ||= @@enable_profiling
options[:limit] ||= @@print_limit
return NoProfileInstance.new unless options[:enabled]

if @@profile_results[name]
puts "EasyProfiler::Profile.start() collision! '#{name}' is already started!"
return NoProfileInstance.new
end

@@profile_results[name] = ProfileInstance.new(name, options)
end

def self.stop(name, options = {})
options[:enabled] ||= @@enable_profiling
options[:limit] ||= @@print_limit
return unless options[:enabled]

unless @@profile_results[name]
puts "EasyProfiler::Profile.stop() error! '#{name}' is not started yet!"
return false
end

total = @@profile_results[name].total

if total > options[:limit]
@@profile_results[name].buffer_checkpoint("results: %0.4f seconds" % total)
@@profile_results[name].dump_results
end

@@profile_results.delete(name)
end
end

class ProfileInstance
def initialize(name, options = {})
@name = name
@start = @progress = Time.now.to_f
@buffer = []
end

def progress(message)
progress = (now = Time.now.to_f) - @progress
@progress = now
buffer_checkpoint("progress: %0.4f seconds [#{message}]" % progress)
end

def debug(message)
@progress = Time.now.to_f
buffer_checkpoint("debug: #{message}")
end

def total
Time.now.to_f - @start
end

def buffer_checkpoint(message)
@buffer << message
end

def dump_results
profile_logger.info("[#{@name}] Benchmark results:")
@buffer.each do |message|
profile_logger.info("[#{@name}] #{message}")
end
end

def profile_logger
root = Object.const_defined?(:RAILS_ROOT) ? "#{RAILS_ROOT}/log" : File.dirname(__FILE__)
@profile_logger ||= Logger.new(root + '/profile.log')
end
end

class NoProfileInstance
def progress(message)
end

def debug(message)
end
end
end

We have defined two class attributes: EasyProfiler::Profile.enable_profiling (to be able to disable or enable profiler globally) and EasyProfiler::Profile.print_limit (to filter out from log code blocks that are fast enough).

Then we defined two methods, which accept name of profile session (for example, “home#index”), and hash of options. Possible options are :enabled (to enable profiling of particular block) and :limit (limit in seconds to filter out fast code fragments).

Method start returns an instance of profiler, which will be used to print check points. It contains two useful methods: debug (to display custom message) and progress (to display a message along with time spent since last checkpoint). Both methods define a new checkpoint.

To simplify usage, let’s create a helper:

1
2
3
4
5
6
7

module Kernel
def easy_profiler(name, options = {})
yield EasyProfiler::Profile.start(name, options)
ensure
EasyProfiler::Profile.stop(name, options)
end
end

And now example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

class HomeController < ApplicationController
def index
easy_profiler('home#index', :enabled => profile_request?, :limit => 2) do |p|
p.progress 'logged in user home page'

@top_videos = Video.top(:limit => 10)
p.progress 'find top videos'

@categories = Category.all(:order => 'name DESC')
p.progress 'build categories list'

@tag_cloud = Tag.tag_cloud(:limit => 200)
p.progress 'build tag cloud'

@featured_videos = Video.featured(limit => 5)
p.progress 'find featured videos'

@latest_videos = Video.latest(:limit => 5)
p.progress 'find latest videos'

@profiler = p
p.debug 'VIEW STARTED'
end
end

private

# Method returns +true+ if current request should ouput profiling information
def profile_request?
params['_with_profiling'] == 'yes'
end
end

and view:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

<div id="top_videos">
<%= render :partial => 'top_videos' %>
<% @profiler.progress 'top videos render' %>
</div>

<div class="tabs">
<ul id="taxonomy">
<li><a href="#" id="categories" class="current">Categories</a></li>
<li><a href="#" id="tags">Tags</a></li>
</ul>
<div class="categories_panel">
<%= render :partial => 'categories' %>
<% @profiler.progress 'categories render' %>
</div>
<div class="categories_panel hidden">
<%= render :partial => 'tag_cloud' %>
<% @profiler.progress 'tag cloud render' %>
</div>
</div>

<div class="box">
<div id="latest">
<%= render :partial => 'videos', :videos => @latest_videos %>
<% @profiler.progress 'latest videos render' %>
</div>
<div id="featured">
<%= render :partial => 'videos', :videos => @featured_videos %>
<% @profiler.progress 'featured video render' %>
</div>
</div>

As you can see from this example, profiler will be enabled only when you pass a _with_profiling parameter with value yes: http://example.com/home?_with_profiling=yes.

That’s all. If you have any question, feel free to post a comment or contact me.

Update: I have created a Rails plugin called easy-prof, which is hosted on GitHub. It’s more powerful and feature complete, so feel free to grab sources and play with it by yourself (check the RDoc documentation at rdoc.info). Do not forget to drop me a line about your feelings.

10 recommendations on using HTML5 today (aka Homo-Adminus Blog 2.0 HTML5ified)

Dmytro Shteflyuk — Mon, 20 Jul 2009 14:00:14 +0000

There was a lot of articles about HTML5 last days, so when Alexey Kovyrin asked me to help him with his new blog design I saw no other choice but using HTML5. There are a lot of new features added since HTML4, and some of them could be used today, like new elements >, >, >, >, >, etc. I think this is a nice addition to the HTML, because these elements add more sense to an unstructured markup. There is a buzzword “semantic” exists to describe this, but I don’t like buzzwords, so I would call it “sense”. So what features we could get from HTML5, that are supported by all modern browsers?

Ok, there is not so much of a difference between HTML4 and HTML5 from web-developer standpoint, just some <div> elements are replaced with more meaningful ones (I’m talking about markup now, not about new APIs. Here is what I mean:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

lang="en">
>
>Homo-Adminus Blog>
rel="stylesheet" href="style.css" type="text/css" media="screen" />

>
>
id="container">
id="page-header">
id="menu">
>
> href="#">Home>>
> href="#">Resume/CV>>
> href="#">Projects>>
> href="#">Photos>>
> href="#">Contact>>
>
>
>
id="page-content">
id="content">
>
>
> href="#">Loops plugin for rails and merb released>>
class="meta">
Posted by Scoundrel under
href="#">Development>,
href="#">My Projects>,
href="#">Networks>
>
>
>
datetime="2008-08-20">Sun 20 />Aug />2008>
>
class="text">
Article content goes here.
>
>
>
Tags:
href="#" rel="tag">github>,
href="#" rel="tag">plugin>
>
>
href="#comments">1 Comment> |
href="#">Bookmark on del.icio.us>
>
>
>
>

id="page-footer">
Copyright © 2009 href="#">Alexey Kovyrin> |
Designed by href="#">Tatiana Kovyrina>
>
>
id="sidebar">
id="categories" class="left">
>>Categories>>
>
> href="#">Admin-tips> (26)>
> href="#">Blog> (2)>
> href="#">Databases> (33)>
>
>
>
>
>
>

In a few words: the only difference is that we use new HTML elements instead of descriptive CSS class names and IDs. I’m not going to describe all new features of HTML5 here, the good introduction could be found here or here.

Here are my recommendations on using HTML5 today:

To enable new HTML5 elements in IE you should use document.createElement — see lines 6-8, explanation is here.
Ensure that new HTML5 elements are block level:

1
2
3
4
5
header, footer, section, nav, article, aside {
margin: 0px;
padding: 0px;
display: block;
}
Use > element if you need to add a block of specific content, like categories list (lines 54-61). Use good old <div> to add a markup element with no specific sense (for example, <div id="container"> on line 11).
Use > element to create any sort of navigation panes (top-level menu, sidebar, footer links). Do not use > to create sidebars — this is wrong!
> and > elements are good candidates to wrap any heading (logo, top level navigation, and almost everything that visually is under the main content) and bottom (article metadata, page footer, bottom links, and almost everything that visually is below the main content) information. You can use these elements inside other elements like >, >, or >, so use them.
New element > contains a block of information inside the main content area. It could be an article or blog post, a comment, a particular book description in bookstore, etc. Be creative and use this exciting element to fill your pages with a sense.
The most controversial element is >. Lots of articles about HTML5 in the Internet suggest to use it for the sidebar navigation, but that is absolutely wrong. Use this tag to take some information out of primary content. Good candidates are footnotes or side notes, quotes, comment author’s avatar, etc. Yes, I’ll repeat: be creative.
Almost everything you do in HTML4 is valid for HTML5: use unordered lists <ul> of list items where order does not matter, use ordered lists <ol> otherwise. Use tables (yes, <table>) for tabular data structures (and not for layout). Paragraphs, emphasizes, quotes, and much more elements should take their own place in your markup.
Use validator to ensure you are doing it right.
Read the code. The good place to start from is HTML5 Gallery.

You can start using HTML5 right now. It’s our future, and we need to know, how would it look like.

PS. Did you notice the nice code syntax highlighting in this blog? I’m using CodeColorer — the best syntax highlighting plugin for WordPress. Try it and you will stay with it forever!

Dmytro Shteflyuk's Home

Advanced Capistrano usage

1. Graceful Passenger restarts

2. Generating deployment stages on the fly in multi-stage environments

3. Campfire notifications

4. Deploy locks

5. Generating servers list on the fly

Submitting a patch to the Open Source project: composite_primary_keys

Step 0. Reproducing the bug

Step 1. Setting up an environment for composite_primary_keys gem

Step 2. Reproducing failing tests inside composite_primary_keys test suite

Step 3. Fixing the bug

Step 4. Committing changes and pulling request

Credits

My top 7 RSpec best practices

1. Use before :all block carefully

2. For each test create exactly what it needs

3. Do not create hundreds of records for a particular spec

4. Do not over-mock

5. Use contexts

6. Create several test suites to speed up your workflow

7. Stop spec_helper from being loaded multiple times

Conclusion

Credits

Weekly Link Dump #2

Weekly Link Dump #1

In reply to latest articles in Thoughtbot blog

Simplifying your Ruby on Rails code: Presenter pattern, cells plugin

Presenter Pattern

Cells Plugin

Scribd open source projects

Table of Contents

Changelog

Creating a simple but powerful profiler for Ruby on Rails

10 recommendations on using HTML5 today (aka Homo-Adminus Blog 2.0 HTML5ified)

1. Use `before :all` block carefully

7. Stop `spec_helper` from being loaded multiple times