Mike Perham

Five Years

Mon, 06 Feb 2017 00:00:00 -0800

Five years ago, I published v0.5.0 of a little project I had great hopes for. Here we are five years later: I’ve gone from selling $50 licenses to negotiating five-figure enterprise deals with the Fortune 500. Five five five!

Gimme Five!

By The Numbers

	1st Birthday	2nd Birthday	4th Birthday	5th Birthday
Downloads	214,300	1,192,259	5,505,145	11,260,039
Stars	2144	3535	5846	7087
Closed Issues	663	1420	1887	2378
Forks	266	563	1003	1247
Closed PRs	228	380	836	938
Versions	44	74	110	127
Customers	25	200	675	726
Employees	0	0	1	1

2016 Year in Review

Sidekiq
- Rails 5.0 support
- Dropped Sinatra dependency
- 16 bug fix releases
Sidekiq Pro
- timed_fetch and super_fetch algorithms
- 17 bug fix releases
Sidekiq Enterprise
- Job data encryption
- Multi-Process with memory monitoring
- Web UI authorization

You Want Five?

Sidekiq 5.0 beta is now ready for testing in non-production environments. The internals have been redesigned to work better with Rails 5.0. It no longer supports older Rubies: Ruby 2.2.2+ only. Read the upgrade notes for more details.

gem 'sidekiq', '5.0.0.beta1'

I still believe Ruby is the most fun, most flexible, highest level language out there for building business applications and Sidekiq is the best framework for processing business transactions. The two are a potent combination.

Here’s to another great five years!

What can I do?

Wed, 01 Feb 2017 00:00:00 -0800

Like many people, I’ve been shaken by the events of the last week. I was raised to believe in freedom for all, the rule of law and the inalienable rights guaranteed to all Americans by the Constitution. But ultimately the law and the Constitution are pieces of paper; it’s up to every one of us to defend those rights against their erosion.

But what can I do? Well, a couple of things:

Join the millions of others and protest.
Donate money to organizations which are defending our rights.
Donate my technical tools to those organizations.

Non-Profit Licensing

I’m offering a free license for Sidekiq Enterprise to nation-wide 501(c)3 non-profits whose mission is to protect our civil rights or the environment. Please email me if you are interested.

Public Relations Blast

Wed, 18 Jan 2017 00:00:00 -0800

Apologies, dear reader, it’s been a while since I blogged; I’ve been busy with many things for the last few months:

We bought a house and moved, kid started kindergarten.
I rewrote Sidekiq Pro’s reliable_fetch into super_fetch, feedback has been quite good with no bugs reported and nice performance improvements noted by the Baremetrics team.
I’ve pushed out 8 patch releases, trying to stabilize Sidekiq 4.2’s new Rails 5 support. Not my finest hour, but on the positive side I wrote an integration test suite to verify Sidekiq’s job execution functionality with Rails 4/5 dev/prod.
I attended Rubyconf 2016 in Cincinnati, it was a high-quality conference as usual from the Ruby Cental folks.
I’ve done 4-5 different interviews on my business and how it’s grown over the last few years.

Business

The last point wasn’t really planned but snowballed. I decided to interview with IndieHackers on a lark and it turns out that several others wanted to interview me based on my success so far.

The net takeaway is that Sidekiq sales are going very well, revenue grew about +75% in 2016, and Sidekiq is now the dominant background solution for Ruby, according to Infinum’s Rubygems analysis.

What’s Next?

Sidekiq 5.0!

Debugging stuck Ruby processes

Fri, 05 Aug 2016 00:00:00 -0700

My Ruby process has stopped doing any work, what’s wrong?

This is an uncommon but occasional problem for many people, especially with large applications using lots of native extensions.

Step 1 - Thread backtraces

A Sidekiq process will print out the backtrace of every thread when you send it the TTIN signal.

kill -TTIN <pid>

Side note: YMMV for other types of Ruby processes: resque, unicorn, puma, passenger, etc. Check the gem documentation or open an issue with the maintainers if it’s not clear how to get backtraces from your process.

Reading these backtraces will tell you what your threads are doing at that instant. Here’s an example:

2016-08-04T16:16:13.535Z 96660 TID-oxpb26trs WARN: Thread TID-oxpbpcmn0
2016-08-04T16:16:13.535Z 96660 TID-oxpb26trs WARN: /Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/connection/ruby.rb:82:in `select'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/connection/ruby.rb:82:in `rescue in _read_from_socket'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/connection/ruby.rb:78:in `_read_from_socket'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/connection/ruby.rb:70:in `gets'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/connection/ruby.rb:362:in `read'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:262:in `block in read'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:250:in `io'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:261:in `read'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:120:in `block in call'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:231:in `block (2 levels) in process'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:367:in `ensure_connected'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:221:in `block in process'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:306:in `logging'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:220:in `process'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:120:in `call'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:209:in `block in call_with_timeout'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:280:in `with_socket_timeout'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:208:in `call_with_timeout'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis.rb:1137:in `block in _bpop'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis.rb:58:in `block in synchronize'
/Users/mike/.rubies/ruby-2.3.0/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis.rb:58:in `synchronize'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis.rb:1134:in `_bpop'
/Users/mike/.gem/ruby/2.3.0/gems/redis-3.3.1/lib/redis.rb:1179:in `brpop'
/Users/mike/src/sidekiq/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
/Users/mike/src/sidekiq/lib/sidekiq.rb:92:in `block in redis'
/Users/mike/.gem/ruby/2.3.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:64:in `block (2 levels) in with'
/Users/mike/.gem/ruby/2.3.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:63:in `handle_interrupt'
/Users/mike/.gem/ruby/2.3.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:63:in `block in with'
/Users/mike/.gem/ruby/2.3.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:60:in `handle_interrupt'
/Users/mike/.gem/ruby/2.3.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:60:in `with'
/Users/mike/src/sidekiq/lib/sidekiq.rb:89:in `redis'
/Users/mike/src/sidekiq/lib/sidekiq/fetch.rb:36:in `retrieve_work'
/Users/mike/src/sidekiq/lib/sidekiq/processor.rb:86:in `get_one'
/Users/mike/src/sidekiq/lib/sidekiq/processor.rb:96:in `fetch'
/Users/mike/src/sidekiq/lib/sidekiq/processor.rb:79:in `process_one'
/Users/mike/src/sidekiq/lib/sidekiq/processor.rb:68:in `run'
/Users/mike/src/sidekiq/lib/sidekiq/util.rb:17:in `watchdog'
/Users/mike/src/sidekiq/lib/sidekiq/util.rb:25:in `block in safe_thread'

Backtraces look like a big, scary blob of text because they are! Reading a backtrace is hard and a learned skill: you will get better at it over time. Pop open a gem with bundle open <gemname> and navigate to the line in the backtrace; getting comfortable with the code in the various gems will help your debugging skills a lot. The key line is here:

/Users/mike/src/sidekiq/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'

This is a quiet Sidekiq thread that is waiting for a job from Redis. It’s paused, doing nothing, but that’s perfectly normal.

Most often with stuck processes you’ll see threads paused in the mysql or postgresql driver, waiting on query results or waiting for a lock.

Step 2 - GDB

If the Sidekiq process doesn’t output anything when you send it a TTIN signal, that’s a sign you’ve got a deeper problem. Almost always the cause is a native extension gem which is performing a long operation without releasing the GVL.

This means Ruby can’t help - we have to drop down into the bowels of the process to understand what is wrong using the GDB debugger. You can attach GDB to a running process like so:

sudo gdb /path/to/ruby/binary [PID]

Now to get the thread backtraces dumped to a text file, run these commands:

(gdb) set logging file gdb_output.txt
(gdb) set logging on
(gdb) set height 10000
(gdb) t a a bt
(gdb) quit

A Sidekiq Enterprise customer recently sent me a GDB dump that I will use as an example. Here’s the first thread:

Thread 32 (Thread 0x3703 of process 40990):
#0  0x00007fff94b87db6 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
#1  0x00007fff98cab728 in _pthread_cond_wait () from /usr/lib/system/libsystem_pthread.dylib
#2  0x0000000102dbd5b3 in gvl_acquire_common ()
#3  0x0000000102db7471 in rb_thread_call_without_gvl ()
#4  0x00000001031feda1 in rsock_ipaddr () from /Users/user/.rbenv/versions/2.2.4/lib/ruby/2.2.0/x86_64-darwin15/socket.bundle
#5  0x00000001031f5c4a in sock_s_getaddrinfo () from /Users/user/.rbenv/versions/2.2.4/lib/ruby/2.2.0/x86_64-darwin15/socket.bundle
#6  0x0000000102dadee3 in vm_call_cfunc ()
#7  0x0000000102d927c9 in vm_exec_core ()
#8  0x0000000102da16d6 in vm_exec ()
#9  0x0000000102da6b03 in invoke_block_from_c ()
#10 0x0000000102d9da80 in rb_yield ()
#11 0x0000000102dbbf0c in rb_thread_s_handle_interrupt ()
#12 0x0000000102dadee3 in vm_call_cfunc ()
#13 0x0000000102d925f4 in vm_exec_core ()

Note that this is a C-level backtrace. We’re lower-level and have less info available but experience helps fill in the gaps. The thread is calling sock_s_getaddrinfo which is a DNS lookup. It looks like it has finished the DNS lookup and is now calling gvl_acquire_common to get the GVL in order to continue executing Ruby code. It is normal for some threads to block, waiting for the GVL, but it is not normal for most threads to be waiting for the GVL. This was the case: dozens of threads were blocked with that exact same backtrace.

When this happens, look for the odd threads. 20+ threads were blocked with the same backtrace as above - you can ignore those. But one or more threads will be doing something else, and indeed there was this backtrace which was completely different:

Thread 10 (Thread 0x2103 of process 40990):
#0  0x00007fff94b892a2 in poll () from /usr/lib/system/libsystem_kernel.dylib
#1  0x00000001035c892d in pqSocketCheck () from /usr/local/opt/postgresql/lib/libpq.5.dylib
#2  0x00000001035c87f9 in pqWaitTimed () from /usr/local/opt/postgresql/lib/libpq.5.dylib
#3  0x00000001035c5fd8 in PQgetResult () from /usr/local/opt/postgresql/lib/libpq.5.dylib
#4  0x00000001035c62be in PQexecFinish () from /usr/local/opt/postgresql/lib/libpq.5.dylib
#5  0x00000001035c31bc in PQsetClientEncoding () from /usr/local/opt/postgresql/lib/libpq.5.dylib
#6  0x000000010359d2a7 in pgconn_set_default_encoding () from /Users/user/.rbenv/versions/2.2.4/gemsets/xxx-v1.1/extensions/x86_64-darwin-15/2.2.0-static/pg-0.18.4/pg_ext.bundle
#7  0x00000001035990b7 in pgconn_init () from /Users/user/.rbenv/versions/2.2.4/gemsets/xxx-v1.1/extensions/x86_64-darwin-15/2.2.0-static/pg-0.18.4/pg_ext.bundle
#8  0x0000000102daa929 in vm_call0_body ()
#9  0x0000000102da9d82 in rb_call0 ()
#10 0x0000000102cbc969 in rb_class_new_instance ()
#11 0x0000000102dadee3 in vm_call_cfunc ()

Note the long pathnames at the end of the middle lines - they indicate we’re executing within a native extension: the pg gem which then calls into libpq, postgresql’s native C API. The pg gem is calling PQsetClientEncoding which looks like it is making a network call since I see terms like “Socket” and “poll”.

Now we find the root issue: there’s no GVL release in the backtrace. MRI’s API is excellent in this regard: you should see rb_thread_call_without_gvl in the backtrace to indicate that the thread is not holding the GVL. Until that network call finishes, the entire process is locked up and can’t do anything. Most of the time you won’t notice because the network is usually pretty fast but in the case where the network is stalled, work grinds to a halt.

All of this assumes you can use GDB in your production environment. If you’re on Heroku or a similar PaaS, try to reproduce the issue on your local machine or a Linux VM.

Conclusion

Performing a network operation while holding the GVL is a huge bug and should be fixed. This is a current pg bug in <= 0.18.4, filed and acknowledged.

Getting thread dumps from a stuck process is critical to diagnosing the root cause. Reading and understanding those backtraces, both in Ruby and in GDB, is a useful skill to have when debugging issues. As you read more code, you’ll get better and faster at diagnosing problems. It took me about 30 seconds to determine the problem in the GDB dump above.

If you are a Sidekiq Pro or Enterprise customer and seeing stuck processes, don’t be afraid to gist me your GDB dumps - I’d be happy to help diagnose.

Profiling Crystal on OSX

Fri, 24 Jun 2016 00:00:00 -0700

How do I profile my program to determine where it is slow?

This is one of the first questions any developer asks after building a non-toy program. Crystal has a reputation for being quite fast but every language has tricks and optimizations we miss.

Since Crystal uses the LLVM compiler suite, we can reuse a lot of the infrastructure which knows about LLVM-compiled binaries. Net result: the OSX Developer Tools include a beautiful profiler that works with Crystal binaries out of the box - so awesome! If you have XCode installed (and if you are reading this blog, it’s very likely you do), you can profile a Crystal binary right now, here’s how.

Instruments

Instruments.app is the OSX profiling tool. We’re going to instrument a command line run and then view the results in its GUI. I assume you have Crystal installed; if not, run this:

brew update
brew install crystal-lang

Now we need an app to profile. Let’s make a toy app that doesn’t do much of anything; put this code in app.cr:

def foo(i)
  "mike" + i.to_s
end

10_000_000.times do |x|
  foo(x)
end

Compile and run in the profiler:

$ crystal compile app.cr
$ instruments -t "Time Profiler" ./app
Instruments Trace Complete (Duration : 4.458741s; Output : /Users/mike/instrumentscli0.trace)
$ open instrumentscli0.trace/

You should now see a lovely UI with a totally rad tree view where you can drill down into the trace to see where your code spent its time. Looks like 80% of my app’s time was spent in Int32#to_s and String#+, not exactly shocking but this is a toy example.

Note that I didn’t use --release flag with the compiler. This was a choice I made for this blog post; the traces are a LOT easier to understand without release optimizations because LLVM doesn’t inline method calls so it’s easier to drill into the code execution. You should profile with the –release flag when profiling your own non-trivial code so you are profiling the same binary as you run in production.

Also note that Instruments supports a lot more modes that just the Time Profiler - it can track memory allocations, syscalls, and many other aspects. Play with it and see what modes are useful to you.

In conclusion, profiling Crystal code is super easy due to Crystal leveraging the LLVM compiler. We can use LLVM-standard tools rather than needing custom profiling APIs and runtime support.

Test Driving Sidekiq and Crystal

Tue, 14 Jun 2016 00:00:00 -0700

It’s alive! I’ve finished the initial port of the three core pieces of Sidekiq, the client API, the job execution engine and the Web UI, to Crystal. Let’s assume we have a Ruby app using Sidekiq.rb and integrate Sidekiq.cr to run beside it.

Getting Started

Sidekiq.cr has a Getting Started wiki page which walks you through the basics of setting up a new Crystal app. Follow those directions. Once complete, you should have a Worker named Sample::MyWorker that we can call. We’re going to build the Sidekiq.cr binary and run it with ./sidekiq -q crystal. Sidekiq.cr is now listening for jobs on the crystal queue only. Any Ruby-based jobs should use other queues so the Crystal worker will never see them - exactly what we want.

We’ve defined one Worker in the Crystal app:

module Sample
  class MyWorker
    include Sidekiq::Worker

    def perform(name : String, count : Int64)
      count.times do
        logger.info "Hello, #{name}!"
      end
    end
  end
end

Open up an IRB console and run this:

require "sidekiq"
Sidekiq::Client.push("class" => "Sample::MyWorker", "args" => ["Ruby", 3], "queue" => "crystal")

Immediately you should see Sidekiq.cr print out three lines:

2016-06-13T21:03:21.676Z 3936 TID-223icg0 JID=896704d437d398abdf13eb06 INFO: Start
2016-06-13T21:03:21.676Z 3936 TID-223icg0 JID=896704d437d398abdf13eb06 INFO: Hello, Ruby!
2016-06-13T21:03:21.676Z 3936 TID-223icg0 JID=896704d437d398abdf13eb06 INFO: Hello, Ruby!
2016-06-13T21:03:21.676Z 3936 TID-223icg0 JID=896704d437d398abdf13eb06 INFO: Hello, Ruby!
2016-06-13T21:03:21.676Z 3936 TID-223icg0 JID=896704d437d398abdf13eb06 INFO: Done: 0.000121 sec

Nearly identical to Sidekiq.rb but look at that execution time: 121µs! In real world code, I’m seeing Crystal execute 5-10x faster than MRI.

So… got tons of little jobs? Got compute-heavy jobs? Crystal and Sidekiq.cr might be the solution you’re looking for! Got questions? Find a bug? Open an issue.

Sidekiq for Crystal

Wed, 25 May 2016 00:00:00 -0700

Sidekiq is a popular job framework for Ruby. Now we’re bringing it to Crystal!

Why Crystal?

I wanted to use a language that was a good complement to Ruby. Its syntax is similar enough to Ruby that I can reuse a lot of code but it adds a huge leap in performance. In summary:

very similiar syntax to Ruby
at least 3-5x faster than Ruby 2.3 on most code
at least 3x smaller in memory footprint
statically typed
compiles to a single, 1MB binary! Deployment is easy.
comes with a large, useful standard library

In other words, Ruby is friendly, flexible and works for most usecases while Crystal is fast and efficient for those usecases where performance is paramount. Use each where appropriate.

How productive is it? I started this project from scratch not knowing the language at all a week ago and had the core job processor running in 3 days.

Gimme!

The project resides at mperham/sidekiq.cr. To get started:

brew update
brew install crystal-lang
# brew install redis
git clone git://github.com/mperham/sidekiq.cr.git
cd sidekiq.cr
crystal deps
make

If you have Redis and the sidekiq gem installed, you can run the benchmarks:

brew install redis
gem install sidekiq
make bench

This is the result for me: with zero optimization on my part, Crystal is 3.6x faster and 3x smaller. To process 100,000 noop jobs:

Runtime	RSS	Time	Throughput
MRI 2.3.0	50MB	21.3	4,600 jobs/sec
MRI/hiredis	55MB	19.2	5,200 jobs/sec
Crystal 0.17	18MB	5.9	16,900 jobs/sec

The codebase is a trainwreck though, right?

The code is shockingly similar to Ruby in many cases. Take a gander at this testcase:

require "./spec_helper"

class MyWorker
  include Sidekiq::Worker

  def perform(a : Int64, b : Int64, c : String)
    #puts "hello world!"
  end
end

describe Sidekiq::Worker do
  describe "client-side" do
    it "can create a basic job" do
      jid = MyWorker.async.perform(1_i64, 2_i64, "3")
      jid.should match /[a-f0-9]{24}/
      pool = Sidekiq::Pool.new
      pool.redis { |c| c.lpop("queue:default") }
    end

    it "can schedule a basic job" do
      jid = MyWorker.async.perform_in(60.seconds, 1_i64, 2_i64, "3")
      jid.should match /[a-f0-9]{24}/
    end

With the exception of a few type hints, that’s identical to Ruby.

Cool, just gonna push this to production…

Whoa, this project is alpha. Hold off porting your nuclear reactor controller code for another week or two, ok? Major functionality is missing, (notably the data API and Web UI), the test suite is still baking, etc. Take it for a test drive and let me know how it goes for you.

Looking for other libraries written in Crystal? Check out the CrystalShards listing. AwesomeCrystal is another curated list of resources. You can find database drivers, web frameworks, etc.

What about Sidekiq Enterprise?

Based on demand, I will port the Sidekiq Pro and Enterprise functionality to Crystal. If you are interested, email me.

Conclusion

Let’s make Sidekiq.cr amazing, try it out and help us improve it!

Serving your own Commercial Rubygems

Tue, 17 May 2016 00:00:00 -0700

It’s not well known but with Rubygems and Bundler, you can distribute access-controlled commercial Rubygems. My gems, Sidekiq Pro and Sidekiq Enterprise, are the most well known example but I’m not a wizard: anyone can distribute gems to a limited set of customers. Here’s how I do it.

A $5/mo DigitalOcean instance running Apache. No database, no app server. I have a set of bash scripts which can build a server in 5 minutes.
To serve gems, you need to have Apache serve a directory of static files. You’ll need a VirtualHost entry with something like this:

  <VirtualHost *:443>
    ServerName my-server.com
    DocumentRoot /var/www/gems

    SSLEngine on

    <Location />
      AuthType basic
      AuthName "private area"
      AuthBasicProvider file
      AuthUserFile /var/www/gems-passwd
      Require valid-user
    </Location>

To control access to those gems, you need a /var/www/gems-passwd file which contains your user / password pairs. The username / passwords are a hash based on the customer’s email address. The contents look like this:

1a7fe3bd:$apr1$0RtyG.1v$D/o5n25i2YmPiTOGAPPI21
d41be46c:$apr1$.dFnnB5F$Ne2YtTP12q9iM9/ZQvoL01
09120f77:$apr1$ERLqbLvS$El2Llj6MJlWDWQnJKcrR80

The command to generate an entry is htpasswd -nb user pwd.

I have a /var/www/gems directory which looks like this:

$ pwd
/var/www/gems
$ ls -l
total 36
drwxrwxr-x 2 mike adm  4096 Apr  5 16:25 gems
-rw-rw-r-- 1 mike mike   65 Apr  5 16:25 latest_specs.4.8
-rw-rw-r-- 1 mike mike   82 Apr  5 16:25 latest_specs.4.8.gz
-rw-rw-r-- 1 mike mike   98 Apr  5 16:25 prerelease_specs.4.8
-rw-rw-r-- 1 mike mike  100 Apr  5 16:25 prerelease_specs.4.8.gz
drwxrwxr-x 3 mike adm  4096 Apr  5 16:25 quick
-rw-rw-r-- 1 mike mike 1307 Apr  5 16:25 specs.4.8
-rw-rw-r-- 1 mike mike  251 Apr  5 16:25 specs.4.8.gz

When you buy Sidekiq Pro, Stripe sends my server a webhook. I have a rake task which looks up the new customer’s email address, adds a new user/pwd to the end of the gems-passwd file and then sends them an email with directions to access the Sidekiq Pro gem:

Put this in your Gemfile:

    source "https://my-server.com" do
      gem "sidekiq-pro"
    end

Run this bundle command:

    bundle config my-server.com username:password

Run `bundle install` and you're done!

In addition to the passwd file, the server has a CSV file with email,username,password so I can look up the Apache username for a customer’s email and regenerate the passwd file from scratch if necessary.
If a subscription runs out, Stripe sends the opposite webhook; the server removes them from the CSV and passwd so they can no longer access the gem server.
I have rake tasks which can push a new gem version, add/remove a customer, send email, etc.
Pushing a new gem version looks like this, note the generate_index call:

task :push => :release do
  require 'net/ssh'
  require 'net/scp'
  require 'sidekiq/pro/version'

  ver = ENV["VERSION"] || Sidekiq::Pro::VERSION

  Net::SCP.start("my-server.com", "user") do |scp|
    scp.upload!("pkg/sidekiq-pro-#{ver}.gem", "/var/www/gems/gems")
  end

  Net::SSH.start("my-server.com", 'user') do |ssh|
    puts ssh.exec!("gem generate_index --directory /var/www/gems")
  end
  puts "Released Sidekiq Pro #{ver}"
end

The Stripe webhook integration is really nice as it automates 90% of my business BUT don’t bother until you have sufficient sales to make it worth your while. For the first two years, I added customers manually when I got the “You just got a sale!” email. The hard part is dealing with churn - Stripe doesn’t email when a subscription ends so you’ll need to automate that or at least send yourself email upon that webhook.
For redundancy, I run two servers at a time so a replica is always available. I switch out servers every six months so software versions on the server stay reasonably current: create new replica, old replica is promoted to primary, old primary is killed. The servers are sync’d with BitTorrent Sync, which is the only “shiny” tech I use but I’ve found nothing better to keep a directory in sync between two servers in near real-time.

End Result

When I get a sale, my “Sidekiqbot” script sends me an email that the customer was set up. It feels pretty awesome to know the onboarding is completely automated.

That’s it: Apache, CSV and some basic Linux/Ruby scripting. It took me weeks to develop all of this but there’s nothing super difficult about it. I try to keep the server as simple as possible so I can focus on Sidekiq development and support, not operations. After all, you’re buying what’s in the gems, not what serves them. This simplicity has real benefits: it’s cheap ($10/mo for two servers), easy to maintain and reliable. If you are thinking about following in my footsteps, I hope this blog post helps light the way.

This article is part of a larger group on the technical nitty gritty of a software business. Related articles are Charging for your Open Source (discussion on the money and legal aspects) and CGI: Ruby’s bare metal (discussion on handling Stripe webhooks).

Postscript

I became a member of Ruby Together because I rely on Bundler and the Rubygems infrastructure. The fact that we can do this at all is the beauty of Bundler and Rubygems: they are easily federated using HTTP and static files only. As an alternative, consider npm; it requires CouchDB, a full copy of the npm dataset to serve and only supports a single package server. This design is much more complex, requiring a much heavier server, and makes it effectively impossible for an individual to run an npm server due to the operational costs and maintenance required. I encourage anyone whose business depends on Ruby to join.

Distributed Locking with Redis and Ruby

Mon, 25 Apr 2016 00:00:00 -0700

It can happen: sometimes you need to severely curtail access to a resource. Maybe you use a 3rd party API where you can only make one call at a time. To handle this extreme case, you need an extreme tool: a distributed lock.

Distributed locks are dangerous: hold the lock for too long and your system throughput plummets. They can easy become a major chokepoint for your app’s performance and scalability.

Recently a blog post talked about using Redis for distributed locking with Sidekiq. I tried the code and it didn’t even work. It did however give me the idea to test Sidekiq Enterprise’s Rate Limiting API, which provides a flexible “concurrent” limiter, against other rubygems which provide a similar lock.

Please Note: I’m not talking about Redlock and other algorithms that provide fault-tolerant locking via distributed consensus. Those algorithms are slower and much harder to get correct; I would never trust myself to write one (or anyone else that’s not a Computer Science Ph.D). In this post, I’m talking about using a single Redis instance to coordinate many worker processes distributed across many machines. This is sufficiently safe and robust for most businesses.

The Setup

I tested four different distributed lock gems, including sidekiq-ent. With any of them we can create a distributed lock which ensures our system executes a block of code exclusively, even with dozens of processes. One thing to understand: sidekiq-ent’s Rate Limiting API does not need to run within a Sidekiq process - it can be used in any Ruby process: puma, unicorn, passenger, sidekiq, etc.

All locking libraries provide similar semantics. You define:

how long should the code wait for the lock before giving up?
how long before the lock times out?

The lock has to have a timeout as that’s the only way to recover from a process crash while holding a lock. Libraries “wait” in two different ways: redis-semaphore and sidekiq-ent block, efficiently waiting to be notified when they can take the lock, the other two gems poll regularly, forcing an unfortunate tradeoff: polling more often means slamming Redis with unnecessary work.

The Test

I created a benchmark exercising all four APIs. The code executes 100 “jobs” using 25 threads. Each job sleeps for 0.1 sec while holding the lock, meaning that a perfect run will take 10.0 sec. Gist of the actual benchmark code here.

sidekiq-ent
  0.110000   0.100000   0.210000 ( 10.433794)
redis-semaphore
  0.150000   0.150000   0.300000 ( 10.487963)
pmckee11-redis-lock
  0.460000   0.550000   1.010000 ( 10.718958)
ruby_redis_lock
  0.280000   0.250000   0.530000 ( 11.655952)

The third column shows you the number of seconds actually running on the CPU; sidekiq-ent’s limiter used 0.21 seconds of CPU time, the others varied from 0.3 to 1.0 seconds.

The theoretical perfect runtime is 10 sec, 100 jobs * 0.1 sec sleep so sidekiq-ent adds about 4% overhead. The latter two gems added notably more overhead. Note in the gist, I had to modify pmckee11-redis-lock to disable exponential backoff, otherwise it would die with a timeout after several minutes.

Metrics

Unfortunately the other three libraries give you no insight into actual lock usage while sidekiq-ent’s concurrent limiter offers real-time metrics so you can understand how the lock is performing – it can answer questions like:

how heavily is this lock contended?
is the lock ever timing out?
how often is the lock granted immediately vs forcing your code to wait?

You can read the metric definitions in the wiki. Here’s the UI:

What have we learned?

The other libraries give you the basics of a distributed lock but two are lacking in performance and all are missing the metrics necessary to debug problems. Some good things about Sidekiq Enterprise’s concurrent limiter:

it provides the highest performance distributed lock for Redis
it blocks, it does not poll or sleep, so it won’t slam Redis with superfluous requests or burn CPU
it can limit access to N callers, not just 1
it provides much better visibility with real-time metrics about limiter usage

If you are using Sidekiq today, the Enterprise upgrade will drop right in. Sidekiq Enterprise starts at $1950/yr, get a quote in seconds here.

Kill Your Dependencies

Tue, 09 Feb 2016 00:00:00 -0800

This post talks about Ruby but it’s true of every language community: Python, JavaScript, Java, etc. The scourge of dependencies spares no one.

This is a dependency visualization of every Rails app I’ve ever used. Does any of this sound familiar:

Gemfile with 100s of entries.
Test gems loading in production.
Each Rails process takes 100s of megabytes of RAM.

The Rubygems system is commendable for how easy it makes packaging up Ruby for others to reuse. But that very ease means it’s also quite easy for those gems to pull in other gems transitively, leading to Rails apps which “download the Internet” and have hundreds of dependencies.

When you publish a Rubygem, every one of your dependencies transitively becomes a dependency for any app using your gem. This multiplies the impact of bugs in those gems.

The curious case of mime-types

The mime-types gem recently optimized its memory usage and saved megabytes of RAM. Literally every Rails app in existence can benefit from this optimization because Rails depends on the mime-types gem transitively: rails -> actionmailer -> mail -> mime-types.

In other words, this gem wasn’t used by your app. It wasn’t used by Rails directly. It wasn’t used by ActionMailer directly. It was used deep in the bowels of the ActionMailer implementation and it was using far too much memory. Every single Rails app in existence was using 10MB too much due to this issue.

App Developers, Listen Up!

Every dependency in your application has the potential to bloat your app, to destabilize your app, to inject odd behavior via monkeypatching or buggy native code. When you are considering adding a dependency to your Rails app, it’s a good idea to do a quick sanity check, in order of preference:

Do I really need this at all? Kill it.
Can I implement the required minimal functionality myself? Own it.

If you need a gem:

Does the gem have a native extension? Look for pure ruby alternatives.
Does the gem transitively pull in a lot of other gems? Look for simpler alternatives.

Gems with native extensions can destabilize your system; they can be the source of mysterious bugs and crashes. Avoid gems which pull in more dependencies than their value warrants. Example of a bad gem: the fog gem which pulls in 39 gems, more dependencies than rails itself and most of which are unnecessary.

Lastly, make sure you only load the gem when necessary. Use Bundler’s group support to disable test gems when not testing:

group :test do
  gem 'rspec'
  gem 'timecop'
  # etc
end

Gem Developers, Listen Up!

Part of your job as a library author is to treat your user and their application with respect. You should make an effort to minimize your own dependencies so they don’t load unnecessary code or cause issues in the user’s application. You control your own code but you don’t control your dependencies. Any bug in a dependency of yours becomes a bug that causes stress for your user and their application.

As a gem developer, for each of your gem dependencies do you:

know how much memory each takes?
know how long each takes to require?
know whether it performs any monkeypatching outside of its own module?

Sidekiq, with all of its functionality, has only 3 runtime dependencies: concurrent-ruby, connection_pool and redis.

Die json, die (German for “The json, the”)

So many gems declare a dependency on json, oj, multi_json, or yajl-ruby. There’s so many ossified layers of cruft around JSON processing that only one course of action makes sense: remove it all. JSON has been in the stdlib since 1.9, you don’t need to declare any dependencies at all. Just require 'json' and let Ruby deal with it.

Rails did it, so can you!

Why choose an HTTP client when you can have them all?

Every Rails app pulls in a half dozen different HTTP clients: faraday, rest-client, httparty, excon, typhoeus, curb, etc. This is because various gems use them internally. A Rubygem should never use anything but Net::HTTP internally! Learn the Net:HTTP API, kill those dependencies and stop forcing extra HTTP client gems on your users.

Let’s say you want to offer an optimized version using curb: ok, but make it optional. Allow the application developer to opt into using curb but net/http should always be the default.

Optimizing Rails 5.0

For the last few weeks, I’ve been working (in tandem with several other developers, hi @_matthewd, @applerebel!) on minimizing gem dependencies in Rails 5.0. Rails 4.2.5 requires 34 gems. Rails 5.0b1 required 55 gems. Rails 5.0b2 required 39 gems. I expect Rails 5.0 to require 37 gems or less. So far we’ve removed Celluloid, EventMachine, thread_safe, and json.

Unfortunately there’s no more low-hanging fruit. I’d love to drop Nokogiri, it’s such a huge dependency with a massive native extension component, but there are some non-trivial dependencies on it. Oga is a nice, simpler alternative. If you ship a gem which depends on Nokogiri, consider making it optional and defaulting to REXML (I know, but at least it’s in stdlib) or Oga instead.

Be Part of the Solution

I can help with Rails 5.0 but I can’t fix every gem. If you are a gem developer, audit your own dependencies and remove as many as you can. If you’re an app developer, take a look in your Gemfile and see if you can find a gem or two to remove. Simplify, simplify, simplify.

As an example, I think it’s possible for the Stripe gem to remove both of its runtime dependencies.

Rules to Remember

Some software engineering rules:

No code runs faster than no code.
No code has fewer bugs than no code.
No code uses less memory than no code.
No code is easier to understand than no code.

Kill those dependencies. Your gems and apps will be better for it.