Tomas Varaneckas

Always check your new server IP addresses

Tue, 29 Nov 2016 16:35:18 +0000

Me and my friend have joined forces to create a side project - fitnessblip.com. The biggest mistake was to get a $5 DigitalOcean droplet and use it for our production without checking the IP address first.

Blocked by Google SMTP

After setting up the website and running it in production for a while, I thought, hey, this $5 droplet is using barely no resources, why can’t I slap Postfix on it and make it send emails? And so I did, just to notice that test emails to my Google inbox have bounced with following error:

Action: failed
Status: 5.7.1
Remote-MTA: dns; gmail-smtp-in.l.google.com
Diagnostic-Code: smtp; 550-5.7.1 [<OUR IP ADDRESS HERE> 1] Our system has
detected an unusual rate of 550-5.7.1 unsolicited mail originating from
your IP address. To protect our 550-5.7.1 users from spam, mail sent from
your IP address has been blocked. 550-5.7.1 Please visit 550-5.7.1
https://support.google.com/mail/?p=UnsolicitedIPError to review our 550
5.7.1 Bulk Email Senders Guidelines. n84si23842177qkn.163 - gsmtp

Okay. That explains a lot (I was scratching my head why Pinterest keeps blocking us at that point). I wrote both Google and DigitalOcean about this. DigitalOcean support was amazing and fast, they offered to recreate the droplet on new IP, and so I did. It helped, Google was no longer blocking the emails from new IP.

However, Pinterest was still blocking us, since they apparently blacklisted the domain name because it was once hosted on spammer IP.

Blocked by Pinterest

I was testing Pinterest integration and noticed that after few days of pinning our website got blocked completely. When trying to pin, it said “Sorry, we blocked this link because it may lead to inappropriate content”. What inappropriate content? I opened a ticket at help.pinterest.com, explaining that I did nothing wrong, and asking why our perfectly legit website could be blocked as “inappropriate content”.

Then the fun started. My first ticket got closed without response. Just closed a couple of days after it was opened. I then received the standard Zendesk follow up email asking “How would you rate our customer support?”. I rated it bad, obviously, and opened another ticket.

Two days later, my second ticket was closed, also with no response, with another “How do you rate our customer support?” email. At that point I already knew that we have hosted our website on IP address that previously belonged to spammers. Since then we have changed the IP and enabled CloudFlare, but Pinterest kept blocking the site.

I gave it a week and tried contacting their support again. Again, got no response, again asking to rate the support. At that point I got so pissed that I registered pinterestcustomersupport.com domain with hopes to put something funny in there, which I still did not do. I am willing to give the ownership of this domain to Pinterest for free, if they help me solve my ticket, how about that?

I also started reaching out to Pinterest through social networks, since there seemed to be no way to contact them though help.pinterest.com. And that was my final mistake.

Dear @pinterest, you need to fix your broken #customersupport. It's impossible to get an answer, tickets get closed without reply.
— Fitness Blip (@FitnessBlip) November 25, 2016

Also, while researching ways to reach Pinterest, I found that their customer support is really, and I mean really horrible. How can such a successful company neglect customer support so much and get away with it? Who knows.

Banned by Facebook

I never used Facebook much aside from integration development tasks at work, and managing business websites of my side projects. Therefore I only had a dummy development account, which unfortunately didn’t have my real name on it, even though I used it for years. And I was naive enough to use that account to post comments under Pinterest’s Facebook posts asking for help and complaining about their broken customer support.

Of course, people at Pinterest flagged my comments, and since I didn’t have much activity on my development Facebook account, Facebook has banned me on premise that my account was created for harassment. If I wanted to unblock my account, Facebook had an appeal process, which consists of giving them a copy of your documents that prove that blocked account is yours. Here I had a major problem. My account was named “Tom Warr”, which is just a shortened variant of my full name. The test account was also registered using an email address that no longer works, and it was not my main email address.

I said myself, oh well, there is tough chance I’ll get my account back, so I’ll just register a new, real account under my main email address, which is my full name at gmail.com. So I did that, created an account, entered my full name, real birthday, and then they suggested me a bunch of friends that they somehow know I’m familiar with, all that.

But then my newly created account got merged with my developer account and got blocked again. When I tried to log in with my main email address, it displays that I'm “Tom Warr” - the name from my blocked development account.

So far I had a 12 emails long conversation with Facebook customer support (which is way better than Pinterest one, because they at least do respond, and do it reasonably fast), though things are going nowhere.

It seems like I’m not welcome to use Facebook anymore, because even though I gave a copy of my passport, drivers license and personal identification card, my appeals to get the account back were refused, because “the data on identity documents do not match the name and birthday on blocked account”.

This sad story ends here. An unfortunate event of starting a project on IP address which belonged to a spammer put both the project and myself in jeopardy. Trying to prove something to customer support of huge corporations is not a battle a nobody like myself can win. And I cannot register a Facebook account and use their developer tools either. I also lost access to all my Facebook Pages and Apps.

I hope others can learn from my mistakes. Check every IP address before putting your project on it in production. Oh, and also, don't keep your apps and business pages on Facebook account that has a different full name and birthday than your passport does.

Overriding Unicorn worker process names

Sat, 08 Feb 2014 07:17:07 +0000

In my previous post I covered troubleshooting unicorn processes. This time I'll share one more technique that can make your devops life easier.

Default unicorn worker process names suck

When you look at server process list using ps auxf or something similar, unicorn looks like this:

... unicorn master -c /etc/app/config.rb -E production -D
...  \_ unicorn worker[0] -c /etc/app/config.rb -E production -D
...  \_ unicorn worker[1] -c /etc/app/config.rb -E production -D
...  \_ unicorn worker[2] -c /etc/app/config.rb -E production -D

Not very useful, and quite redundant - your workers inherit the argument line from master process. There's no way to tell what they're doing at given moment.

Overriding unicorn worker process names in Ruby on Rails app

How about something like this?

... unicorn master -c /etc/app/config.rb -E production -D
...  \_ unicorn worker[0] 66.249.78.123: last status: 200. Waiting for req.
...  \_ unicorn worker[1] 91.9.93.234: GET /items/3134
...  \_ unicorn worker[2] 62.227.53.32: POST /auth/login

It shows the IP of your user along with current request method and path. And in case worker is slacking, last user IP and response status. Now, let's make ourselves a piece of Rack Middleware that will do the above.

Create /app/middlewares/set_unicorn_procline.rb:

class SetUnicornProcline
  def initialize(app)
    @app = app
  end

  def call(env)
    user_ip = env['HTTP_X_FORWARDED_FOR'] || env['REMOTE_ADDR']
    request_info = "#{env['REQUEST_METHOD']} #{env['REQUEST_PATH']}"
    worker_line = $0.split(']')[0] + "] #{user_ip}"
    set_procline "#{worker_line}: #{request_info}"[0..200]
    status, headers, body = @app.call(env)
    set_procline "#{worker_line}: last status: #{status}. Waiting for req."
    [status, headers, body]
  end

  def set_procline(value)
    $0 = sanitize_utf8(value)
  end

  def sanitize_utf8(string)
    return string.force_encoding('utf-8') if string.valid_encoding?
    string.chars.select(&:valid_encoding?).join.force_encoding('utf-8')
  end
end

Then enabled it in config/application.rb:

...
module <YourApp>
  class Application < Rails::Application
    ...
    require File.join("#{Rails.root}/app/middlewares/set_unicorn_procline.rb")
    config.middleware.use SetUnicornProcline
  end
end

It should work with Ruby >= 1.9.3 and Rails >= 3.2.x. The middleware also takes care of broken encoding that would otherwise cause $0.split to crash on next request after malicious REQUEST_PATH was provided.

Now run watch 'ps auxf | grep unicorn' and enjoy the show.

Ruby Threads, Unicorn, and You

Sat, 18 Jan 2014 18:50:21 +0000

If you are working with highly intensive Unicorn deployments, you will be familiar with following problems.

How to find out what Ruby processes are doing?

Occasionally some Ruby processes are running pedal to the metal, possibly burning CPU like crazy, you know they shouldn't be, and you just can't tell what's happening. They are getting stuck, but can't tell where or why - it's frustrating.

Tracing sometimes works. Run strace -s 999 -fp <pid> on a suspicious process, and you may see some useful output. It may end with something like this, without any movement:

[pid  7990] select(18, [16 17], NULL, [7 8], {133, 947558} <unfinished ...>
[pid 17728] select(19, [14 18], NULL, [7 8], {22, 500000} <unfinished ...>

It's selecting something from file descriptor 19, and you can try to find out what it is:

$ lsof -p 17728
COMMAND   PID    USER   FD   TYPE             DEVICE SIZE/OFF      NODE NAME
...
ruby    17728    ruby   19u  IPv4          808319634      0t0       TCP web1.infra.net:35682->db3.infra.net:mysql (ESTABLISHED)

In this case it's waiting for MySQL, so you can look at what's happening over there. But sometimes you see that worker is hanging on select from a pipe or socket:

$ lsof -p 17728
COMMAND   PID    USER   FD   TYPE             DEVICE SIZE/OFF      NODE NAME
...
ruby    58610    ruby   12r  FIFO        0,8        0t0 3435272094 pipe
ruby    58610    ruby   13u  sock        0,6        0t0 3435234779 can't identify protocol
ruby    58610    ruby   14w  FIFO        0,8        0t0 3435270598 pipe

If your FD shows a pipe or socket, it's difficult to dig further.

Another case is when Ruby process is endlessly spilling it's guts into trace output, and you can see a pattern repeating, meaning that there is some sort of infinite loop going on, but it's hard to tell where.

There are more advanced techniques, but they are pretty cumbersome, and you may get your process killed before you manage to find out anything useful.

What Ruby is doing at any given moment?

Wouldn't it be nice if you could be able to find out what line of code your Ruby process is executing at any given moment? Good news is that you can. Combine Signal.trap with Thread.list and Thread#backtrace, and you're set.

Signal.trap('USR2') do
  pid = Process.pid 
  puts "[#{pid}] Received USR2 at #{Time.now}. Dumping threads:"
  Thread.list.each do |t|
    trace = t.backtrace.join("\n[#{pid}] ")
    puts "[#{pid}] #{trace}"
    puts "[#{pid}] ---"
  end
  puts "[#{pid}] -------------------"
end

Place this code in Unicorn's after_fork, Resque after_fork, beginning of Rake tasks, or wherever it fits. When something is stuck or otherwise suspicious, find the pid, and then do:

$ kill -USR2 26749

Take a look at unicorn log. It should show something like this:

[26749] Received USR2 at 2014-01-19 04:28:35 +0000. Dumping threads:
[26749] /srv/app.intra.net/releases/shared/config/unicorn-config.rb:291:in `backtrace'
[26749] /srv/app.intra.net/releases/shared/config/unicorn-config.rb:291:in `block (3 levels) in reload'
[26749] /srv/app.intra.net/releases/shared/config/unicorn-config.rb:290:in `each'
[26749] /srv/app.intra.net/releases/shared/config/unicorn-config.rb:290:in `block (2 levels) in reload'
[26749] /srv/app.intra.net/releases/20140118185816/app/controllers/welcome_controller.rb:8:in `call'
[26749] /srv/app.intra.net/releases/20140118185816/app/controllers/welcome_controller.rb:8:in `sleep'
[26749] /srv/app.intra.net/releases/20140118185816/app/controllers/welcome_controller.rb:8:in `index'
[26749] /usr/local/lib/ruby/gems/2.0.0/gems/actionpack-3.2.16/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
[26749] /usr/local/lib/ruby/gems/2.0.0/gems/actionpack-3.2.16/lib/abstract_controller/base.rb:167:in `process_action'
[26749] 
...
[26749] ---
[26749] -------------------

You can see that it's hanging on sleep in welcome_controller.rb, line 8. Bullseye.

Dumping Ruby threads when killing lazy Unicorn workers

You can't sit all day and watch those processes manually. Unicorn has a timeout configuration option, where it kills workers that run longer than given amount of seconds.

Trouble is that Unicorn sends SIGKILL when terminating stuck workers, and SIGKILL cannot be trapped. Workers get killed, and all you get are these lousy messages:

E, [2013-11-19T12:30:08.905908 #61893] ERROR -- : worker=75 PID:62991 timeout (46s > 45s), killing
E, [2013-11-19T12:30:08.939120 #61893] ERROR -- : reaped #<Process::Status: pid 62991 SIGKILL (signal 9)> worker=75

That's not very helpful. It just tells us that something went wrong, and that the perpetrator got a bullet in the head.

But we can patch things up by overriding Unicorn::HttpServer#murder_lazy_workers. Here's a monkey patch that will send SIGTERM 2 seconds before sending SIGKILL:

class Unicorn::HttpServer
  # forcibly terminate all workers that haven't checked in in timeout seconds.
  # The timeout is implemented using an unlinked File
  def murder_lazy_workers
    next_sleep = @timeout - 1
    now = Time.now.to_i
    WORKERS.dup.each_pair do |wpid, worker|
      tick = worker.tick
      0 == tick and next # skip workers that haven't processed any clients
      diff = now - tick
      tmp = @timeout - diff

      # monkey patch begins here
      if tmp < 2
        logger.error "worker=#{worker.nr} PID:#{wpid} running too long " \
                     "(#{diff}s), sending TERM"
        kill_worker(:TERM, wpid)
      end
      # end of monkey patch

      if tmp >= 0
        next_sleep > tmp and next_sleep = tmp
        next
      end
      next_sleep = 0
      logger.error "worker=#{worker.nr} PID:#{wpid} timeout " \
                   "(#{diff}s > #{@timeout}s), killing"
      kill_worker(:KILL, wpid) # take no prisoners for timeout violations
    end
    next_sleep <= 0 ? 1 : next_sleep
  end
end

Place this code somewhere where it gets loaded on unicorn startup. It can be next to Unicorn's before_fork. You may want to adjust your Signal.trap to be a little more universal:

['TERM', 'USR2'].each do |sig|
  Signal.trap(sig) do
    pid = Process.pid 
    puts "[#{pid}] Received #{sig} at #{Time.now}. Dumping threads:"
    Thread.list.each do |t|
      trace = t.backtrace.join("\n[#{pid}] ")
      puts "[#{pid}] #{trace}"
      puts "[#{pid}] ---"
    end
    puts "[#{pid}] -------------------"
    exit unless sig == 'USR2'
  end
end

Restart your unicorns (this change may require full stop and clean start) and wait for any worker to receive a timeout. When it does, read your logs and rejoice:

E, [2014-01-19T04:17:19.340983 #26722] ERROR -- : worker=5 PID:27033 running too long (45s), sending TERM
[27033] Received USR2 at 2014-01-19 04:17:19 +0000. Dumping threads:
[27033] /srv/app.intra.net/releases/shared/config/unicorn-config.rb:291:in `backtrace'
[27033] /srv/app.intra.net/releases/shared/config/unicorn-config.rb:291:in `block (3 levels) in reload'
[27033] /srv/app.intra.net/releases/shared/config/unicorn-config.rb:290:in `each'
[27033] /srv/app.intra.net/releases/shared/config/unicorn-config.rb:290:in `block (2 levels) in reload'
[27033] ... your code ...

And you can still use kill -USR2 <pid> on your unicorn workers (or any other Ruby process for that matter) to get the backtrace on demand. Just don't send USR2 to unicorn master process expecting to get a thread dump, it will reload your application instead.

Everything is so much simpler when you can pinpoint problems in your code.

How to get Hubot on HipChat using Hubot Control on CentOS Linux

Thu, 26 Dec 2013 18:36:08 +0000

What is Hubot and why should I want it?

Hubot is an awesome chat bot open sourced by GitHub. It's easily extendable using CoffeeScript, and you can run it on pretty much every popular messaging service using a variety of adapters. Hubot is primarily a company chat bot that can be incredibly useful in daily operations - take a look at ChatOps at GitHub screencast to get the idea.

However, Hubot can be quite tricky configure and operate, and that's where Hubot Control comes in - it allows you to configure and manage multiple Hubot instances using a web interface.

This article will guide you through running Hubot on HipChat, but it is pretty similar to use any other adapter in same fashion, with exception of Skype, that requires you to jump some additional hoops.

Create a HipChat account for your Hubot

Add a new user from your existing HipChat account. Log in with this new user, enter desired chatrooms. Set your bot's @mention name to @hubot in account settings.

Prepare your system

We will be doing the setup on CentOS 6.4, but since there is nothing too CentOS specific in the steps, you can do the same using corresponding packages for your favorite Linux distribution.

For CentOS, you should have EPEL repo enabled to have access for all required packages.

Install system dependencies

You will need several things to get everything running. Let's make sure you have everything.

# Development libraries for compiling ruby and some gems 
$ yum install -y readline-devel zlib-devel libyaml-devel libffi-devel openssl-devel sqlite-devel

# Node and NPM
$ yum install npm --enablerepo=epel

# Required node packages
$ npm install -g coffee-script
$ npm install -g hubot
$ npm install -g hubot-hipchat

Prepare Hubot Control source

Hubot Control is a plain and simple Ruby on Rails webapp, there is no native packaging available, so we will clone the source code to our /opt directory:

$ cd /opt
$ git clone https://github.com/spajus/hubot-control.git

You may also want to use your own fork of Hubot Control.

Add hubot user

Running things as root is a very dangerous practice, so let's set up a fresh user just for Hubot, pointing it's home directory to /opt/hubot-control, where we just checked out our rails app:

$ useradd -d /opt/hubot-control hubot

Then make sure this new user has all the rights to /opt/hubot-control dir:

$ chown -R hubot:hubot /opt/hubot-control

Install RVM and Ruby 2.0.0

We will install RVM to get desired version of Ruby without messing up the system. As hubot, run:

$ su - hubot

# Will install RVM and Ruby 2.0.0. This will take a while.
$ \curl -sSL https://get.rvm.io | bash -s stable --ruby=2.0.0

# Will load RVM for the first time.
$ source ~/.rvm/scripts/rvm

Afterwards ruby -v should show something like ruby 2.0.0p353 (2013-11-22 revision 43784) [x86_64-linux]/

Configure the database

You can configure Hubot Control to use any database you want - refer to Ruby on Rails configuration guide to do that. In this article we will be using SQLite in development mode, so it will work out of the box. However, to avoid installing PostgreSQL gem, which requires development libraries, we will remove gem 'pg' line from /opt/hubot-control/Gemfile, like so:

$ sed -i "/gem 'pg'/d" /opt/hubot-control/Gemfile

Install Ruby Gems

To be able to start Hubot Controll, you have to install the dependencies. We will do this as hubot user:

$ su - hubot
$ cd /opt/hubot-control
$ bundle install

What to do if some gem fails to install

If some gem will fail to install, you will most likely need to install a development library for a missing native ruby extension to compile properly. Then just try again. For example, if sqlite gem fails and sqlite-devel package is not installed, you will see this error:

An error occurred while installing sqlite3 (1.3.8), and Bundler cannot continue.
Make sure that `gem install sqlite3 -v '1.3.8'` succeeds before bundling.

To fix it, simply run as root:

$ yum install -y sqlite-devel

It should succeed this time, and then run bundle install again, to proceed with the rest of gem dependencies.

Prepare the database

A final step before starting Hubot Control for the first time is to create your database. If it's properly configured, this will succeed, and you will be good to go:

$ rake db:migrate

Run Hubot Control

Now it's time to run Hubot Control for the first time. We will do it in a straightforward way, by running:

$ cd /opt/hubot-control
$ rails server --port=3000

Rails will boot and you will see all the output right in your console. Now point your browser to http://your-server-ip:3000/status, login with admin@hubot-control.org / hubot, and if you see a green heading saying You can run Hubot - you're almost there. If something is missing, you will see the cause along with instructions what to do.

When you will get everything running, you can start Hubot Control as a daemon using:

$ unicorn_rails -p <port> -D

Create a Hubot

To create a new Hubot, click Add Hubot and fill out the form. Set Title and Name to hubot, Adapter to hipchat and click Create. Now wait patiently for a while until you see a congratulatory screen with log output. You have just created a vanilla hubot instance. Click Control hubot button to proceed with further configuration.

Configure your Hubot

There are some final steps to take before your Hubot can connect to HipChat.

Variables

Click Configure button and go to Variables tab. This is where you have to add some configuration variables for the HipChat adapter. It should look something like this:

{
  "HUBOT_HIPCHAT_JID": "12345_123456@chat.hipchat.com",
  "HUBOT_HIPCHAT_PASSWORD": "MySuperSecretPassword",
}

Set the JID to the Jabber ID shown on your bot's XMPP/Jabber account settings.

package.json

In package.json tab you have to add hubot-hipchat dependency to dependencies block. You can find the exact version of the package in https://npmjs.org/package/hubot-hipchat.

{
  ...    
  "dependencies": {
    "hubot":         ">= 2.6.0 < 3.0.0",
    "hubot-scripts": ">= 2.5.0 < 3.0.0",
    "hubot-hipchat": ">= 2.6.4 < 3.0.0"
  },
  ...
}

Now hit Save all changes and you are ready to start.

Starting Hubot

Click hubot in the left sidebar and then click Start to start it. Be patient, since it's the first run, all dependencies will be installed, and Hubot Control will simply remain unresponsive. Go to your terminal where you started Hubot Control to check the output. When everything is installed, your Hubot should be running.

Testing Hubot on HipChat

Hubot will respond to @mention name - say @hubot ping in the chatroom and @hubot will respond with PONG. Say @hubot help to get the list of all commands.

You can extend this list by enabling third party scripts or writing your own using Hubot Control.

Surviving in Open Plan Office

Tue, 24 Sep 2013 04:04:42 +0000

Many software development companies choose to have an open-plan office space and unfortunately I happen to work at one. Now, one thing makes me wonder. Why would any software development company want to use open plan office? It looks nice on pictures and you can throw great parties in it, but how does it affect the most important thing why people gather there in the first place - getting shit done?

Just think about this - if a guy has chosen computer science as his way of life, would he rather be more happy and productive hacking on code in a small, quiet, distraction free room, or in a big open space full of noise and loud conversations? Would the guy enjoy people walking behind his back and constantly pulling him out of his sweet matrix of code by asking work related questions that could be asked in chat or by email?

Funny thing, google for open plan office and see the results on first page. You will find so many negative headlines:

Actually, there was not a single headline saying open plan office is great or anything alike, but for some strange reason those who make critical decisions have different beliefs. Let's dig down to the arguments open-plan office fans may put on the table. Those fans obviously don't write code at work, they are courageous business people who like to face their problems head-on, like James Bond:

Offices with walls are dull, boring and old school - we don't want that in our company

Software development is dull and boring for most people, and it's done best in dull and boring rooms where you don't get distracted by all the fun around you. Quoting Wikipedia - Neither open or closed plan offices are perfect for any one situation or individual. The right balance is required. any office design is likely to involve trade-offs for the workers, with some positives and negatives.

When you force developers to work in open space, you are making a trade-off for their productivity and happiness.

Open plan office is more social

Computer science, software development, coding, math, concentration. Social doesn't belong next to these words. GitHub says it's Social Coding, but it's easy to get the meaning wrong. GitHub is awesome, because you can get social when coding in your calm, quiet room, alone with 4 walls, no sunlight, just computer screens glaring gently in your face. That's the social that gets shit done, not a crowd moving in front of your face while you're trying to concentrate on the code.

Developers are introverts who prefer to socialize in digital form, where it's easy to respond by picking your words carefully, by doing a research on the subject, by choosing a right time to get involved in a conversation. They don't want to be social all day long, but they are forced to, against their will. If you're their boss, they probably won't say it in your face, since it could get them fired for bad attitude, but that's the hard truth. Software developers would have chosen to become salespeople if they wanted to be social.

You cannot hide from work in an open office

If you want to hide from work, you can find numerous ways to look busy with any office plan, and if you have employees who want to hide from work, you have a much bigger problem. Those who want to get work done, will do it anyway, question is, will they be happy doing it in the conditions they're given?

And those who would want to pretend they're working hard would most likely make the loudest noise, so everyone would see how involved they are, right? Talking about work does not get shit done. Talking about work next to those who are trying to do it slows them down. Loud mouths should be banned from open space offices.

It enhances communication among employees

Sure, people communicate around developers all the time. Walk across the office in the usual, noisy moment. Most developers will be wearing their headphones, hiding from communication. Sadly, good headphones are an absolute requirement to get any work done in an open-plan office.

Communication in the middle of the office raises the noise level, which raises stress and blood pressure levels of everyone who is trying to concentrate on getting their work done. You're literally killing your developers.

There may be more arguments, why someone would say open-plan is superior to other options, but none of them would be valid when it comes to delivering results of labor that requires high levels of concentration.

Surviving in open plan office

If you are trapped in such an office, there are ways to survive:

Come to work as early as possible. Noise and distractions start around 9-10 AM. If you get to your office as early as 7AM, you could get more work done in 3 hours than in rest of the day.
Do work from home. If your company frowns upon remote work, you can still do it without telling anyone, before or after normal office hours, while coming in a little late or getting out a little earlier. Just be in the office at day time, when the noise is the loudest. This way everyone will think you are getting work done right in the office, while being social and shit. Or do long calm work sessions on Saturdays and Sundays. These are the best, but I have a feeling that no one will be a big fan of that. You can burn yourself out after doing that for a while.
Get some serious noise canceling headphones. Not hipster beats by rapper crap, not anything from Apple store. Look for Studio monitor headphones or DJ oriented products. The price tag will be steep, but you will get what you pay for.
Get a curtain wall. Open-plan fans will want to keep it open, but you can close it again and again. We have one in our office, and it's always closed shut. We need more of those, but I still need to convince people. Hopefully this rant will do it.
Hide in meeting rooms and lounges. It will take a while for those who want to distract you to find you in a secluded meeting room. And often they wouldn't bother to look for you further than your desk.
Reduce chaos by improving company processes to require less direct interactions. Implement Scrum properly, and you will get much more peace and quiet even in open-plan office, since there will be significantly less clueless people trying to find out what to do next, less poking about task statuses, less work given to developers directly out of thin air.
Convince people that remote work is healthy now and then. I used to work in a company where one day a week could be spent working remotely from home, and those were the most productive days - no hard task could resist a one day cracking in a peaceful atmosphere of your choice.
When someone interrupts you, say you are working on something at the moment and ask them to send you an email about it. Promise to respond as soon as you'll be available. I know one person who used to do it, and it worked as a charm. When you reject someone with direct request several times, they start writing emails and come in person only when it's really necessary.
Convince your company to adopt Library rules, like in 37 Signals. All conversations should be either in whisper, or in remote meeting rooms.

It is sad that remote work is not a part of software development culture in most companies. Management almost always relates remote work with slacking or failing at communication, but that's far from being true. There are plenty of high profile companies (i.e.: Automattic), that are wildly successful with remote workers. A well organized team doesn't have to talk face to face to get work done.

Make walls, not noise.

The Law of Startup Lottery

Fri, 22 Feb 2013 05:01:39 +0000

I've been willing to write about this for quite a while, but didn't do it because some people will want to throw stones at me for trying to break their spirit. You don't have to take my word for it if you don't like what I'm saying. I don't claim that my point of view is absolutely correct, so don't blame me for having one.

I love people who love startups. Who work hard and believe that they can change the world. Who are fearless and choose a different path than most of those who are playing it safe. There is something about them that triggers a sense of respect.

I consider myself a startup minded person too. I've been trying to spin off my own business for at least 10 years. I've built software services, created products with optimistic hope that people will find them useful and it will somehow evolve into something I could do for living. I've been getting up at 5AM every day to find the time to code things of my own while having a full time job. I've took breaks from job to create something of my own. But I was always dragged back by the cruel reality. I still read NH nearly every day, stories about new successful software startups always makes my heart beat faster and my pupils dilate.

Like most of things in life, only a small fraction of everything that people create is actually ingenious. Take music, for instance. Everyone can learn to play or compose it, but it takes more than time, stubbornness, dedication and hard work to produce a true masterpiece. And usually one doesn't make it twice. Same goes with technology. Most software products and services that startups produce are just worthless or only a little useful. But good enough doesn't cut it - you have to be the best to succeed.

More bad news for those who are dying to create a hit - you will never create one on purpose. I don't know if there's a law for that, and if it's not, I'm coining it as The Law of Startup Lottery: chances of creating a really successful startup are roughly equal to winning the grand prize of national lottery. And if you took some classes of statistics or have a common sense, you will know that it's not worth the try.

Another interesting observation is that the most relatively successful things I've built were not built with any intention of becoming a success whatsoever. For instance, my most successful part time project was a mirror of JAD Java Decompiler, it's homepage once disappeared from the internet and I had pretty hard time finding a copy, so I decided to put out a mirror for other people to save their time and to keep that great piece of free software alive. It took me 2 hours to do that, and now I'm getting increasingly growing traffic from Wikipedia, StackOverflow, Eclipse Wiki, etc. It's not a huge traffic, since potential audience is just a small fraction of curious Java developers and random hackers, but it's way bigger than anything I've had with my other pet projects. And it was so easy to set up. Brilliant things doesn't have to take much time or strategical planning, you just have to be at the right place at the right time. It's much easier when you do it first.

Read the stories of the greatest startups ever built. How many of them involve accidental luck, and how many were built by intention to become a success?

I've been trying hard enough, spending endless hours of my spare time planning, writing code, contemplating on various ideas, reading books and blogs about startups, trying to apply the learned techniques. It was definitely and rewarding and fun, I've learned a heap of new things that I would otherwise never learn. That was the best part.

But then, ask yourself, why do you do it? It feels bad when you pour your heart into something no one ever wants to use. Over and over again. It's fun to learn, but you can't do it just for the sake of learning, can you?

This is where I put down my hat, at least for a while. I will no longer try to create anything just because it's fun and it may become something more. I'll rather do what I love for those who have won the startup lottery. And if the day comes, I'll create something useful unwillingly, but chances are just one in a billion.

If you're trying hard to become a success, you're doing it wrong. It does not work that way.

Information Diet

Tue, 15 Jan 2013 06:00:41 +0000

There are plenty of lifehacks for keeping up with the information flow. Organizing it, having read later lists, processing it once a day, etc. I've tried many of them, but none of them worked in the long run. When you have a torrent of news coming at you from all directions, it's impossible to keep up.

There are social news sites like Reddit or HN that are supposed to give you handpicked trendy stuff. It's a step forward, but you only get what ticks for the masses, and if it's good, you'll get it more than once - popular content is usually reposted and relinked multiple times - everyone wants to be the one to spread the important news. Next time, before posting a link from HN top page to your Twitter or Google+ account, think again. Internet is redundant enough. Stop sharing same things over and over.

I've taken a few steps to reduce my own digital footprint in this bloated era of useless information. I've stopped using following services:

Facebook: No value whatsoever. Sadly I couldn't just deactivate my account because I'm a developer and have some apps I want to keep alive, so I just locked myself out by changing my password to something random.
Google+: Never understood the need for it.
LinkedIn & all other online career tools: I usually got my jobs as a result of direct recommendation by someone I know in real life. Over ~10 years of use, LinkedIn gave me nothing but lots of recruiter spam. So long, annoying skill endorsements.

I wanted to deactivate my Twitter account too, but kept it for authentication purposes. Sometimes it's the only acceptable type of auth that some app offers. I promise to tweet less though.

And for my own information diet:

No social networks.
No television - haven't been watching it for ages anyway. And a while ago I disconnected the external antenna.
No local news sites. There are some online gazettes that everyone in Lithuania feels obligated to read daily. Sorry, I'm not into politics, local scandals or people bitching about high costs of winter heating.
As few RSS as possible. Nothing that reposts content from other places. Nothing that posts daily.
No pointless random browsing to amuse myself. Although it's a hard habit to get rid of, I'm doing my best.

Reading a book is so much better than reading random crap on the internet.

Life has just become much simpler.

How to Work with Large Rails Codebase Effectively

Sat, 27 Oct 2012 13:58:34 +0000

Welcome to the Real World

A couple of weeks ago I've started working on a pretty big Rails app. After dealing with my lightweight pet projects, a few inconveniences occurred. If you're lazy, just jump to the end to see the summary of the important things I've learned.

App, Y U No start?

An empty rails app starts in 1-2 seconds. My biggest pet project starts in 5 seconds. The big app starts in 25 seconds. That's a guaranteed 25 second overhead for every operation that has to load Rails - console, runner, tests, server. Multiply this overhead by number of times you do those during the day and it can easily build up into a solid amount of time wasted.

Testing woes

All tests run a solid half-hour, and you only see failures in the end. Continuous integration server helps, but at least in the very beginning I'd rather make sure everything works on my machine before pushing my commits to origin.

An when you run tests one by one, you get to wait those extra 25 seconds every time. That can give you cancer.

The Tricky Extras

There is a small zoo of additional stuff that has to be installed and running for the main app to work. That stuff may also run in several instances, one for each environment you want to use (development, test). And when you suddenly realize that something required is not running, you waste more precious time to find what's wrong and to fox it. Basically it's a never ending fight with the environment, until you gradually get used to it and everything starts to work, because just like an experienced puppet master you have an eye for things and grow to feel what strings should be pulled to get back on track.

And of course, those extra dependencies have their own rituals, like you occasionally may have to rebuild the indexes for the full text search server.

The Code Base

Navigating around a huge codebase of a heavily dynamic language is quite tricky, especially in the very beginning.

This is mostly it. The rest is just pure awesomeness!

Lessons Learned

Speeding things up

I've tried profiling, looking for bottlenecks, cutting out routes, removing gems, but nothing gave a significant speed up for Rails boot time. When running individual tests, spork gave a pretty good performance gain, but then a colleague pointed me to something better - zeus. It's a nifty thing that pre-boots various parts of rails and allows rapid execution for all those tasks that used to make me wait for 25 seconds. Running an individual test now takes 5 seconds instead of 30. Great success!

Tame the Zoo with Bootstrap Script

I've found a nice presentation by a GitHubber Zach Holman, which suggested using a script/bootstrap which would handle installing dependencies and more things that hurt newbies. Haven't done it yet, but I'm eventually going to write one.

Use the Right Tools

I started out with RubyMine - thought it would help me get around the code easier. It has a couple of goodies in it's pocket, but you can do nearly everything using good old Vim, proper shell and terminal - and you will have much more fun too. Use ctags to jump around the code like a boss. Lots of people are in love with Sublime, so it's worth checking out.

I've also found some tools that are very helpful for getting to know the application.

Rails Footnotes appends every page with a footer that contains lots of debug information and links that open your text editor with current controller, etc.

Pry is great for debugging. You just add binding.pry anywhere in your code and it stops with an interactive shell. Just like ipdb.set_trace does in Python. You can use pdb, but ipdb is better, but let's not get carried away. Also take a look at pry-nav, because for some weird reason pry alone won't give you the possibility to step around your code, and you should really want that.

Summary (last update: Oct 27, 2012)

Use zeus to get super fast Rails boot time.
Get some inspiration here and write a script for installing/starting/managing your development environment.
If you want to have more fun, don't use a heavy IDE. Use Sublime or vim. Here's my vim config.
Use ctags with your text editor (it should support the integration).
Use Pry and pry-nav gem to debug.

Generating Static Website with Rails 3

Wed, 10 Oct 2012 12:05:09 +0000

Why Rails -> Static HTML is good

So you want to build a static website - a bunch of HTML files, some CSS, images - something you can host anywhere. Yet, you want to reuse your code, whenever you can - shared navigation, partial blocks, etc. It would also be nice to have some basic content management. You could build it with Rails and export a bunch of static html files. My website and this blog is built that way, and it's pretty simple to do. You can also use this technique with other frameworks, like Django.

The Quick Formula

Make HTML friendly routes
Run in production mode on local machine
Grab the static version using wget
Deploy HTML files to your hosting server

HTML Friendly Routes

There is only one rule - your routes must end with .html, so your URLs will look like blog.html. Here is an example route:

#in `routes.rb`:
get "blog.html" => "posts#index", :as => :blog

#in views:
<%= link_to "Blog", :blog_path %> <!-- will generate <a href="/blog.html">Blog</a> -->

If you don't want your URLs to have that .html extension, you can do this folder trick:

#in `routes.rb`:
get "blog/" => "posts#index", :as => :blog
get "blog/index.html" => "posts#index"   

#in views:
<%= link_to "Blog", :blog_path %> <!-- will generate <a href="/blog/">Blog</a> -->

#in `application.rb`, for generating urls with trailing slash:
config.action_controller.default_url_options = { :trailing_slash => true }

With the example above, wget will create a folder named blog/ and put index.html inside, and /blog/ URL will be accessible.

Grabbing The Site

Make sure you are running Rails server in production mode before grabbing. Why production? In development mode Rails appends query strings with versions to static assets to avoid browser cacheing them, and wget will be stupid enough to save files named application.css?body=1. You will not be able to deploy these without renaming them. It's much simpler to run Rails in production instead:

rake assets:clean
rake assets:precompile
# Rails may want you to have a synced production database. SQLite works great.
RAILS_ENV=production db:migrate
rails server -e production

Now, go to someplace clean (i.e.: temp) and run this:

wget -m localhost:3000

It will recursively download your whole site and put it into localhost:3000/ directory. You won't be able to see it properly right from file system because of relative URLs, but try deploying that and it will work like a charm. For easier deployment tar the files:

cd "localhost:3000"
tar -cf website.tar *

Place website.tar in document root of your server and extract it:

tar -xf website.tar

It should work. If you want to serve those files from subdirectory, i.e.: example.com/subdir/, you will have to scope all your paths in routes.rb and set ENV['RAILS_RELATIVE_URL_ROOT'] = "/subdir" in your environment.rb. I haven't tried it though.

Bonus: Dead Simple Local Admin Mode

To manage my content, I found this simple trick to work amazingly well - generate scaffolds to manage your content and add this around links that lead to content manipulation:

<% if params[:admin] %>
  ... links to "new", "edit", "delete" content ...
<% end %>

Then simply run Rails in production mode and manually add ?admin=1 to URLs where you want admin features to appear. Since you're exporting static HTML, there is no security to concern for. Just don't forget to change this method if you ever decide to deploy your Rails app instead of exported static content.

Next steps

To make it even easier, automate your deployment with Capistrano. I found it useful to have a two step deployment - first generate and deploy a test version of the static site, see if everything is OK and then finish the deployment by wiping out production directory and moving test files there. And don't forget to add your production database to source control, assuming you went with SQLite.