A Typing Lout

Learning Language N+1

2016-01-11T00:00:00+00:00

One of the best books I’ve ever read in my career is The Pragmatic Programmer. It is one of those books that does not go particularly deep on any subject, but it does a great job of filling in a lot of the gaps that are the difference between being a decent mid-level developer and a master of the craft. One of the many tips is to invest in your knowledge portfolio and one specific way to do that is by learning one new language per year. In this post, I’ll talk about the process I take to learn a new language because there is a lot more to it than simply just mastering the syntax.

Step 0: Pick a language

How do you pick a language? A lot of it is a matter of taste. I would recommend finding a language that introduces some foreign concepts that you don’t understand. If you don’t understand the value of macros and homoiconic programming, then picking up a lisp is a great idea. If you have only used dynamically typed languages like Ruby and Python, try out Haskell or OCaml. I’d recommend taking up JavaScript and C as you’ll probably bump into them later on in your career.

Step 1: Hello World

I like to start small with Hello World. Much of the hurdle of learning a new language is getting the tool chain working correctly. Hello world forces you to understand how to run and maybe compile your code. I generally also start by writing a test around hello world, can I make it say “hello Tony”? If so, time to move on to step two.

Step 2: Don’t be too ambitious

One then I’ve noticed about myself, is if I make step two too ambitious, I will get frustrated and quit or decide the language is terrible and rant. Don’t make the same mistake as me, once you have hello world working, start with a smaller, but more complicated program. I like something that can be completed in a few hours. Does your company require a code sample during the interview process? That is a great option. The Bowling Kata or other code katas make good candidates. Recently, when learning Go, I wrote a toy cron daemon. The goal of this stage is to get a decent grasp of things, not become an expert. Your code will probably look like crap and that’s okay. Rome wasn’t built in a day. As far as resources, I found resources like Go By Example to be really good to find specific solutions to my problems.

Step 3: Immerse yourself

During step two, you probably Googled a thousand things. You’ve started to form little islands of knowledge about the language, but you don’t have the full picture. This is when I tend to pick up a book and start reading. I’ve been enjoying The Go Programming Language by Donovan and Kernighan. I find reading a book fills in many of the gaps of the “why” things are done the way they are. For Go, understanding the differences between strings and byte-slices and when to use them, the when and why of pointers, and a lot of the nuance around the type system was stuff I didn’t immediately pick up while thrashing through step two.

Another important step is to start reading open source code. How do people that actually know what they are doing write stuff in your chosen language. I’ll generally look at the trending repos on Github and start poking around to see if I pick up any tips. I’ll look at how they’ve split up their logic. How do they test things? Find a style guide and read through it. For many languages “style” can help you sidestep common mistakes.

Step 4: Write!

Now that you have a better idea what good code looks like in your chosen language, start writing some. Find little projects to build. Who cares if you throw them away? Start building the muscle memory to fluency. This is by far the longest step, but also the most fun. At this point, you’ve gotten over the hump of this new language being hard. Now it is time to put it through it’s paces and see what it can and cannot do.

Step 5: Get real

Have you come across a problem at work that your new found knowledge will be perfect for? Time to use it. Most of the time for me this doesn’t involve actually deploying my newly learned language in production, but using one of the techniques that I was exposed to in whatever language we use at work. The most important thing you’ve built with this process is your own knowledge, not any particular bit of software.

Strong opinions, weakly held

2016-01-04T00:00:00+00:00

One thing I’ve learned about myself is that I have strong opinions when it comes to software development. I know I’m not alone. You’ll often see people turn up their nose at the thought of writing Java. Perl is mocked for being too hacky. Haskell is mocked for being too academic. Rubyists think Python is ugly and Pythonistas think Ruby is slow. Maybe it is our need for a sense of belonging that we create an us vs. them mentality around the tools we choose.

A good friend gave me some great career advice. “Have strong opinions, weakly held.” This took me a very long time to internalize. I first understood it as “occasionally allow people to make decisions you disagree with,” as if allowing others to be “right” from time to time was the definition of teamwork. I’ve since realized that it is much deeper than that.

What is a strong opinion? It is easy to misconstrue them for values or principles. Strong opinions are not fundamental truths that serve as the foundation of your decision making. An opinion is defined as “a view or judgement formed about something, not necessarily based on fact or knowledge.” So what makes them strong? Personally, I think a better term for this would be well-reasoned opinions. A strong opinion is one you are prepared to defend in a reasoned manner. You have some evidence, however anecdotal, that you are right.

So what about weakly held? This means that while you will defend your opinion, you will listen when someone has a contradictory opinion. It is important to be an empathetic listener and respect that the other person’s opinion is held just as strongly as yours. You should each be willing to understand the experiences that created the opinion. Sometimes by personal experimentation of their ideas, to try and empathize with them more.

Recently, I’ve run into a difference of opinion that caused some heartburn on my team. We were building out a new system using a microservice-based approach. Many of these services had some shared code for cross-cutting concerns such as database access, logging, etc. The application teams eventually agreed on using a monorepo to store all the projects, so the shared libraries can be easily updated in lockstep across all the services. This was in conflict with our platform teams that were building a system for deployment that used git repositories as the unit of building and deployment.

We had two conflicting opinions, monorepo vs many repos. Both had their merit. Both were right in some ways, wrong in others. It was largely a matter of perspective. As we discussed it between the teams, it was clear who was holding on to their opinions a little too strongly. To be fair, one of those people was me. We eventually settled on the monorepo approach, but it could have gone smoother.

I believe that in order to get better at anything, you need to practice. Letting go of my opinions is something that I want to get better at, so I’ve devised a way to practice. Like my first examples of opinions, there are certain programming languages I find objectionable. I realize my bias against them is not based upon facts of any kind. In the case of Go, the language that I’ve decided to learn next, it is based in my thinking that the language is ugly and the type system is flawed. These opinions are either superficial or based upon the experience of others. By building something non-trivial, I can form my own strong opinions and maybe I’ll like Go a little more. At the very least I will have a better understanding why others like it.

Thinking Functionally

2015-12-31T00:00:00+00:00

I had a moment of deep understanding recently. The skies cleared, the earth moved, angels sang and I finally understood why functional programming is just great. It is a little embarrassing because I’ve been using Clojure in personal projects off and on for years now, I’ve dabbled in OCaml, devoured the Functional Javascript book, but I still didn’t really get it.

Like most programmers that came on the scene in the mid-2000’s I’ve spent a lot of time thinking about objects. Objects have encapsulation and hierarchies. Objects have interfaces and collaborators. There are books written on the best ways for objects to collaborate. I was double dispatching and visitor-patterning to my hearts content. I was happy, I understood how the world works. The I started reading about functional programming.

I’m sure a lot of you have seen this slide

It is from Scott Wlaschin’s fantastic talk on Functional Design Patterns. You should go check out is stuff on fsharpforfunandprofit.

My world collapsed. The years of study and practice around Object Oriented Programming needed to be thrown out. “I can do this” I thought. So I starting looking around for advice to understand how to best write functional programs. A few hours later I was curled in a heap on the floor, rocking back and forth, repeating “A monad is just a monoid in the category of endofunctors”.

I was taken back to my third semester of Physics in college where we stopped talking about Newtonian theory and began talking about Relativity. Things became very abstract and I changed majors. Instead of focusing on how objects collaborate and send messages to each other, I was trying to understand algebra and category theory. Monads were burritos, then they weren’t. It was impenetrable.

I stuck with it though. There were a lot of appealing ideas coming out of the Clojure community. Immutability and simplicity really resonated with me. I kept doing side projects that were just atrocious, terrible want-to-be-OO Clojure code, but slowly it improved. I started to understand the value of 100 functions for 1 data structure over 10 functions for 10 data structures. Then I noticed -> showing up in my code more and more. That is when inspiration struck.

You see, all these OO patterns are really trying to teach you is to think about how objects interact with each other and how they manage state. Often when you are visualizing an object-oriented program, you are thinking in graphs. Nodes are the objects that hold state and the edges are the functions that they use to collaborate. Here is a fairly common example of transferring money between two bank accounts.

class TransferService
  def transfer(from_account_id, to_account_id)
    from_account = AccountRepository.find(from_account_id)
    to_account = AccountRepository.find(to_account_id)
    from_account.transfer_to(10, to_account)
    AccountRepository.save(from_account)
    AccountRepository.save(to_account)
  end
end

I think if you were to draw this out, it would look something like this.

+-----------------+   find    +-------------------+
| TransferService +-----------+ AccountRepository |
+--------------+--+   save    +-------------------+
               |
               |             +---------+
               +-------------+ Account |
                  transfer   +---------+

So, let’s think about this program functionally using stateless function and immutable data. Here’s what it could look like.

module TransferService
  def self.transfer(db_connection, from_account_id, to_account_id)
    from_account = AccountRepository.find(db_connection, from_account_id)
    to_account = AccountRepository.find(db_connection, to_account_id)
    updated_from_account, updated_to_account = Account.transfer(from_account, to_account, 10)
    AccountRepository.save(db_connection, updated_from_account)
    AccountRepository.save(db_connection, updated_to_account)
  end
end

The difference is subtle, but very important. In the OO example, our state is found in objects. The AccountRepository has the database connection as part of its state and knows how to find and save records. The Account object has the state of each account and knows how to move money between itself and other Account objects. In the functional example, all the state is explicit. The state is not maintained by functions, it flows through the functions. Because from_account and to_account are just data, not objects, we treat them as immutable. The transfer method returns new copies of the data containing the new account balances. It is best visualized as data flowing through functions, like so.

            +
            |
            | db_connection, from_account_id, to_account_id
            |
+-----------v------------+
| AccountRepository.find |
+-----------+------------+
            |
            | db_connection, from_account, to_account
            |
  +---------v--------+
  | Account.transfer |
  +---------+--------+
            |
            | db_connection, updated_from_account, updated_to_account
            |
+-----------v------------+
| AccountRepository.save |
+------------------------+

Notice, that the edges and nodes are reversed. When we model the problem functionally, we are moving our data through our functions. Functions are the core building block of the system. Data is the contract of how functions interact. This is the exact inverse of OO where functions are the contract of how objects interact.

I mentioned -> or what is called the threading macro in Clojure. This is what lead me to that insight. The threading macro will translate (-> 1 (a) (b)) to (b (a 1)) basically it will pass the result of the previous value as the first value of the next function. Sounds useful, but not life altering. Here is how we can write this code in Clojure, using the threading macro.

(-> {:db-connection db-connection :from-account-id "123" :to-account-id "456"}
    (find-accounts)
    (transfer 10)
    (save-accounts))

It looks just like the diagram. When you start “thinking functionally” you will see opportunities to use the threading macro everywhere. Thankfully, Clojure is not the only place you can find this functionality. Elixir has a pipe operator |>, F# has several different pipeline operators built into the language. There is even a proposal to introduce it into JavaScript

That’s my tale, hopefully it can help you get over the hump of understanding functional programming.

On Consulting and Consultants

2012-01-03T00:00:00+00:00

In today's job market talented software developers are in absurd demand, far exceeding the supply. Everybody is looking for them. When they become available the competition for their services is fierce. This leaves companies in a difficult position. There are a few options available: greater investment in hiring, lowering the bar for qualified candidates, choosing not to fill the position, or hiring a third party firm.

I spent several years as a senior consultant at ThoughtWorks. Now, as an employee of Braintree I have experienced life having consultants join our team. I will attempt to remain as objective as possible, but hey, it's a blog post.

Situations where consultants are valuable

Hiring a third-party firm is not cheap. At ThoughtWorks, I regularly billed over $150 an hour, occasionally upwards of $200 an hour as a developer. With some quick math, assuming 40 hour weeks and 50 working weeks a year, that consultant you are paying $150 an hour is costing you $300k a year. At that rate, there are several places that consultants can prove their worth.

Providing expertise

There are many companies out there that are experts in very specialized skills. Microsoft is a very good example of this. Microsoft Consulting Services and their partners are experts at very specific Microsoft products. Many open source tools offer similar consulting and support models. If, for example, you would like to set up an instance of Microsoft Dynamics CRM, Microsoft is happy to provide you with consultants that know that product inside and out. Bringing one or two of these folks in to train and educate your team can be a very successful partnership. As a short-term investment, they can get you pointed in the right direction.

Specialized consultants are not without caveats. Often people with very deep, specialized skills are very weak in other areas. So while this person may be an expert in Dynamics, its likely they are lacking in knowledge of testing or automation, or anything else that is necessary to successfully ship software. They most assuredly do not have a deep understanding of your domain. Also, any context that they have built about your particular solution will be gone when they leave. Therefore, I like to think of specialized consultants as bay leaves, they can add wonderful flavor to your dish, but you want to take them out before you serve it to your guests.

Independent Delivery

A much more difficult proposition, but another that can be successful is to have the consultancy provide you a team and build a project independently. This can be quite valuable when the opportunity cost greatly exceeds the cost of the consultants and you simply don't have the development capacity to build it yourself. A discrete project also opens up the opportunity for fixed-bid contracts that may help keep costs under control.

The war in Iraq has taught us, when starting a large expensive venture, its best to have an exit strategy. The vast majority of the cost of software is maintenance and having consultants doing maintenance is prohibitively expensive. At some point the maintenance of the application needs to be turned over to another team. There are probably more, but I have seen three exit strategies have some modicum of success:

Seed the consultant team with the future maintenance developers. It can be a subset of the team that will eventually take over development, but you need someone to gain the context that the consultants will be taking with them when they leave. They will also aid in providing context to the consultants about your development environment, existing APIs, deployment requirements, etc. Adding an employee later in the development process is better than nothing, but there will be some historical context that will be forever lost.
Allow the consultants to hire their replacements. This can be must more risky as hiring is what defines the culture of your team, but it can be useful when you have built a foreign artifact, for example a Ruby on Rails application in an enterprise Java organization, and simply don't have anyone on staff with the knowledge to properly vet candidates. You should be warned that this can take quite a long time. I did this with a client for ThoughtWorks and it extended the engagement 6-9 months longer than necessary.
Throw it away. Not all software lives forever. Some is built for a specific event like a presidential election. When the election is over, the software is useless. The consultants leave, servers are turned off, its like it never happened. This is by far the easiest of tactics, but obviously the least common.

Things to watch out for

Consulting is a business. They are out to make money just like you are. It pays to understand how their business works and how firms game things to turn a profit. There are some things that will happen during your relationship that you need to be aware of.

Frequent Rotation

One of the alluring aspects of being a consultant is to have constant exposure of new and exciting projects. Working for a consultancy generally does not pay as well as working for a product company, so changing projects is one of the primary ways that the developers are kept happy. While this is the number one reason I recommend people coming directly out of college to join a consultancy, it can be damaging to a client. When that consultant walks out the door, all the context that they have built about your company and software is gone too. Constant churn and training can be damaging to the team morale and productivity.

Makeup of the Blended Rate

Often, the statement of work for a relationship will include a blended rate. This means that you pay the same rate for each consultant regardless of experience level and skill. If you are paying one amount for a team of two senior people and two junior people, and a senior person is replaced with a junior person, that rate looks a lot worse. This is understandable from the point-of-view of a consultancy because they maximize profits by staffing a blended rate project with as junior of a team as possible.

Incompetence

Hiring is very difficult. When you bring in a group of consultants, you are basically outsourcing your hiring to them. You need to make sure that the developers joining your team are competent. Relying on the reputation of the firm in insufficient. You need to hold the consultants that join your team to the same standards that you hold developers you would hire. This is important not only for the quality of the work that is done, but also for the morale of the team. A very toxic atmosphere can be created when part of the team is perceived to not be pulling their weight.

Situations to avoid

Staff Augmentation

One of the silliest arguments I hear in favor of working with consultants is that they become your "developer cloud". They are capacity that can be easily added and removed to respond to business demand. If you believe this is true, I suggest you stop reading now and immediately buy this book. There is no faster way to slow a team down than to introduce churn into the makeup of the team. Not only are software projects usually large complex beasts, their development is an incredibly social activity. Learning a domain and gelling with a team are not things that are done easily and without effort from the team itself.

Intermingling of consultants and your development team generates an incredible amount of waste. As I said before, you typically will have a mix of senior and junior people. Both groups need to gain understanding of your domain. That knowledge takes time to disseminate. The junior people also need investment in their skills to become competent senior developers. That investment takes time. It takes time away from building great products. It takes time away from building up your own junior people.

Meddling in your Core Domain

Every software project has that core part that is its essence. Its what differentiates you from your competitors. I strongly believe that you should never give any sort of control of your core to any third-party. Does that mean that every line of code in your core needs to be written by an employee? No, not at all. But I do believe that every decision should be vetted and every line of code should be reviewed. Its simply too precious to give up control of.

Wrapping up

Hopefully I've given you some food for thought when it comes to working with a consulting firm. Used tactically, consultants can help deliver really great value. They are not a magic bullet. Use them wisely.

Destructuring with Ruby

2011-08-08T00:00:00+00:00

One of my side projects has been coding a bot for the Google AI Challenge. I have chosen to implement my bot in Clojure as a good excuse to learn the language with a problem that has some meat to it. One of the great things about learning a new language is learning some techniques that can help you with your day job. The one I'd like to talk about today is destructuring.

The Basics

So what is destructuring? The most concise definition I found is from Common Lisp the Language. Destructuring allows you to bind a set of variables to a corresponding set of values anywhere that you can normally bind a value to a single variable. It is a powerful feature of Clojure that lets you write some very elegant code. For more information about Clojure's features, I recommend you check out Jay Field's blog post on the subject. While destructuring in Ruby is not quite as powerful as Clojure, you can still do some cool stuff. Let's kick this off with an example.

x, y = [1, 2]
x # => 1

If you have done Ruby for a while, you have probably seen some code like that before. One place where it can be quite useful is to return multiple values from a function like so: (apologies for the contrived example)

def min_max(array)
  [array.min, array.max]
end
min, max = min_max([3,5,2])
min # => 2
max # => 5

The Splat Operator

Ruby has some more tricks up its sleeve. One of these is the splat (*) operator. Splat will perform two different operations depending on which side of the assignment it is used. The operation you've most likely run into is called slurp or collect. This takes a variable number of arguments and collects it into an array. Its often used to define methods that can take a variable number of arguments. One caveat is that you can only splat the last parameter to a method. Let's refactor our last example to use splat.

def min_max(*values)
  [values.min, values.max]
end
min, max = min_max(3,5,2)
min # => 2
max # => 5

Splat's other mode of operation is called split. This will convert an array into a list of arguments. This is often used when you have an array and you need to call a method with the arguments of the array. Here is an example.

def divide(numerator, denominator)
  numerator / denominator
end
divide(*[4,2]) # => 2

Let's go back to our first example. What happens if our array has more elements than the number of variables we supplied?

x, y = [1, 2, 3]
x # => 1
y # => 2

Oh snap, we lost 3! Not to worry, we can use the splat operator to slurp up the rest of the array elements.

first, *rest = [1, 2, 3]
first # => 1
rest # => [2, 3]

Hey, now we are on to something. If you have looked into Clojure, Scheme, or another Lisp variant that pattern should look familiar. There is a bevy of algorithms that take very complicated problems and break them down into much simpler problems using recursion and a destructuring similar to the one that we have above.

Destructuring Block Arguments

One of my favorite uses of destructuring in Ruby is inside of a block argument. The syntax for this is that anything within parenthesis will be destructured using the rules we've outlined above. Again, here are some examples.

triples = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]

triples.each { |(first, second, third)| puts second }
# 2
# 5
# 8

triples.map { |(first, *rest)| rest.join(' ') } # => ["2 3", "5 6", "8 9"]

Destructuring Hashes

Thus far, we have only talked about destructuring arrays. We can also destructure hashes in a very interesting way. A Ruby Hash has a method called values_at that takes a list of keys and returns the values for those keys in th order they were given. Check it out.

hash = {:a => 1, :b => 2, :c => 3}
a, b = hash.values_at(:a, :b)
a # => 1
b # => 2

We can also do block-based destructuring with hashes. One very common use case is using inject. Let's say we wanted to write our own implementation of Ruby's Hash#invert method. We could do that using inject with some destructuring.

hash = {:a => 1, :b => 2}
hash.inject({}) { |new_hash, (key, value)| new_hash.merge(value => key) }
# => {1 => :a, 2 => :b}

In the words of Billy Mays, who may rest in peace, "but wait, there's more". We can destructure our destructuring.

hash = {:a => [1, 2, 3], :b => [4, 5, 6]}
hash.inject({}) { |new_hash, (key, (first, second, third))| new_hash.merge(second => key) }
# => {2 => :a, 5 => :b}

I know, that's nuts. I'm not sure if that would ever be useful, but it's a nice trick to have up your sleeve. Let take it one step further, will splat work in there?

hash = {:a => [1, 2, 3], :b => [4, 5, 6]}
hash.inject({}) { |new_hash, (key, (first, *rest))| new_hash.merge(rest => key) }
=> {[2, 3]=>:a, [5, 6] => :b}

Hells yes it does.

Destructuring Objects

As far as I know, Ruby does not have any useful native facility to perform destructuring on an arbitrary object. One of the beautiful things about the language is we can add features we think are missing. Let's give it a shot. We will use the Hash#values_at method as inspiration.

class Object
  def values_at(*attributes)
    attributes.map { |attribute| send(attribute) }
  end
end

class Dog
  attr_reader :color, :breed, :weight

  def initialize(color, breed, weight)
    @color, @breed, @weight = color, breed, weight
  end
end

dog = Dog.new("black", "lab", "heavy")
color, breed = dog.values_at(:color, :breed)
color # => "black"
breed # => "lab"

Ignore the naivety of the implementation, I know there are better ways to monkey patch. That aside, its a fairly cool concept. This would also allow you to use Objects and Hashes interchangeably when destructuring.

In Conclusion

If you've made it this far with me, I'm impressed. I hope that I've shown you some pretty cool things you can do with destructuring to help you write more elegant Ruby. Did I miss something? If so, call it out in the comments.

Command-Query Separation in Ruby

2011-08-04T00:00:00+00:00

One of the features of Ruby that can cause some confusion is that just about everything is an expression. This means that nearly everything that can return a value, does. This leads to some very interesting and sometimes confusing code.

weather = if season == "Winter"
  "cold"
else
  "warm"
end

The previous example is contrived because you would typically write it using the ternary operator, but you get my point. Things get even more interesting when it comes to method definitions. Take a look at this method, what does it return?

def lock_door
  @door_state = :locked if @door_state == :closed
end

It would return :locked if the door was closed, otherwise it would return nil. Is this useful? Not really. What should we do here? Always return nil regardless of if the action was taken or not? Return the final state of the door? I'd propose neither. Leave the method how it is. Bertrand Meyer's principle of Command-Query Separation states that every method should either be a command that performs an action or a query that returns data to the caller. Martin Fowler has a slightly different take on the matter. He prefers to call commands "modifiers" because they are changing the state of the object they are being called on.

To apply this principle to Ruby I like to think in term of grammar. If a method is a verb you shouldn't care what it returns. If a method is a noun or adjective, it had better not change any state.

Like most rules, they are meant to be broken. Martin mentions popping from a stack as a good example. You don't need to look any further than Rails to find some other nice caveats:

p customer.errors unless customer.valid?

The valid? is a query, yet it mutates the object by populating the errors array. I think this is a very pragmatic decision. If we were to refactor this code to follow Command-Query Separation to the letter, it would look something like this:

customer.validate
p customer.errors unless customer.valid?

The problem is, you would almost never have a reason to call validate and never call valid? to see if it worked correctly. Fine choice Rails. Here is another caveat:

customer = Customer.create! :name => "Tony"

This is also a pragmatic approach because if create! did not return the object, we would have to make another round trip to the database to retrieve it.

Command-Query Separation is an important thing to keep in mind. This is doubly true when writing Ruby code because every method returns something whether you like it or not. So if you are going to break the rule, have a good reason.

Update: fixed some typos

Ping Pong Pairing

2011-07-29T00:00:00+00:00

Pair programming is an oft misunderstood beast. "What a waste to have two people do the work of one!" managers say. "I think better when I'm working by myself, with headphones on" exclaim the uninitiated. With this post, I'll explain a technique called Ping Pong Pairing that I find makes for an awesome pairing experience.

There are two roles you can play while pairing, navigating and driving. The driver is the one that is typing on the keyboard. The navigator is the one who isn't. The navigator isn't sitting idle while the driver is going. He is thinking a step ahead. Looking at the code the driver is writing and looking for weaknesses or missing features.

Ping Pong Pairing is a technique where the driver and navigator swap roles once there is a broken test. The process generally goes something like this:

Fred writes a failing test
Jim writes the code that makes that test pass then writes another failing test
Fred makes Jim's test pass and writes another failing test
and so on...

This process works best when the roles swap happen as frequently as possible. When a pair has gotten into a good flow, they can be swapping roles every 30 seconds. Think of it as a game of hot potato.

Outside-In testing has been growing in popularity in the Rails community. Outside-In is where you start developing a new feature with a very high-level test using cucumber or a Rails controller test. This can throw a monkey wrench into the rules of Ping Pong because it may be a very long time before the first high-level test passes. So we tweak the rules and allow a role swap if the driver can write a failing test at a lower level in the architecture. Using a login form in Rails as an example, it would go a little something like this:

Fred writes a failing controller test that you can login by POSTing a username and password
Jim writes a failing test on user that verifies the password
Fred creates the User class and makes the verify_password method return true, then writes a test for the inverse case
Jim implements the rest of the verify_password method, all the user tests should now pass. He reruns the controller test
Fred implements the controller method and writes another test for an error case
and so on...

The really great thing about Outside-In is that you know when you are done. Outside-In with Ping Pong is a win. To sum up, the driver types, the navigator thinks ahead. Try to spend as little time driving as possible. A great game can be made by getting one of these and making the one who drove the most buy the first round.

Gain confidence before you refactor

2009-05-17T00:00:00+00:00

Every code base as its rough spots. Its just a fact of life. Even on a team of rock stars, there still exists hairy parts of the code that nobody wants to touch. When it does come time for this code to change, there are some great techniques that can reduce the pain. The crux of the problem with changing nasty code is that there may be insufficient or no code coverage around the code you want to change. The technique that I describe below will help you gain the confidence you need to renovate the code to a squeaky clean state.

Step 1. Start Clean

Its important to start with a clean state. If you have done some exploratory spiking, make sure that you revert those changes or diff them off to a file. If there are tests around the method run them all and make sure they are green. If there are no tests, create one that will cover the success path of the function. This should be at a fairly high level and should have as little mocks as possible. We really want to understand the behavior here, not necessarily the interactions. If you are finding this test difficult to write, get a copy of Michael Feather's Working Effectively with Legacy Code. Its a great resource.

Step 2. Probe for Weakness

It may seem counter-intuitive, but the next step is to break the code in every way imaginable. Delete if statements, swap parameters used to call other methods, negate boolean expressions, anything you can think of. Each time you make a change, re-run the tests and make sure that at least one fails. If there are no failures, create a failing test. After each change has caused a failing test, revert your changes to the method and ensure that the tests once again pass. Repeat this until you feel comfortable that you have good coverage. This is usually a good point to check in.

Step 3. Make Room for the new feature

Now that we have sufficient coverage around the code, we can now begin changing the method to make room for our new feature. The key here is not to introduce any new functionality. We will lean on the tests that we just wrote to ensure that we have not broken anything. Once we have "extract method"-ed and "extract class"-ed enough to fit our new feature in, its time to check-in again.

Step 4. TDD the new feature

We have now gone to all the trouble of building a solid test suite around our code, it would be a shame to screw it up now by introducing untested changes. Therefore, we will TDD the new feature in by writing new tests or changing existing tests that fail and add new production code to make them pass. This may span multiple check ins depending on the complexity of the feature.

Step 5. Applaud yourself for being a mature developer and good teammate

Nobody likes cowboys and cowgirls in their code base. You should feel proud that you have just completed a feature like a professional. It may of taken more time that it would have to "just put it in", but now you are less likely to have an expensive defect in production. Give yourself a pat on the back.

Showing your longest running tests

2008-09-26T00:00:00+00:00

Slow builds got you down? Have no idea where that slow test is? Get the latest in monkey-patched hotness:

require 'test/unit/ui/console/testrunner'
Test::Unit::UI::Console::TestRunner.class_eval do
 @@test_times = []

 old_attach_to_mediator_method = instance_method(:attach_to_mediator)

 define_method :attach_to_mediator do
   old_attach_to_mediator_method.bind(self).call
   @mediator.add_listener(Test::Unit::TestCase::STARTED, &amp;method(:record_start_time))
   @mediator.add_listener(Test::Unit::TestCase::FINISHED, &amp;method(:record_elapsed_time))
   @mediator.add_listener(Test::Unit::UI::TestRunnerMediator::FINISHED, &amp;method(:print_times))
 end

 def record_start_time(name)
   @start_time = Time.now
 end

 def record_elapsed_time(name)
   @@test_times << {:name => name, :time => (Time.now - @start_time)}
 end

 def print_times(suite_elapsed_time)
   unless @@test_times.empty?
     puts ''
     puts "Displaying 10 longest running tests:"
     @@test_times.sort_by { |t| t[:time] }.reverse[0...10].each do |test_timing|
       puts "#{test_timing[:time]} seconds for #{test_timing[:name]}"
     end
   end
 end

end

And you get some output like this:

Displaying 10 longest running tests:
0.159756 seconds for test_awarding_0.04_points_per_passing_yard(Units::PlayerStatsTest)
0.000432 seconds for test_awarding_6_points_per_return_touchdown(Units::PlayerStatsTest)
0.000367 seconds for test_give_1_point_per_point_after_touchdown_made(Units::PlayerStatsTest)
0.000316 seconds for test_awarding_6_points_per_receiving_touchdown(Units::PlayerStatsTest)
0.000305 seconds for test_awarding_0.1_points_per_receiving_yard(Units::PlayerStatsTest)
0.000297 seconds for test_copping_out_and_giving_3_points_per_field_goal(Units::PlayerStatsTest)
0.00029 seconds for test_awarding_6_points_per_rushing_touchdowns(Units::PlayerStatsTest)
0.00029 seconds for test_penalizing_-2_points_for_fumbles_lost(Units::PlayerStatsTest)
0.000288 seconds for test_penalizing_-1_points_for_interceptions(Units::PlayerStatsTest)
0.000282 seconds for test_awarding_0.1_points_per_rushing_yard(Units::PlayerStatsTest)

Some useful things you can do with mod_rewrite

2008-09-08T00:00:00+00:00

On my current project we are implementing a strangler application to replace a legacy mod_perl application with ruby on rails. For our first release, we are not replacing all of the functionality. In order to get our application to seamlessly interact with the vintage code base, we are making heavy use of mod_rewrite. So I figured I would share what I would consider some of the more interesting rewrites that we have.

Proxy all non-public content to another server, for instance an HAProxy.

This is a very common rule for Rails applications that use the Apache, HAProxy, Mongrel stack. Here the goal is for Apache to serve all static content, i.e. everything under /public in our rails app, because it will be much more efficient. Here our RewriteCond will evaluate to true if the requested resource does not exist on the file system. We will then proxy everything to our HAProxy instance, here running on port 4000.

DocumentRoot /path/to/rails-app/public

RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^.*$ http://localhost:4000%{REQUEST_URI} [P,QSA,L]

Put the protocol into the environment variable based upon the header set by a load balancer

Did you ever get the "This page contains secure and insecure items" warning? Yeah it's an annoyance. To make matters even worse, you have a butt-load of legacy webapps that now need to point to assets from your site but are too brittle and cumbersome to change directly. To make matters double worse, these legacy webapps serve up secure and insecure content (http & https). To make matters triple worse, all of these applications are behind a hardware load balancer that manages the SSL negotiation so all traffic behind it is requested as http. We thought this problem was unsolvable, but found out that our load balancer could set a header whether an SSL negotiation had happened or not. So we put something like this in our vintage apache configuration:

# set the 'protocol' environment variable to default to http
RewriteRule .* - [E=protocol:http]

# if the header exists with the value active, change the protocol variable to https
RewriteCond %{HTTP:X-SSL-State-MTG} ^active$ [NC]
RewriteRule .* - [E=protocol:https]

RewriteRule ^/images/.* %{ENV:protocol}://newapp.com/%{REQUEST_URI}

Transform part of a URL from upper to lowercase

Linux can be a tricky beast. We do all of our development on Macs, so you'd think you'd have your bases covered with operating system incompatibilities. And you'd be wrong. OS X does not use case sensitive path names, so if you are running an Apache locally and you have an asset named foo.gif and you try to access it through http://localhost/FOO.gif it will work fine. However, once you deploy to your sever, in our case running Enterprise SUSE, you will get a 404 - Not Found. There are multiple ways to solve this problem, one you could install mod_speling which will make all of your urls case insensitive. We chose to go another route, mostly because we didn't want to install yet another apache module and we had a pretty simple case. All of our legacy URLs were in upper case and all our resources were in lowercase.

#define a function 'lowercase' that is an alias of the internal tolower function
RewriteMap lowercase int:tolower

#rewrite the image names for everything in the teams folder to lowercase and redirect to the new application
# i.e. /images/teams/ATL.gif => http://www.newapp.com/images/atl.gif
RewriteRule ^/images/teams/(.*).gif http://www.newapp.com/images/${lowercase:$1}.gif

Show a maintenance page if it exists

Capistrano provides some handy tasks to enable and disable your web site. It does this by creating a maintenance.html file in public/system/. In order for your webapp to respect that file you need a rewrite rule like the following:

RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
RewriteRule ^.*$ /system/maintenance.html [L]

Show a default image if the requested image does not exist

Here is a crazy one. Our legacy site is going to request images based upon some key in the database. This information in the database is fairly volatile, so the likelihood that there are missing images is high. This is unacceptable, so we need to show some sort of default image if the specific one is not available. Since the legacy site does not have access to the filesystem where the images live, there is no way for it to know if that image exists before it writes the image tag. Here we can move this logic into the apache configuration of our new application. Here if the requested logo file we are requesting does not exist we will serve up mlb.jpg instead.

RewriteCond %{REQUEST_URI} ^/images/logos/.*
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^/.*$ /images/logos/mlb.jpg [R,QSA,L]

Test Driven Deployment

2008-09-02T00:00:00+00:00

Let me start by saying the I think TDD is the best way to develop quality software. It helps you develop a rhythm, keeps you focused, and has the nice side effect of leaving your application surrounded by unit tests. I have always felt uncomfortable doing work in environments where the rhythm of TDD was not possible. Lately, I have been feeling this pain while developing deployment scripts for a rails application. Utilizing the power of capistrano, Paul and I came up with a rather novel way to test that our deployment scripts were working.

namespace :verify do
 
  task :mongrels, :roles => :app do
    (0...mongrel_count).each do |port_offset|
      assert_status_code '200', "http://localhost:#{mongrel_port + port_offset}/pulse"
    end
  end

  task :ha_proxy, :roles => :app do
    assert_status_code '200', "http://localhost:#{proxy_port}/pulse"
  end
   
  task :apache, :roles => :app do
    assert_status_code '200', "http://localhost:#{apache_port}/monit/token"
  end
end

def assert_status_code expected, url
  assert_equal expected, %{curl -s -o /dev/null -w '%{http_code}' #{url} }
end

def assert_equal expected, command
  errors = []
  run command do |ssh_channel, stream, output|
    errors << "Expected: #{expected} but was #{output} on #{ssh_channel.connection.host}" unless output == expected
    end
  raise "Errors on servers: \n  #{errors.join("\n  ")}\n\n" unless errors.empty?
end

We then execute these 'verify' tasks after the deploy task has completed.

after :deploy do
  verify.mongrels
  verify.ha_proxy
  verify.apache
end

Suite setup and teardown in Test::Unit

2008-05-28T00:00:00+00:00

Test::Unit does not provide an easy way for suite setup and suite tear down methods to be executed. However, like most things in ruby, if there is a will there is a way. Today we managed to implement them using exit hooks. Before I show you that code, it is important that we understand exactly how Test::Unit works. Here is a simple test:

require 'test/unit'

class ExampleTest < Test::Unit::TestCase
  def test_truth
    puts 'testing truth'
    assert true
  end
end

The part that I am most interested in is the require 'test/unit'. Hidden in the bottom of the code for this class is this code:

at_exit do
  unless $! || Test::Unit.run?
    exit Test::Unit::AutoRunner.run
  end
end

In short, this is the code that allows you to execute your test case rb file and have it actually run the test. This code will be called just before the interpreter exits. To prove to yourself that this is true (don't take my word for it) add this line to your test case anywhere after the require.

at_exit { exit! }

exit! will cause the interpreter to quit while ignoring any exit hooks. Since exit hooks are executed in reverse order of registration, this will effectively short circuit the execution of Test::Unit's exit hook. Now we are getting somewhere. Using what we have just learned, we can now implement suite setup and teardown. Check it out:

require 'test/unit'

class ExampleTest < Test::Unit::TestCase
  def test_one; puts 'one'; end
  def test_two; puts 'two'; end
end

successful = false
at_exit { exit! successful }
at_exit { puts 'suite tear down' }
at_exit do
  unless $! || Test::Unit.run?
    successful = Test::Unit::AutoRunner.run
  end
end
at_exit { puts 'suite setup' }

When you run this, it will print:

suite setup
Loaded suite *the file*
Started
one
.two
.
Finished in 0.000564 seconds.

2 tests, 0 assertions, 0 failures, 0 errors
suite tear down

Enjoy!