Atomic Spin - Home

Responsible Estimation Tool

2010-03-03T19:53:00Z

Many times during the sales process we are asked to give a cost estimation based on our early understanding of an application’s core features. In order to responsibly and efficiently estimate, we conduct a 50/90 range analysis. We use this technique because we want to establish a responsible middle ground between our optimistic and pessimistic estimates. Our estimate is appropriately buffered for the risk we see in the defined tasks.

This spreadsheet is the tool that we use to conduct our analysis. Tasks can be estimated in any time duration that you choose (i.e. hours, days, weeks). At the start of a large project we generally estimate tasks in days or weeks.

Startup Weekend in Grand Rapids

2010-02-28T22:36:00Z

I attended the tail end of Startup Weekend at The Factory this weekend. Kudos to Michael Boyink and Aaron Schaap for organizing this in West Michigan. Marc Nager, one of two people who run Startup Weekends all over the country, commented on the energy in the room and the incredible willingness of strangers to get behind ideas and work together. Evidently what we lack in VC we make up for in hard work and generosity.

The final pitches were pretty good considering that some of these ideas weren’t even hatched until Friday afternoon. An app for helping people keep track of things that need to get done even had a pretty functional backend implemented over the weekend. It was great to see eight teams and 40 or so people challenging the economic doom and gloom of our state, putting their ideas out there, listening to others, and taking action to make them real.

git-deploy to three servers in 83 characters

2010-02-26T22:42:00Z

I smiled this morning when I typed this command to git-deploy fresh code to three machines in quick succession.

Sure, there’s nothing groundbreaking here that you can’t do with plenty of other tools. But it was still fun.

Atomic Cookies

2010-02-24T17:08:00Z

No, they won’t explode.

Resident cookie-master, Marissa Christy, went the extra mile yesterday for the SoftwareGR meeting. While our meetings always have great snacks, Marissa really raised the bar this time with Atomic Object logo cookies:

If you want some great snacks, and great information too, come to the next SoftwareGR meeting.

"All input is evil until proven otherwise. That's rule number one."

2010-02-24T16:34:00Z

Yesterday SoftwareGR hosted Dr. Richard Enbody from MSU. He spoke about security, and he quoted the above security rule from the book Writing Secure Code by Michael Howard and David LeBlanc (sample chapter.)

The main topic of the talk was recent research performed by Richard’s team to develop a new security feature called Secure Bit.

One of the biggest classes of vulnerabilities out there is the buffer overflow. Richard showed a list of vulnerabilities taken from CERT’s homepage the preceding day and showed us a slide full of recently discovered buffer overflows. He told us he’s always been able to fill a slide using just the first page.

Buffer overflows are caused by feeding the software some carefully crafted input that allows a malicious user to redirect the flow of execution. Richard’s research discovered that a vulnerability can be present even when the user can’t inject bad code – it turns out that taking over the execution is sufficient. Read more about buffer overflows on Wikipedia.

We heard about an interesting technique for finding these vulnerabilities: fuzzing. This is a way of partially automating an attack. It works by feeding the target software a randomized stream of inputs and waiting until a fault occurs. These faults (and the recent inputs) can be logged and analyzed to discover an attack – for both attacking systems and for fixing your own. Richard’s team developed an ingeniously simple defense against buffer overflows, called Secure Bit. It is a very simple algorithm:

Label all input as tainted.
Forbid jumps to a tainted address.

This is accomplished with a combination of software and hardware changes. Richard adds a single bit to every word stored which indicates whether the input is tainted or not (called the secure bit). This addition can be done at the operating system level and is a relatively simple change. In the Linux kernel Richard’s team was able to make it happen in 4 lines of code. The jump protection is added at the hardware level in the processor. This was tested by making the change in a CPU emulator. The simplicity of this approach means that the instruction set doesn’t have to change and the code stays binary-compatible. For this to be installed on a system, all it would need is the updated hardware and the kernel – no other software needs to be aware of it.

Richard’s test system ran Red Hat, and they tested out a number of known exploits against vulnerable targets and demonstrated that the Secure Bit approach does protect against malicious code execution.

Read all about it on Richard’s web site.

Software GR meets on the fourth Tuesday of the month from 6PM to 8PM at Atomic Object offices. The latest schedule can always be seen at softwaregr.org. Meetings are always free and open to the public.

Better Java with Google Collections

2010-02-23T01:59:00Z

Version 1.0 of the Google Collections library was officially released December 30, 2009. I have been using the library for the past 6 months or so on a variety of Java projects, with great success. Today I gave a brown-bag talk to share with some interested Atoms.

I have included below the sample code I used as presentation material, in case others are interested.

Note: As I was putting this post together I read that Google Collections is now part of Google’s guava-libraries project, which has not been officially released yet.

Code Retreat GR Recap

2010-02-17T14:06:00Z

On February 6th I hosted the first Code Retreat to hit West Michigan, and we really couldn't have asked for a nicer day for a Code Retreat. Well, maybe a little bit warmer weather, but hey, it's February in Michigan, what do you expect? So after a quick stop at Panera Bread to get some bagels, scones and muffins, I made my way down to Atomic Object HQ to start the coffee brewing in preparation for the attendees. Shortly after sunrise, Nayan Hajratwala showed up to help with any last minute preparations before everyone else showed up.

Soon about 20 people from all parts of the region had showed up to practice TDD and learn with each other, including one guy who came all the way down from Marquette, MI just to attend. He officially got the "I traveled the furthest" award for the day. We were also joined by Ron Jeffries and Chet Hendrickson, who had agreed to come and be my professional trouble makers for the day. Shortly after 9:00, once everyone had been sufficiently caffeinated, we decided to get started. One of the attendees had mentioned something about Corey Haines putting together a set of Cucumber features at one of the previous Code Retreats in Chicago, so some of the pairs decided to give that a whirl. After some yak shaving we managed to get through the first iteration of the morning and retrospected on what happened and continued on into the second iteration of the day.

My original plan was to just sort of float around, help facilitate, and observe everyone else pairing, however when I noticed Ron Jeffries didn't have a pair for the second iteration, I took the opportunity to pair with him. Neither of us knew Cucumber very well, so we decided to give that a whirl. If ever you get the chance to pair with either Ron or Chet, don't think twice about it, just do it. Ron had at one point in the day managed to - as one participant described "...[kick his] BDD mindset a bit out of place. . . "

Before we knew it, lunch was upon us. It turns out that Corey Haines was hosting another Code Retreat in Seattle that day, so we fired up Skype and greeted our fellow Code Retreaters on the west coast as they were just getting ready to start for the day. Then we all proceeded to enjoy the taco bar that had been delivered for lunch and continued to retrospect on the days events so far. When we were all finished stuffing our faces with Qdoba, Mike Sweiton and myself gave our participants a quick tour of Atomic Object HQ, showing off our open space, stoplight, CI server, and our embedded projects workbench.

Now that our food had a chance to settle, it was back to pairing for a few more iterations of Conway's Game of Life. After one of the afternoon retrospectives, for a little bit of a distraction, we watched a video of someone implementing Conway's Game of Life in APL. This was spawned by an email thread that circulated right before the Code Retreat about how to implement the Game of Life in a single line of APL, which still blows my mind.

Finally, by the time the end of the day had finally arrived, we had lost a few of our fellow coders and we were ready to call it a day. Those of us who were still left standing at the end of the day took a walk around the corner to The Green Well, one of the many local establishments in the Eastown area, for some much needed unwinding. We continued to retrospect on the day's happenings over a few local microbrews and some delicious items from the menu. All in all I would have to say this was a successful Code Retreat. Everyone had a great time, we all got to pair program with some great folks we wouldn't normally get to pair with, and - most importantly - learning happened. Though many of the Code Retreats in the past have used Java as their language of choice, in my opinion I think using Ruby for this Code Retreat was the right choice. It afforded us much less yak shaving than would have probably been necessary had we been using Java. I'm looking forward to hosting another Code Retreat later this year when the weather is a little warmer, and hopefully attending the upcoming Code Retreat being hosted in Philadelphia by Sebastian Hermida.

Information Radiators

2010-02-08T15:23:00Z

For a business that by definition occupies a highly technical domain, we have some relatively low-tech means of conveying information internally. Sure we e-mail, Twitter, Yammer, and even Skype with each other on a regular basis, but we also make use of yarn, note cards, cork boards, and old traffic equipment. I’m talking about our information radiators, objects around our office that align the team by sharing important information in a fun and hard-to-miss way.

Perhaps the best example of this is the large traffic light positioned in the center of our office. The stoplight is rigged to our continuous integration monitor (which displays the integration status of all active projects), displaying an aggregate of this information. A red light points out a problem, a yellow light shows that we’re processing a recent change, and a green light means that everything is running smoothly.

Another information radiator is our Atomic Spin Blog visitor tracker, a large cork board positioned in the center of our office. Each day, after our stand up meeting, I update the chart for the previous day, making any appropriate “annotations,” like the primary source of traffic for that day. I also like to add people’s photos to the board when they create a new blog post. Last month, for instance, our shooting star Scott Miller literally jumped off the chart with his post on TDD.

These information radiators are not intended to be a sole source of information. The stoplight follows our continuous integration monitor, and the blog board is a somewhat artistic interpretation of the information already stored in Google Analytics, annotations and all. But these visual and dynamic sources of information are not just about the “what” but also about the “how.” The stoplight is able to represent in a very simple way a company value – creating high-quality, ridiculously good, tested and working software at each step. It also provides an opportunity for us to talk about our development methods with visitors to the office. In the same vein, if someone has a hugely successful blog post, what is the best way to share that information? Spam the office with a company-wide email? Tell everyone to access the analytics account themselves and keep tabs? Or is it to get out a ladder so that you can tack a big yellow star near the ceiling? There is something about yarn and stars that just works.

And how do I know? People have told me. “I really want to blog now – so I can get a star.” Okay, so that was probably sarcastic, but the chart nevertheless has the power to encourage all of us to pay attention and contribute to the blog. It makes it very easy to represent – for instance – the fact that this month’s line is higher than last month’s line, or that so-and-so’s blog posts contributed to that success. The chart provides a connection between blogging effort and the popularity of our blog. And it makes me do my job better and promote our blog like crazy.

In sum, the information radiators do more than radiate information. They do it in a way that – because it’s highly visual and fun – is able to rally the team and create excitement about the various goals and objectives we have as a company.

PS – Help me get a star by sharing this article!

Shawn Anderson to present at LA RubyConf and SCALE

2010-02-05T17:05:00Z

Later this month, Shawn Anderson will be going back to Cali, his previous home, to make back-to-back presentations at two different conferences: LA RubyConf and SCALE (the second time in two years). For both, he is presenting on developing 2D games using Ruby and open source tools (including a library that he wrote).

From an early age, Shawn distinguished himself as one who enjoyed gaming. Unlike other kids who were “reading” or engaging in “outdoor sports,” Shawn was content to stay in and finish off King Hippo in Punch-Out!. Over the last couple years he has renewed this early childhood interest. Since college, he has been using his free time to develop games in Ruby; he has even created his own library for gaming in Ruby – Gamebox. Gamebox was designed to spring board game development. It allows the developer to define business rules about a game very quickly without having to worry about resource loading, sound/music management, creating windows, or messing with viewports. Shawn says he wrote Gamebox for two reasons: first, to aid in 48 hour game writing competitions and second, to allow him to write simple educational games for his kid(s). He is currently working on a game called Snelps RTS (noneducational, from what I’ve seen), and the slightly educational Free Radicals.

This workshop is intended for people who already know Ruby but are interested in writing games in Ruby. At the end of the class the participants will have a basic understanding of how 2D games work and will have created a small space invaders clone with animation and sound.

Links:

Info for Shawn’s LA RubyConf Presentation

Info for Shawn’s SCALE Presentation

Getting Started with Gamebox (Source Code)

Snelps Screenshot:

Free Radicals Screenshot:

Beyond UX and Agile

2010-02-01T22:24:00Z

Last weekend I had the privilege of attending a UX Agile Retreat at Cooper. The retreat was an informal event where ~33 practitioners from the UX and Agile fields came together to discuss how we can better create effective and successful products. The event schedule was loosely defined and it self organized based on the trending topics. We used a variety of exercises and discussion techniques to voice our experiences, concerns and ideas to help build a vision of a better, collaborative future.

Where We’ve Been

The UX and Agile communities have been flirting with how they could integrate their practices more seamlessly and respectfully. Both groups respected one another yet remained uneasy about having to alter their respective approaches for an integrated vision and process. UX Designers were worried that some of their practices would be minimized or marginalized due to Agilistas’ strong belief of minimal upfront design. Fear existed that quality design would be sacrificed to the almighty iteration once an agile project was underway. Agilistas were worried about designers doing too much work up front without creating functioning software. Developers were ignorant of the UX Design process and the immense value it provides. Individuals, like Jeff Patton and Lane Halley, have been working very hard to increase knowledge and respect between the two groups. Anders Ramsey organized and dedicated himself to making last week’s event happen. He did an outstanding job. The UX and Agile camps are coming together because they recognize and respect each other’s dedication to the craft of creating high quality products.

Where We Are

Most of our discussions during the retreat focused on vision, values and principles instead of outlining a strategy for collaboration. I’m glad the group was able to remain focused at such a high level because good process stems from alignment and a mutually shared vision.

We took a stand that the “Us and Them” mentality is dead and that we have to move forward as “We.” Management and business processes have kept our groups apart in the past due to a focus on efficiency and a misunderstanding of what it takes to create successful digital products. Management often times does not know how to ask us for what they want, nor do they know how to properly manage creative teams. We believe that we can create better products more efficiently by focusing on being effective. We believe that effectiveness is increased by working together closely.

The Future

The new vision is forming. The retreat group is committed to evangelizing and furthering this vision. I left the retreat with strong feelings of alignment, momentum and excitement. We are the makers of amazing, complex things and we are taking a stand on how we’ll bring the dreams of others into the world. We’re building a case for businesses to employ design thinking at all levels, to focus on the satisfaction of their customers and and to let individuals who understand the creative process to manage it for effectiveness.

The case for embedding jruby-complete into your application

2010-02-01T13:11:00Z

Why in the world would you want to embed JRuby into your application instead of relying on a regular Ruby or JRuby installation? I can think of three reasons.

Note: this is a sister post to my description of the arguments needed to run JRuby via jruby-complete. Here I discuss the rationale for using JRuby like this.

What’s the upside?

You can bundle your Ruby runtime dependence into the application.

If you don’t know what I’m talking about here then you should probably stop reading right now. An old post by Err the blog summarizes the “vendor everything” principle quite well:

Quickly: fix it! Tell everyone to install the gem locally. Install the gem on your staging server. Carefully install the gem on your production server. Phew. Everyone’s got the same version, right? Right. Well, maybe. (At least the build works.)

I like vendoring the Ruby runtime for several reasons:

I don’t want to ask my users to install anything outside of my application. I definitely don’t want to count on the end user having something installed. Depending on end users having a compatible version of Java is frustrating enough, let alone a JRuby installation. Depending on the JRuby installation can be annoying for developers as well. Take culerity as an example. You could spend a bunch of time screwing around with getting your system-wide JRuby installation just right. Or you could embed JRuby directly into culerity and be done with it.
I don’t need to worry about version incompatibilities between the various runtimes. Incompatibilities can mean features I depend on are unavailable, subtly different, or outright broken. An interesting example of “subtly different” came up in an application I recently developed. At one point the sort implementation in JRuby changed from a stable sort to an unstable sort. As far as I know, Ruby doesn’t guarantee sort stability one way or another, so the change was ok from a Ruby perspective. Not so much for me. It turns out my application assumed a stable sort. Had I been dependent on the user’s Ruby installation, my application may have been using either the stable or unstable sort implementation, which was a variance it could not tolerate. (I’m glad I had automated integration tests that revealed this problem to me. I eventually concluded that the application’s behavior was fine with either the stable or unstable sort.)
Using JarJar and a few tricks means you can distribute your Ruby application as a single jar file.

Your application may need to run in a constrained environment where you can’t install much, but Java is available.

A couple of years ago I started development on a new application for monitoring PeopleSoft services. Installing Ruby on the servers I was deploying to was not an option. Java was already there to support PeopleSoft. Thank goodness JRuby 1.0 had just been released.

Managing one file is easier than managing many files.

Copying around one jar file is easier than copying a directory with hundreds of files in it. Yeah, manipulating a directory isn’t always a pain, but the times where manipulating a big directory is a pain can be a really big pain.

What’s the downside?

Your application gets bigger.

jruby-complete is currently (as of version 1.4.0) sitting at around 11.5 megabytes. Do I care? Nope. This can be annoying but it generally doesn’t bother me. Especially when my application is already measured in tens of megabytes.

Figuring out and typing the correct commands to run your application isn’t trivial.

This part sucks. Running commands through jruby-complete is tedious when you don’t have any help from something like Rake tasks. The JVM boot time sucks as well. For these reasons I don’t recommend using jruby-complete for everything: when running small, one-off scripts, there’s just too much overhead. Thankfully, when I do depend on jruby-complete, I’m already working on an application where I’m using Rake to automate all kinds of things. So running lengthly java commands isn’t a big deal because my Rake support is already there.

Thankfully, this companion post is chock full of tips for alleviating the pain from those nasty commands.

Running a Ruby application with jruby-complete

2010-02-01T13:10:00Z

One of the great things about the JRuby project is that it’s easy to run Ruby programs without installing Ruby. In fact, you don’t even need to install JRuby. All you need is a JVM runtime and jruby-complete.

Rationale

Check out this other post for a discussion of my reasons for locking down your JRuby runtime. In summary, embedding jruby-complete gives you complete control of your Ruby runtime. That’s a good thing. The downside is that discovering and executing commands through jruby-complete can be a pain. The rest of this post describes how to ameliorate the pain.

Running jruby-complete

The base jruby-complete command is:

java -jar jruby-complete-1.4.0.jar

This gives you the same behavior as typing ruby or jruby.

java -jar jruby-complete-1.4.0.jar -e "puts 'Hello'"

prints “Hello.” (Of course, this depends on what your jruby-complete jar is named. I usually put the version number in there so I know what it is. I expect anyone reading this to be able to figure out what their jar is named and fill it in appropriately. If this is a hurdle, then too bad for you.)

I lied.¹ To get the same JVM heap and stack sizes as typing jruby, you need to pass a couple of JVM options:

java -Xmx500m -Xss1024k -jar jruby-complete-1.4.0.jar -e "puts 'Hello'"

Want to run irb? Try this:

java -Xmx500m -Xss1024k -jar jruby-complete-1.4.0.jar -e 'load "META-INF/jruby.home/bin/jirb"'

1234	irb(main):001:0> puts "Hello"Hello=> nilirb(main):002:0> %

As you can see, the java command is getting to be a pain. It’s time to introduce some rake tasks to help us out.

JRUBY_COMPLETE = "jruby-complete-1.4.0.jar"
JRUBY = "java -Xmx500m -Xss1024k -jar #{JRUBY_COMPLETE}"

namespace :jruby do
  desc "Run JRuby help"
  task :help do
    sh %+#{JRUBY} --help+
  end

  desc "Run any command with JRuby"
  task :run do
    sh %+#{JRUBY} -e '#{ENV["cmd"]}'+
  end
end

Now I can type rake jruby:run cmd='puts "Hello"'. Shell escaping is becoming a real annoyance at this point. Thankfully, I’m usually not using jruby-complete to run silly little commands. By the time I’ve introduced a Rakefile I’ve got a real application with tasks oriented around testing and running it, so it’s rare that I’m using a task like jruby:run very often.

Running my application may introduce a task that looks something like this:

123	task :run do sh "#{JRUBY} lib/application_bootstrap.rb"end

Running it:

fletcher-git/github/jruby-complete-example(master) rake run
(in /Users/fletcher/git/github/jruby-complete-example)
java -Xmx500m -Xss1024k -jar jruby-complete-1.4.0.jar lib/application_bootstrap.rb
Hello from application_bootstrap

Of course, my application comes with RSpec specs. The standard jruby-complete distribution comes with RSpec built in. How can I use it? The -S parameter runs files in JRuby’s bin directory:

  namespace :spec do
    desc "Run RSpec against a specific file"
    task :run do
      raise "You need to specify a spec with spec=" if not ENV["spec"]
      sh %+#{JRUBY} -S spec -f specdoc #{ENV["spec"]}+
    end
  end

Here’s a spec:

describe "John Galt" do
  it "does not tolerate logical fallacies" do
    "A".should == "A"
  end
end

Running it:

fletcher-git/github/jruby-complete-example(master) rake spec:run spec=spec/unit/objectivism_spec.rb
(in /Users/fletcher/git/github/jruby-complete-example)
java -Xmx500m -Xss1024k -jar jruby-complete-1.4.0.jar -S spec -f specdoc spec/unit/objectivism_spec.rb

John Galt
- does not tolerate logical fallacies

Finished in 0.123 seconds

1 example, 0 failures

Is this more typing than the usual spec spec/unit/objectivism_spec.rb? Yes. Do I care? No. I know how to use my shell.

fletcher-git/github/jruby-complete-example(master) which sp
sp () {
        rake spec:run spec=$@
}
fletcher-git/github/jruby-complete-example(master) sp spec/unit/objectivism_spec.rb
(in /Users/fletcher/git/github/jruby-complete-example)
java -Xmx500m -Xss1024k -jar jruby-complete-1.4.0.jar -S spec -f specdoc spec/unit/objectivism_spec.rb

Alright, so now that my application has been built up, I might want to start compiling the .rb files into .class files. Here comes jrubyc:

require "rake/clean"

namespace :jruby do
  output_directory = "classes"
  directory output_directory
  CLEAN.include output_directory

  desc "Compile Ruby files in lib"
  task :compile => output_directory do
    sh %+#{JRUBY} -S jrubyc -p com/atomicobject -t #{output_directory} lib+
  end
end

fletcher-git/github/jruby-complete-example(master) rake jruby:compile
(in /Users/fletcher/git/github/jruby-complete-example)
mkdir -p classes
java -Xmx500m -Xss1024k -jar jruby-complete-1.4.0.jar -S jrubyc -p com/atomicobject -t classes lib
Compiling all in '/Users/fletcher/git/github/jruby-complete-example/lib'...
Compiling lib/application_bootstrap.rb to class com/atomicobject/lib/application_bootstrap

fletcher-git/github/jruby-complete-example(master) rake jruby:run cmd='require "classes/com/atomicobject/lib/application_bootstrap"'
(in /Users/fletcher/git/github/jruby-complete-example)
java -Xmx500m -Xss1024k -jar jruby-complete-1.4.0.jar -e 'require "classes/com/atomicobject/lib/application_bootstrap"'
Hello from application_bootstrap

Remember that since we’re executing a java command, we can pass any typical JVM parameters before the -jar parameter. We’ve done this for things like:

enabling antialiasing in Apple’s JVM via a Java property.
tweaking Substance’s widget behavior via a Java property.
enabling Yourkit Java Profiler via the -agentlib parameter.
including libraries and directories in the JVM’s classpath via the -cp parameter.

Since these parameters need to be passed before the -jar parameter, a more sophisticated method for setting up the JRuby command is needed than the constant I’ve used. A method like that is specific for your application and beyond the scope of this post, but is not be difficult to create.

Conclusion

There are an uncountable number of good things about JRuby and jruby-complete is one of them. A little help from scripts and your shell means you can build and run your application with a controlled Ruby runtime.

Additional resources

The Rakefile, jruby-complete, and other files used in this post are available in this GitHub project.
JarJar Links is an ant library that is useful for combining multiple jar files together.
I wrote a post a while ago about using JarJar to combine jruby-complete and other application dependences into a single file.
The AGI Production Simulator is built using the jruby-complete commands described in this post as well as the above jar-rolling technique.

Replicating the true jruby behavior is way, way beyond the scope of this post. Check out the jruby script if you really care. Most of the time the JVM heap and stack sizes are the most important things to worry about. ↩

Edit 2/6/2010: Reduced -e ‘load…’ parameters to -S

Yammer Time

2010-01-30T17:04:00Z

Atomic Object has been using Yammer internally for about a month. Yammer is enterprise level micro-blogging. Essentially, it is a private Twitter service.

Yammer has proven to be a great way to share information within our organization. Unlike Twitter, the service is restricted to our organization so the team can freely discuss internal projects, company tactics, and technology information. If any of the commentary is interesting to the outside world it is then broadcasted through the company twitter feed.

Our favorite part about Yammer is that it has provided a channel for our off-site Atoms to connect with the rest of the office. It has helped the team stay connected that much better, and is quickly replacing company wide emails.

Testing terminology

2010-01-29T15:35:00Z

Working with some new customers this year has given me fresh perspective on how confusing the terminology of the testing world can be. I think terms and definitions matters a lot, and not just because I’m a recovering academic. Several times recently I’ve found myself in a conversation taking a position in apparent opposition to my companion, only to eventually determine that we were in fact talking about the same thing. Or we might have been using the same words, but actually meaning very different things. I don’t know that the testing world is any more inconsistent in their terms than other technical fields, but it sure seems like it sometimes.

The terms I prefer (and generally what Atomic developers use as well, though I’m sure we have some internal variation) are useful in the context of an agile development process. They’re also meaningful in more traditional testing cultures, though it’s quite likely that dedicated testers or “QA engineers” are performing the testing.

I have a simple taxonomy for the sorts of tests that developers create: unit, integration, and system. My friend and former colleague Paul Jorgensen taught this to me years ago and it’s worked quite well.

Unit

Tests which validate that a single method or function performs as expected. These tests are almost always “state-based” testing, meaning they make assertions on the return value of a unit, or the change of state in the system.

Example from Ken Beck’s Test-Driven Development

Integration

Tests which validate that multiple units or subsystems play well together. Software systems are the composition and integration of many small pieces of functionality. Integration tests may be state-based or interaction-based. Mocking libraries are often used for integration tests.

System

Tests which validate that the system as a whole, from the perspective of the “end-user”, operates correctly. I put “end-user” in quotes, since that might mean other systems or hardware, depending on the project.

The unusual breadth of Atomic’s range of projects gives me a lot of examples to draw upon. To make the definitions above more concrete, I’ve included examples from web, desktop and embedded applications.

Examples

Unit test examples

A function calculates the square of a floating point argument. A test for this function might include assertions that it returns 4.0 when given 2.0, 0.0 when given 0.0, 81.0 when given -9.0, etc.

A Ruby on Rails web app would have unit tests that confirm that the Model classes behave correctly.

An embedded application might have unit tests that verify computations, as above, or tests for the manipulations of registers.

Integration test examples

A running application written in an object-oriented language can be viewed as a series of messages sent between objects. An integration test in such an application would make assertions that the messages sent between objects were as expected (by name, frequency, parameters, etc).

Ruby on Rails supports and encourages integration tests. A Rails integration test validates that the Controller, View and Model operate properly together. This test isn’t a system test, by my definition, because it doesn’t include the client side (the browser).

System test examples

Desktop applications are used by people. Most often today they have graphical user interfaces. A system test of such an application exercises the app from the user’s perspective. In other words, it pushes buttons, types into text fields, and makes assertions about what’s displayed or changed in the app.

System testing web apps requires driving the application through a browser, just like an end user would. Such tests exercise the browser, Javascript, HTTP, and the full stack of the server side code.

Every embedded application interacts with the world in some fashion. A system test for a speed controller on an automated guided vehicle, for instance, might inject an analog speed signal from a programmable test rig and validate the expected digital output to a motor controller.

Other terms

Regression The term “regression tests” is a common misnomer. Regression means running all your tests every time you make a change, or every time you test. There are no separate, special types of tests that are regression tests. Automation obviously helps to do regression testing, since it dramatically reduces the cost compared to manual regression testing.

Automated tests Tests that can be run without any human intervention (other than starting the test runner). Applies to all levels of testing.

Exploratory testing An exemplar of the context-driven school of testing, exploratory testing is a form of manual testing where the tester learns her way through the application making observations, finding bugs, and identifying interesting new test cases.

Acceptance tests Tests, generally system tests, that are used as a contract between the development team and their customer. The customer agrees that the development team has satisfied the contract when the acceptance tests run clean. Acceptance tests are generally system-level tests and should probably be tied closely to requirements.

Functional tests A synonym for system tests. Also a term used in the Rails community for integration tests that validate the interaction between Controller and View, isolated from the Model.

Non-functional tests (My personal favorite.) Doesn’t refer to broken tests, just those that validate aspects of the product which have nothing to do with application functionality, such as installation, performance, upgradability, etc.

January SoftwareGR meeting: Eric Sink on version control

2010-01-27T00:33:00Z

Eric Sink, of Source Gear was January’s SoftwareGR speaker. Twenty five or so people turned out for the meeting. Eric chose to speak on the topic that he’s passionate about: version control. This is a deeply nerdy interest and his wife is tired of the subject, evidently. Version control in a nutshell is undo & history of change.

Eric started his career in scientific data visualization tools. Great fun, not much of a market. He worked at SpyGlass. (the origin of Internet Explorer), then founded SourceGear when SpyGlass switched to pursuing mobile browsers.

SourceGear has several version control products. TeamPrise, a client for Microsoft Team Foundation Server (TFS), was recently sold to Microsoft.

There are dozens of successful players in this industry. Some open source tools have millions of users, many successful commercial tools (> $1B market). It’s an unusual market this way, in that there isn’t a single, dominant player.

Eric identified several new, and powerful trends in version control: integration and distribution. The trend toward integration is happening in the enterprise. The trend toward DVCS is happening in the open source community.

Integration Integrating your version control tool with burndown charts, bug tracking, time tracking, project management, requirements, testing, builds, agile planning, etc, is where the enterprise is taking version control.

Eric included heterogeneous environment support (various IDEs, platforms, databases, etc) in the integration trend. The leaders in the integration field are Borland (via StarTeam), MS, Serena, and IBM/Rational.

Distribution Distributed version control (DVCS) doesn’t require a centralized server. Branching is so intrinsic to this approach it’s not called branching.

With distributed version control, everyone has a private workspace. You get much better performance (especially for distributed teams), since you’re committing locally. You get good flexibility, which is handy for large companies and complex situations.

With DVCS you’re committing before you merge. There is no longer a single authoritative version. This stops a lot of people (Eric said you shouldn’t let it stop you from learning about these tools). Several people offered advantages and personal testimonials of DVCS over traditional version control.

Examples of DVCS tools include BitKeeper, Git, Mercurial, and Bzr. BitKeeper got the Linux kernel (a major win), then had a very public falling out with that team (a major PR disaster). However, the BitKeeper disaster led to Git, and other important open source DVCS projects.

The enterprise has problems most of us don’t have. For example, Eric cited Cisco, with 15 terabytes in source control, and 50 developers writing perl scripts on top of ClearCase to serve 6000 users. That scale of problem is something the open source, distributed tools aren’t able to handle.

As interesting signs of change, Eric’s company lost a major retailer client to Collabnet + svn. This is a sign that the enterprise is becoming more open to open source tools. Another sign of change: Microsoft is supporting mercurial. That’s significant because mercurial is a DVCS licensed under the GPL, a double hatred for Microsoft.

Eric claimed major enterprise vendors (TFS, Perforce, and Rational) are trying to convert to DVCS. He predicted that these efforts are bound to failure. Integration and distribution are inherently opposed. Eric predicts the clash of these trends will be resolved by integrated, enterprise version control systems picking up just enough distributed features to make people happy.