The Workflow

When I took a photo I usually marked the location with a waypoint in my GPS. I accumulated a set of around 1000 of these points spread out over three gpx (xml) files. My plan is to:

1. Read in the three gpx files and combine them into a distinct list.
2. For each day I have at least one gpx point, get all of my flickr images for that data.
3. For each image, find the waypoint timestamp with the least difference in time.
4. Update that image with the waypoint data on Flickr.

Getting Started

If you’re going to be doing anything with Scala, learning sbt is essential. Luckily, it’s pretty straightforward, but the documentation across the internet is somewhat inconsistent. As of this writing, Twitter’s Scala School SBT Documentation, which I used as a reference to get started, incorrectly states that SBT creates a template for you. It no longer does, with the preferred approach to use giter8, an excellent templating tool. I created my own simplified version which is based off of the excellently documented template by Yuvi Masory. Some of the versions in build.sbt are a outdated, but it’s worthwhile reading through the code to get a feel for the Scala and SBT ecosystem. The g8 project also contains a good working example of custom sbt commands (like g8-test). One gotcha with SBT: if you change your build.sbt file, you must call reload in the sbt console. Otherwise, your new dependencies will not be picked up. For rubyists this is similar to running bundle update after changing your gemfile.

Testing

I’m a big fan of TDD, and strive for a test-first approach. It’s easy to get a feel for the small stuff in the scala repl, but orchestration is what programming is all about, and TDD allows you to design and throughly test functionality in a repeatable way. The two main libraries are specs (actually, it’s now specs2) and ScalaTest. I originally went with specs2. It was fine, but I wasn’t too impressed with the output and not thrilled with the matchers. I believe these are all customizable, but to get a better feel for the ecosystem I switched to ScalaTest. I like ScalaTest’s default output better and the flexible composition of testing styles (I’m using FreeSpec) and matchers (ShouldMatchers) provide a great platform for testing. Luckily, both specs2 and scalatest integrate with SBT which provides continuous testing and growl support, so you don’t need to fully commit to either one too early.

#

My favorite aspect of learning technology while traveling abroad did not come from any book or video. A large part of computer science is about optimizing systems under the pressure of constraints. Efficient algorithms, clean code, improving performance. The world is full of sub-optimal processes. Burmese hotels, the Lao transportation system, and Nepalese immigration to name a few. On a larger scale sun-optimal problems are created by geographic, socio-economic, or political constraints. People try the best they can to improve their way of life, and unfortunately, the processes are often “implemented” with a “naïve” solution. Some are also inspiring. It was powerful to see these systems up close, with cultural and historical factors so foreign. One thing is certain: when you optimize for efficiency, everyone wins.

Below are a selection of books and resources I found particularly interesting. I encourage you to check them out, hopefully away from a computer in a foreign land:

97 Things Every Programmer Should Know : A great selection of tidbits from a variety of sources. Nothing new for the experienced programmer, but reading through the sections is a great refresher to keep core principles fresh. Worthwhile to randomly select a chapter now and again for those “oh yeah” moments.

Beautiful Code by Andy Oram and Greg Wilson: My favorite book. Not so much about code, but the insight about solving problems makes it a great read. I appreciate the intelligent thought process which went into some of the chapters. Python’s hashtable implementation and debugging prioritization in the Linux kernel are two highlights.

Exploring Everyday Things with R and Ruby by Sau Sheong Chang: This is a short book with great content. You only need an elementary knowledge of programming and mathematics to appreciate the concepts. It’s also a great way to get a taste of R. The book covers a variety of topics from statistics, machine learning, and simulations. My favorite aspect is how to use modeling to verify a hypothesis or create a simulation. The chapters involving emergent behavior are particularly interesting.

Machine Learning for Hackers by Drew Conway and John Myles White: I’ve been interested in machine learning for a while, and I was very happy with this read. Far more technical and mathematical than Exploring Everyday Things, this book digs into supervised and unsupervised learning and several aspects of statistics. If you’re interested in data science and are comfortable with programming, this book is for you.

Scala in Action by Nilanjan Raychaudhuri: Scala and Go have been on my radar for a while as new languages to learn. It’s funny to learn a new programming language without being able to test-drive it, but I appreciated the separation. My career has largely been focused on OOP: leveraging design patterns, class composition, SOLID principles, enterprise architecture. After reading this book I realize I was missing out on great functional programming paradigms I was only unconsciously using. Languages like Clojure and Haskell are gaining steam for a radically different approach to OOP, and Scala provides a nice balance between the two. It’s also wonderfully expressive: traits, the type system, and for-comprehension are beautiful building blocks to managing complex behavior. Since returning I’ve been doing Scala full-time and couldn’t be happier. It’s everything you need with a statically typed language with everything you want from a dynamic one (well, there’s still no method-missing, at least not yet). I looked at a few Scala books and this is easily on the top of the list. Nilanjan does an excellent job balancing language fundamentals with applied patterns.

HBase: The Definitive Guide by Lars George: I’ve been deeply interested in distributed databases and performance for some time. I purchased this book a few years ago when first exploring NoSQL databases. Since then, Cassandra has eclipsed the distributed hashtable family of databases (Riak, Hbase, Voldemort) but I found this book a great read. No matter what implementation you go with, this book will help you think in a column-orientated way, offering great tidbits into architectural tradeoffs which went into HBase’s design. At the very least, this book will give you a solid foundation to compare against other BigTable/Dynamo clones.

The Architecture of Open Source Applications: I was excited when I stumbled upon this website. It offers a plethora of information from elite contributors. The applied-practices and deep architectural insight are valuable lessons to learn from. Andrew Alexeev on Nginx, Kate Matsudaira on Scalable Web Architecture and Martin Sústrik on ZeroMQ are highlights.

iTunes U

I was also able to check out some courses on iTunes U while traveling. The MIT OCW Performance Engineering of Software Systems was my favorite. Prof. Saman Amarasinghe and Prof. Charles Leiserson were both entertaining lecturers, and the course provided great insight into memory management, parallel programming, hardware architecture, and bit hacking. I also watched several lectures on algorithms giving me a new found appreciation for Big-O notation (I wish I remembered more while on the job interview circuit). I’ve been gradually neglecting the importance of algorithmic design since graduating ten years ago, but found revisiting sorting algorithms, dynamic programming, and graph algorithms refreshing. Focusing on how well code runs is as important as how well it’s written. Like most things, there’s a naïve brute-force solution and an elegant, efficient other solution. You may not know what the other solution is, but knowing there’s one lurking behind the curtain will make you a better engineer.

So, if you can (and you definitely can!) take a break, grab a book, read it distraction free, gaze out in space and think. You’ll like what you’ll find!

1) What problem do I need to a solve? 2) How do I want to solve it?

Once I take this approach I have an opinionated basis for decision-making and I have a clear direction in how to make that decision. Frameworks–web or javascript–are excellent examples on taking this approach. Most of these frameworks were born on the simple premise of solving a problem in an opinionated way. Backbone takes a bare-bones approach to a front-end, event-driven structure; Ember offers a robust, “things just happen” framework. Sinatra and co. offers an http-first approach to development. Rails and variants are opinionated in web application structure. Do you agree with that approach? Yes, excellent! No? Find something else or roll your own.

Don’t know the answer? That’s okay too. Most beginners want to make the “right” choice on what to learn. But the thing is there is no “right” answer. For a beginner choosing python vs. ruby vs. php vs scala wastes effort. Just build something using something: you’ll soon develop your own opinions, with “how easy is this to learn” probably the first. Next, when your rails codebase is out of control and you’re drowning in method_missing issues maybe you’ll want a more granular, service-orientated approach and the type-safety of Scala. Maybe not… But you’ll have a valid problem to solve and a reasonable opinion to go with it.

I suggest reading Andrew Alexeev’s reason on why he built NGINX and Poul-Henning Kamp’s rationale on how you write a modern application. Like so many others these incredible open-source systems were born from a problem and the way someone wanted it solved. But those systems didn’t happen overnight and the authors didn’t start from scratch. They spent years encountering, learning, and dealing with problems in their respective spaces. They knew the problem domain well, they knew how they wanted the problem solved, and they solved it.

So put your choice in a context and don’t sweat the details which are irrelevant to the task at hand. When you need to know those details you’ll know them, and when you hit problems you’ll know how you want them solved.

• Easy to edit: I should be able to update and output with minimal effort.
• Easy to design: Something simple, but not boilerplate.
• Export to Html and PDF: For easy distribution.

I’m a big fan of Markdown and happy to see the prevalence of Markdown across the web, however fragmented. I use Markdown to publish this blog and felt it would work well for writing a resume. The only problem is layout: you have minimal control over structural html elements which can make aspects of design difficult. For writing articles this isn’t a problem but when you need structural markup for CSS it can be limiting. Luckily I found Maruku, a ruby-based markdown interpreter which supports PHP Markdown Extra and a new meta-data syntax for adding id, css, and div elements to a page. It does take away from Markdown’s simplicity but adds enough structure for design. Combined with CSS I had everything I needed to fulfill my requirements.

My markdown resume is on GitHub. I was surprised it rendered well with GitHub-Flavored Markdown despite the extraneous Maruku elements. I knew I was on the right track. Maruku lets you add your own stylesheets to the html output which I used for posting online. One simple command gets me from markdown to ready-to-publish html. Exactly what I wanted.

Markulu supports pdf output as well, but requires a heavy LaTex install which I wasn’t happy with. I also wasn’t impressed with the LaTex PDF output. Luckily there’s an easy alternative: printing to PDF. I used some SASS media query overrides on top of Html 5 Boilerplate’s default styles to control the print layout in the way I wanted. You can even specify page breaks and print margins via CSS. I favored Safari’s pdf output over Chrome’s for the sole reason Safari automatically embedded custom fonts in the final PDF.

At the end of the day I realized I probably didn’t need to add explicit divs to Markdown; I could have gotten the layout I wanted with just vanilla Markdown and CSS3 queries. I also could have a semantically better markup if I used HAML to add <section> tags instead of divs where appropriate, but HAML would have added a considerable amount of extraneous information to the markup. I’m also not sure editing the raw HAML text would have been as easy as Markdown.

At the end of the day, it’s all a tradeoff. GitHub flavored markdown, Markdown Here and other interpreters support fenced code blocks; I like the idea of adding fenced blocks to get <section> elements to get semantic correctness and layout elements in the html output. Unfortunately there’s no official Markdown spec and support is somewhat fragmented across various implementations, but hopefully it will come together soon. Until then, if you need it, you can always fork. Luckily I didn’t have to take it that far.

Here’s the loader.io results from static content from http://www.michaelhamrah.com, comparing apache with nginx. I suggest clicking through and exploring the charts:

Apache only handled 33.5k successful responses up to about 1,300 concurrent connections, and died pretty quickly. Nginx did far better:

160k successful response with a 22% error rate and avg. response time of 142ms. Not too shabby. The apache run effectively killed the server and required a full reboot as ssh was unresponsive. Nginx barely hiccuped.

The results of my wordpress/php performance is also interesting. I only did 1000 concurrent users hitting blog.michaelhamrah.com. Here’s the apache result:

There was a 21% error rate with 13.7k request served and a 237ms average response time (I believe the lower average is due to errors). Overall not too bad for an ec2-micro instance, but the error rate was quite high and nginx again did far better:

A total of 19k successes with a 0% error rate. The average response time was a little higher than apache, but nginx did serve far more responses. I also get a kick out of the response time line between the two charts. Apache is fairly choppy as it scales up, while nginx increases smoothly and evens out when the concurrent connections plateaus. That’s what scalability should look like!

There are plenty of guides online showing how to get set up with nginx/php-fpm. The Nginx guide on WordPress Codex is the most thorough, but there’s a straightforward nginx/php guide on Tod Sul. I also relied on an nginx tuning guide from Dakini and this nginx/wordpress tuning guide from perfplanet. They both have excellent information. I also think you should check out the html5 boilerplate nginx conf files which have great bits of information.

If you’re setting this up yourself, start simple and work your way up. The guides above have varying degrees of information and various configuration options which may conflict with each other. Here’s some tips:

1. Decide if you’re going with a socket or tcp/ip connection between nginx + php-fcm. A socket connection is slightly faster and local to the system, but a tcp/ip is (marginally) easier to set up and good if you are spanning multiple nodes (you could create a php app farm to compliment an nginx front-facing web farm).
   I chose to go with the socket approach between nginx/php-fpm. It was relatively painless, but I did hit a snag passing nginx requests to php. I kept getting a “no input file specified” error. It turns out it was a simple permissions issue: the default php-fpm user was different the nginx user the webserver runs under. Which leads me to:
2. Plan your users. Security issues are annoying, so make sure file and app permissions are all in sync.
3. Check your settings! Read through default configuration options so you know what’s going on. For instance you may end up running more worker processes in your nginx instance than available cpu’s killing performance. Well documented configuration files are essential to tuning.
4. Plan for access and error logging. If things go wrong during the setup, you’ll want to know what’s going on and if your server is getting requests. You can turn access logs of later.
5. Get your app running, test, and tune. If you do too many configuration settings at once you’ll most likely hit a snag. I only did a moderate amount of tuning; nginx configuration files vary considerably, so again it’s a good idea to read through the options and make your own call. Ditto for php-fcm.

I am really happy with the idea of running php as a separate process. Running php as a daemon has many benefits: you have a dedicate process you can monitor and recycle for php without effecting your web server. Pooling apps allows you to tune them individually. You’re also not tying yourself to a particular web server; php-fpm can run fine with apache. In TCP mode you can even offload your web server to separate node. At the very least, you can distinguish php usage against web server usage.

So my only question is why would anyone still use apache?

The Problem is Simple

At a basic level accessing a web page is when one computer, the client, connects to a server and downloads some content. A problem occurs when the number of people requesting content exceeds the ability to deliver content. It’s just like a restaurant. When there are too many customers people must wait to be served. Staff becomes stressed and strained. Computers are the same. Excessive load causes things to break down.

Optimization Comes in Three Forms

To handle more requests there are three things you can do: produce (render) content faster, deliver (download) content faster and add more servers to handle more connections. Each of these solutions has a limit. Designing for these limits is architecting for scale.

A page is composed of different types of content: html, css and js. This content is either dynamic (changes frequently) or static (changes infrequently). Static content is easier to scale because you create it once and deliver it repeatedly. The work of rendering is eliminated. Static content can be pushed out to CDNs or cached locally to avoid redownloading. Requests to origin servers are reduced or eliminated. You can also download content faster with small payload sizes. There is less to deliver if there is less markup and the content is compressed. Less to deliver means faster download.

Dynamic content is trickier to cache because it is always changing. Reuse is difficult because pages must be regenerated for specific users at specific times. Scaling dynamic content involves database tuning, server side caching, and code optimization. If you can render a page quickly you can deliver more pages because the server can move on to new requests. Most often, at scale, you want to treat treat dynamic content like static content as best you can.

Adding more servers is usually the easiest way to scale but breaks down quickly. The more servers you have the more you need to keep in sync and manage. You may be able to add more web servers, but those web servers must connect to database servers. Even powerful database servers can only handle so many connections and adding multiple database servers is complicated. You may be able to add specific types of servers, like cache servers, to achieve the results you need without increasing your entire topology.

The more servers you have the harder it is to keep content fresh. You may feel increasing your servers will increase your load. It will become expensive to both manage and run. You may be able to achieve a similar result if you cut your response times which also gives the end user a better experience. If you understand the knobs and dials of your system you can tune properly.

Make Assumptions

Don’t be afraid to make assumptions about your traffic patterns. This will help you optimize for your particular situation. For most publicly facing websites traffic is anonymous. This is particularly true during spikes like the Super Bowl. Because you can deliver the same page to every anonymous user you effectively have static content for those users. Cache controls determine how long content is valid and powers HTTP accelerators and CDNs for distribution. You don’t need to optimize for everyone; split your user base into groups and optimize for the majority. Even laxing cache rules on pages to a minute can shift the burden away from your application servers freeing valuable resources. Anonymous users will get the benefit of cached content with a quick download, dynamic users will have fast servers.

You can also create specific rendering pipelines for anonymous and known users for highly dynamic content. If you can identify anonymous users early you may be able to avoid costly database queries, external API calls or page renders.

Understand HTTP

HTTP powers the web. The better you understand HTTP the better you can leverage tools for optimizing the web. Specifically look at http cache headers which allow you to use web accelerators like Varnish and CDNs. The vary header will allow you to split anonymous and known users giving you fine grained control on who gets what. Expiration headers determine content freshness. The worst thing you can do is set cache headers to private on static content preventing browsers from caching locally.

Try Varnish and ESI

Varnish is an HTTP accelerator. It caches dynamic content produced from your website for efficient delivery. Web frameworks usually have their own features for caching content, but Varnish allows you to bypass your application stack completely for faster response times. You can deliver a pre-rendered dynamic page as if it were a static page sitting in memory for a greater number of connections.

Edge Side Includes allow you to mix static and dynamic content together. If a page is 90% similar for everyone, you can cache the 90% in Varnish and have your application server deliver the other 10%. This greatly reduces the work your app server needs to do. ESI’s are just emerging into web frameworks. It will play a more prominent role in Rails 4.

Use a CDN and Multiple Data Centers

You don’t need to add more servers to your own data center. You can leverage the web to fan work out across the Internet. I talk more about CDN’s, the importance of edge locations and latency in my post Building for the Web: Understanding the Network.

Your application servers should be reserved for doing application-specific work which is unique to every request. There are more efficient ways of delivering the same content to multiple people than processing a request top-to-bottom via a web framework. Remember “the same” doesn’t mean the same indefinitely; it’s the same for whatever timeframe you specify.

If you run Varnish servers in multiple data centers you can effectively create your own CDN. Your database and content may be on the east coast but if you run a Varnish server on the west coast an anonymous user in San Fransisco will have the benefit of a fast response time and you’ve saved a connection to your app server. Even if Varnish has to deliver 10% dynamic content via an ESI on the east coast it can leverage the fast connection between data centers. This is much better then the end user hoping coast-to-coast themselves for an entire page.

Amazon’s Route 53 offers the ability to route requests to an optimal location. There are other geo-aware DNS solutions. If you have a multi-region setup you are not only building for resiliency your are horizontally scaling your requests across data centers. At massive scale even load balancers may become overloaded so round-robin via DNS becomes essential. DNS may be a bottleneck as well. If your DNS provider can’t handle the flood of requests trying to map your URL to your IP address nobody can even get to your data center!

Use Auto Scaling Groups or Alerting

If you can take an action when things get rough you can better handle spikes. Auto scaling groups are a great feature of AWS when some threshold is maxed. If you’re not on AWS good monitoring tools will help you take action when things hit a danger zone. If you design your application with auto-scaling in mind, leveraging load balancers for internal communication and avoiding state, you are in a better position to deal with traffic growth. Scaling on demand saves money as you don’t need to run all your servers all the time. Pinterest gave a talk explaining how it saves money by reducing its server farm at night when traffic is low.

Compress and Serialized Data Across the Wire

Page sizes can be greatly reduced if you enable compression. Web traffic is mostly text which is easily compressible. A 100kb page is a lot faster to download than a 1mb page. Don’t forget about internal communication as well. In todays API driven world using efficient serialization protocols like protocol buffers can greatly reduce network traffic. Most RPC tools support some form of optimal serialization. SOAP was the rage in the early 2000s but XML is one of the worst ways to serialize data for speed. Compressed content allows you to store more in cache and reduces network I/O as well.

Shut Down Features

A performance bottleneck may be caused by one particular feature. When developing new features, especially on a high traffic site, the ability to shut down a misbehaving feature could be the quick solution to a bad problem. Most high-traffic websites “leak” new features by deploying them to only 10% of their users to monitor behavior. Once everything is okay they activate the feature everywhere. Similar to determining page freshness for caches, determining available features under load can keep a site alive. What’s more important: one specific feature or the entire system?

Non-Blocking I/O

Asynchronous programming is a challenge and probably a last-resort for scaling. Sometimes servers break down without any visible threshold. You may have seen a slow request but memory, cpu, and network levels are all okay. This scenario is usually caused by blocking threads waiting on some form of I/O. Blocked threads are plugs that clog your application. They do nothing and prevent other things from happening. If you call external web services, run long database queries or perform disk I/O beware of synchronous operations. They are bottlenecks. Asynchronous based frameworks like node.js put asynchronous programming at the forefront of development making them attractive for handling numerous concurrent connections. Asynchronous programming also paves the way for queue-based architectures. If every request is routed through a queue and processed by a worker the queue will help even out spikes in traffic. The queue size will also determine how many workers you need. It may be trickier to code but it’s how things scale.

Think at Scale

When dealing with a high-load environment nothing can be off the table. What works for a few thousand users will grow out of control for a few million. Even small issues will become exponentially problematic.

Scaling isn’t just about the tools to deal with load. It’s about the decisions you make on how your application behaves. The most important thing is determining page freshness for users. The decisions for an up-to-the-second experience for every user are a lot different than an up-to-the-minute experience for anonymous users. When dealing with millions of concurrent requests one will involve a lot of engineering complexity and the other can be solved quickly.

Writing tests is one thing. Striving for full test coverage, writing tests first and leveraging integration and unit tests is another. Some people find writing tests cumbersome and slow. Others may ignore tests for difficult scenarios or code spikes. When first working with tests I felt the same way. Over time I worked through issues and my feeling towards TDD changed. The pain was gone and I worked more effectively.

TDD is about speed. Speed of development and speed of maintenance. Once you leverage TDD as a way to better produce code you’ve unlocked the promise of TDD: Code more, debug less.

Stay In Your Editor

How many times have you verified something works by firing up your browser in development? Too many times. You build, you wait for the app to start, you launch the browser, you click a link, you fill in forms, you hit submit. Maybe there’s a breakpoint you step through or some trace statements you output. How much time have you wasted going from coding to verifying your code works? Too much time.

Stay in your editor. It has everything you need to get stuff done. Avoid the context switch. Avoid repetitive typing. Have one window for your code and another for your tests. Even on small laptops you can split windows to have both open at once. Gary Bernhardt, in an excellent Peepcode, shows how he runs specs from within vim. Ryan Bates, in his screencast How I Test, only uses the browser for UI design. If you leave your editor you are wasting time and suffering a context switch.

Every language has some sort of continuous testing runtime. Detect a file change, run applicable tests. Take a look at Guard. Selenium and company are excellent browser testing tools. Jasmine works great for Javascript. Rspec and Capybara are a solid combination. Growl works well for notifications. By staying in your editor you are coding all that manual verification away. Once coded you can repeat indefinitely.

Start with Tests

Test driven doesn’t mean test after. This may be the hardest rule for newcomers to follow. We’ve been so engrained to write code, to design classes, to focus on OOP. We know what we need to do. We just need to do it. Once code works we’ll then write tests to ensure it always works. I’ve done this bad practice myself.

When you test last you’re missing the why. Customer gets welcome email after signing up means nothing without context. If you know why this is needed you are in a better position to define your required tests and start shaping your code. The notification could be a simple acknowledgement or part of some intricate flow. If you know the why you are not driving blind. The what you will build and the how you will build it will follow. If you code the other way around, testing later, you’re molding the problem to your solution. Define the problem first, then solve succinctly.

Start with Failing Tests

One of my favorite newbie mistakes is when a developer writes some code, then writes a test, watches the test pass, then is surprised when the code fails in the browser. But the test passed!

Anyone can write a green test. It is the action of going from red to green which gives the test meaning. Something needs to work, it doesn’t. Red state. You change your code, you make it work. Green state. Without the red state first you have no idea how you got to a green state. Was it a bug in your test? Did you test the right thing? Did you forget to assert something? Who knows.

Combined with the why going from red to green gives the code shape. You don’t need to over-think class design. The code you write has purpose: it implements a need to make something work that doesn’t. As your functionality becomes more complex, your code becomes more nimble. You deal with dependencies, spawning new tests and classes when cohesion breaks down. You stay focused on your goal: make something work. Combined with git commits you have a powerful history to branch and backtrack if necessary. As always, don’t be afraid to refactor.

Testing First Safeguards Agile Development

Testing first also acts as a safeguard. Too often developers will pull work from a backlog prematurely. They’ll make assumptions, code to those assumptions, and have to make too many changes before release. If the first thing you do after pulling a story is ask yourself “how can I verify this works” you’re thinking in terms of your end-user. You’re writing acceptance tests. You understand what you need to deliver. BDD tools like Cucumber put this paradigm in the foreground. You can achieve the same effect with vanilla integration tests.

Always Test Difficult Code

Most of the time not testing comes down to two reasons. The code is too hard to test or the code is not worth testing. There are other reasons, but they are all poor excuses. If you want to test code you can test code.

Code shouldn’t be too hard to test. Testing distributed, asynchronous systems is hard but still testable. When code is too hard to test you have the wrong abstraction. You’re API isn’t working. You aren’t adhering to SOLID principles. Your testing toolkit isn’t sufficient.

Static languages can rely on dependency injection to handle mocking, dynamic languages can intercept methods. Tools like VCR and Cassette can fake http requests for external dependencies. Databases can be tested in isolation or faked. Asynchronous code can be tricky to test but becomes easier when separating pre and post conditions (you can also block in unit tests to handle synchronization).

The code you don’t test, especially difficult code, will always bite you. Taking the time to figure out how to test will clean up the code and will give you incredible insight into how your underlying framework works.

Always Test Your Code

I worked with a developer that didn’t write tests because the requirements, and thus code, were changing too much and dealing with the failing tests was tedious. It actually signified a red flag exposing larger issues in the organization but the point is a common one. Some developers don’t test because code may be thrown out or it’s just a spike and not worth testing.

If you’re not testing first because it’s a faster way to develop, realize that there is no such thing as throw away code (on the other hand, all code is throw away code). Mixing good, tested code with untested code creates technical debt. If you put a drop of sewer in a barrel of wine you will have a barrel of sewer. The code has no why. It may be just a spike but it could also turn out to be the next best thing. Then you’re left retrofitting unit tests, fitting a square peg in a round hole.

Balancing Integration and Unit Tests

Once you start testing first a lot of pieces fall into place. The balance between integration and unit tests is an interesting topic when dealing with code coverage. There will be overlap in code coverage but not in terms of covered functionality.

Unit tests are the distinct pieces of your code. Integration tests are how those pieces fit together. You have a customer class and a customer page. The unit tests are the rules around the customer model or the distinct actions around the customer controller. The integration tests are how the end user interacts with those models top to bottom. Pivotal Labs talks about changing state in cucumber steps showing how integration tests monitor the flow of events in an application. Unit tests are for the discrete methods and properties which drive those individual events.

Automate

Developing applications is much more than coding. Focusing on tools and techniques at your disposal will help you write code more effectively. Your IDE, command line skills, testing frameworks, libraries and development paradigms are as important as the code you right. They are your tools and become more powerful when used correctly.

We’ve been travelling around turkey for the past three weeks and have relied heavily on bus travel to get around. Travel books have great info, but there are a couple of more things to think about when dealing with buses in Turkey.

First, there are several companies that serve various routes. Pamukkale, KamilKoc and Metro are the big ones. There are several more depending on where you are and where you’re going. If you don’t see the bus time you want, or if a bus is full, check with another company. Prices are fairly set so I don’t think it’s worth negotiating down. If you are booking through a tour operator, hotel or another reseller they will most likely book through another company, most likely one of the above.

It’s important to ask what type of bus you’ll be taking. There are big coach buses, older buses, and minibuses. Ideally you want to be on a big coach bus, often referred to as a big bus. There is usually wifi and tv (turkish only) on big buses, but I haven’t been on one yet with power. One did have USB outlets but was unable to charge the iPad. Minibuses and older buses may not have air conditioning, so it’s important to ask. Big buses have the smoothest ride and the most legroom. If you don’t like the bus you’re getting at the time you want, see if there’s another time with a better bus or go to another company. Always get your ticket from someone behind a desk. There will be plenty of people trying to sherpa you here and there, but just go right to the desk. At some otogars there are valets to help you. They may appear to be trying to sell you something. Just ask the right questions and you’ll be fine. Turkish people are very nice and very helpful.

The bus may make a lot of stops. We were on a minibus from Denizli to Fethiye and the bus stopped for anybody along the road. It was nuts! People would be waiting on the road no more than 50 meters away from each other and the bus would pull up, slow down, see if anybody needed to get on. Also, some buses will stop at rest areas every 45 minutes to an hour for breaks. On our way to Selcuk we had to stop at a rest area for 15 minutes even though we were only five minutes out from our destination. Ask if it is a direct bus and how many stops it will make. Usually the big coaches are better than the minibuses in terms of stopping. If possible, just avoid minibuses. The ride will most likely be bumpy as well unless it is a newer minibus or a tourist minibus.

Seats are assigned on buses, so ask for a seat up in the front. Some bus companies have seat maps so you can see where you’ll be seated. You don’t need to rush onto the bus, just put your bags on, get on, and find your seat. Everyone is very nice and will gladly help you out. You’ll also get tea or coffee on the bus with a snack. If the bus is really bumpy don’t get anything hot. You’ll probably spill it, need to drink it really quickly, than have to go to the bathroom. Most buses don’t have bathrooms (they do make a lot of stops, so don’t worry, but some restrooms cost 1TL). Another funny thing is that on minibuses in small towns there will be guy walking up and down with lemon or rose oil for your hands. A nice little refresher!

Dolmuses are fantastic. These are little vans that go around towns to pick people up and drop them off along the way. They are extremely cheap, extremely frequent and should be leveraged. They are just as good as taxis and cost a lot less. They are great within cities to get to more remote areas and to travel among smaller towns. They are great on the Turqouise coast to explore different beaches. Essentially, you just wait outside on the road in the direction you want and a van with people will pull up. Hotel, pensyon and guest house operators are very helpful with Dolmus transport. Depending on where you are on the Lycian way you could even send your bags ahead to be picked up by your next stop.

If you stick with the big buses and know your options bus travel in Turkey is a great and economical way to get around. Always bring earplugs and an eyemask, especially on night buses. There will always be a crying baby and someone reading with the light on.

Caching Levels

There a three broad levels of caching:

• HTTP Caching allows for full-page caching via HTTP headers on URIs. This must be enabled on all static content and should be added to dynamic content when possible. It is the best form of caching, especially for dynamic pages, as you are serving generated html content and your application can effectively leverage reverse-proxies like Squid and Varnish. Mark Nottingham’s great overview on HTTP Caching is worth a read.

• Fragment Caching allows you to cache page fragments or partial templates. When you cannot cache an entire http response, fragment caching is your next best bet. You can quickly assemble your pages from pre-generated html snippets. For a page involving disparate dynamic content you can build your result page from cached html fragments for each section. For listing pages, like search results, you can build the page from html fragments for each id and not regenerate markup. For detail pages you can separate less-volatile or common sections from high-volatile or per-user sections.

• Object Caching allows you to cache a full object (as in a model or viewmodel). When you must generate html for each user/request, or when your objects are shared across various views, object caching can be extremely helpful. It allows you to better deal with expensive queries and lessen hits to your database.

The goal is to make your response times as fast as possible while lessening load. The more html (or data) you can push closer to the end-user the better. HTTP caching is better than fragment caching: you are ready to return the rendered page. When combined with a CDN even dynamic pages can be pushed to edge locations for faster response times. Fragment caching is better than object caching: you already have the rendered html to build the page. Object caching is better than a database call: you already have the cached query result or denormalized object for your view. The deeper you get in the stack (the closer to the datastore) the more options you have to vary the output. Consequently the more expensive and longer the operation will take.

Break Content Down; Cache for Views

A cache strategy is dependent on breaking content down to store and reuse later. The more granular you can get the more options you have to serve cached content. There are two main dimensions: what to cache and whom to cache for. It is difficult to HTTP cache a page with a “Hello, {{ username }}” in the header for all users. However if you break your users down into logged-in users and anonymous users you can easily HTTP cache your homepage for just anonymous users using the vary http-header and defer to fragment caching for logged-in users.

Cache key naming strategies allow you to vary the what with the who for in a robust way by creating multiple versions of the same resource. A cache key could include the role of the user and the page, such as role:page:fragement:id, as in anon:widget_detail:widget:1234 and serve the widget detail html fragment to anonymous users. The same widget could be represented in a search detail list via anon:widget_search:widget:1234. When widget 1234 updates both keys are invalidated. Most people opt for object caching for an easy win with dynamic pages, specifically by caching via a primary key or id. This can be helpful, but if you break down your content into the what and who for with a good key naming strategy you can leverage fragment caching and save on rendering time.

The vary http header is very helpful for dealing with HTTP caching and is not used widely enough. By varying URIs based on certain headers (like authorization or a cookie value) you can cache different representations for the same resource in a similar way to creating multiple keys. Think of the cache key as the URI plus whatever is set in the vary header. This opens up the power of HTTP caching for dynamic or per-user content.

You are ready to deliver content quickly when you think about your cache in terms of views and not data. Cache a denormalized object with child associations for easy rendering without extra lookups. Store rendered html fragments for sections of a page that are common to users on otherwise specific content. “Popular” and “Recent” may be expensive queries; storing rendered html saves on processing time and can be injected into the main page. You can even reuse fragments across pages. A good cache key naming strategy allows for different representations of the same data which can easily be invalidated.

Cache Invalidation

Nobody likes stale data. As you think about caching think about what circumstances to invalidate the cache. Time-based expirations are convenient but can usually be avoided by invalidating caches on create and update commands. A good cache key naming strategy helps. Web frameworks usually have a notion of “callbacks” to perform secondary actions when a primary action takes place. A set of fragment and object caches for a widget could be invalidated when a record is updated. If cache values are granular enough you could invalidate sections of a page, like blog comments, when a comment is added and not expire the entire blog post.

HTTP Etags provide a great mechanism for dealing with stale HTTP requests. Etags allow a more invalidation options than the basic if-modified-since headers. When dealing with Etags the most important thing is to avoid processing the entire request simply to generate the Etag to validate against (this saves network bandwidth but does not save processing time). Caching Etag values against URIs are a good way to see if an Etag is still valid to send the proper 304 NOT MODIFIED response as quickly as possible in the request cycle. Depending on your needs you can also cache sets of Etag values against URIs to handle various representations.

If you must rely on time-based expiration try to add expiration callbacks to keep the cache fresh, especially for expensive queries in high-load scenarios.

Edge Side Includes: Fragment Caching for HTTP

Edge Side Includes are a great way of pushing more dynamic content closer to users. ESIs essentially give you the benefits of fragment caching with the performance of HTTP caching. If you are considering using a tool like Squid or Varnish ESIs are essential and will allow you to add customized content to otherwise similar pages. The user panel in the header of a page is a classic example of an ESI usage. If the user panel is the only variant of an otherwise common page for all users, the common elements could be pulled from the reverse-proxy within milliseconds and the “Welcome, {{USER}}” injected dynamically as a fragment from the application server before sending everything to the client. This bypasses the application stack lightening load and decreasing processing time.

Distributed or Centralized Caches are Better

Distributed and/or centralized caches are better than in-memory application server cache stores. By using a distributed cache like Memcache, or a centralized cache store like Redis, you can drop duplicate data caches to make caching and invalidating objects easier. Even though caching objects in a web app’s memory space is convenient and reduces network i/o, it soon becomes impractical in a web farm. You do not want to build up caches per-server or steal memory space away from the web server. Nor do you want to have to hunt and gather objects across a farm to invalidate caches. If you do not want to support your own cache farm, there are plenty of SaaS services to deal with caching.

Compress When Possible

Compressing content helps. Memory is a far more valuable resource for web apps than cpu cycles. When possible, compress your serialized cache content. This lowers the memory footprint so you can put more stuff in cache, and lightens the transfer load (and time) between your cache server and application server. For HTTP caching the helpful vary http header can also be used to cache content for browsers supporting compression and those that don’t. For object caching, only store what you need in the cache. Even though compression helps reduce the footprint, not storing extraneous data further reduces the footprint and saves serialization time.

NoSQL to the Rescue

One of the interesting trends I am reading about is how certain NoSQL stores are eliminating the need for separate cache farms. NoSQL solutions are beneficial for a variety of reasons even though they create significant data-modeling challenges. What NoSQL solutions lack in the flexibility of representing and accessing data (i.e. no joins, minimal search) they can make up in their distributed nature, fault-tolerance, end access efficiency. When you model your data for your views, putting the burden on storing data in the same way you want to get it out, you’re essentially replacing your denormalized memory-caching tier with a more durable solution. Cassandra and other Dynamo/Bigtable type stores are key-value stores, similar to cache stores, with the value part offering some sort of structured data type (in the case of Cassandra, sorted lists via column families). MongoDb and Redis, (not Dynamo inspired) offer similar advantages; Redis’ sorted sets/sorted lists offer a variety of solutions for listing problems, MongoDb allows you to query objects.

If you are okay with storing (and updating) multiple-versions of your data (again, you are caching for views) you can cut the two-layer approach of separate cache and data stores. The trick is storing everything you need to render a view for a given key. Searches could be handled by a search-server like Solr or ElasticSearch; listing results could be handled by maintaining your own index via a sorted-list value via another key. When using Cassandra you’d get fast, masterless, and scalable persistant storage. In general this approach is only worthwhile if your views are well-defined. The worst thing you want to do is refactor your entire data model when your views change!

How Web Frameworks Help

There is always debate on differences between frameworks and languages. One of the things I always look for is how easy it is to add caching to your application. Rails offers great support for caching, and the Caching with Rails guide is worth a read no matter what framework or language you use. It easily supports fragment caching in views via content blocks, behind-the-scene action caching support, has a pluggable cache framework to use different stores, and most importantly has an extremely flexible invalidation framework via model observers and cache sweepers. When choosing any type of framework, “how to cache” should be a bullet point at the top of the list.