What:

…did we do before (historical trending, etc)
…is going on right now? (troubleshooting, health, etc.)
…is coming down the road (capacity planning, new feature adoption, etc.)
…can we do to make things better (business intelligence, user-behavior, etc.)

All of which, of course, should be considered mandatory in order to help your business increase its awesome. Yay metrics!

Some time ago, Matthias wrote great a blog post about some of the metrics that can reasonably profile the effectiveness of web operations, taken from the ITIL primer, VisibleOps.

In my opinion, there’s nothing on that list of things that isn’t valuable, as long as the cost of gathering those metrics isn’t too behaviorally, technically, or organizationally expensive. The topics included in that list of metrics and the context they live in is fodder for many, many blog posts.

But in the category of historical trending, I’m more and more fascinated by gathering what I’ll call “meta-metrics”, which is data about how you respond to the changes your system is experiencing.

One of the best examples of this is gathering information about operational disruptions. Collecting information about how many times your on-call rotation was alerted/paged/woken-up, during what times, and for what service(s) can be enlightening to say the least.  We’ve been tracking the volume of alerts a lot closer recently, and even with the level of automation we’ve got at Flickr, it’s still something you have to keep on top of, especially if you’re always finding new things to measure and alert on.

Now ideally, you have an alerting system that only communicates conditions that need resolvable action by a human. Which means every alert is critically important, and you’re not ignoring or dismissing any pages for any reasons that sound like “oh, that’s ok, that cluster always does that…it’ll clear up, I’ll just acknowledge the page so I can shut up nagios.” In other words, our goal is to have a zero-noise alerting system. Which means that all alerts are actionable, not ignorable, and require a human to troubleshoot or fix. Over time, you push as much of this work as you can to the robots. In the meantime, save humans for the yet-to-be-automated work, or the stuff that isn’t easily captured by robots.

Why is this important to us? I may be stating the obvious, but it’s because interrupting humans with alerts that don’t require action has a mental and physical context switching cost (especially if the guy on-call was sleeping), and it increases the likelihood of missing a truly critical page in a slew of non-critical ones.

Of course in the reality of evolving and growing web applications, even if we could reach a 100% noise-free alerting system, it’s impossible to sustain for any extended period of time, because your application, usage, and failure modes are constantly changing. So in the meantime, knowing how your alerts affect the team is a worthwhile thing to do for us. In fact, I think it’s so important that it’s worth collecting and displaying next to the rest of your metrics, and exposing these metrics to the entire dev and ops groups.

Something like this: (made-up numbers)

Tracking Critical Alerts

Gathering up info about these alerts should give us a better perspective on where we can improve. So, things like:

• How many critical alerts are sent on a daily/hourly/weekly basis?
• What does a time histogram of the alerts look like? Do you get more or less alerts during nighttime or non-peak hours?
• How much (if any) correlation is there between critical alerts and:

- code deploys?
- software upgrades?
- feature launches?
- open API abuse?

• What does a breakdown of the alerts look like, in terms of: host type, service type, and frequency of each in a given time period?

and maybe the most important ones:

• How many of those alerts aren’t actually critical or demand human attention?
• How many of them always self-recover?
• How many (and which) don’t matter in their role context (like, a single node in a load-balanced cluster) and could be turned into an aggregate check?

We’ve built our own stuff to track and analyze these things. My question to the community is: I’m not aware of any open-source tool that is dedicated to analyzing these metrics. Do they exist? Nagios obviously has host/hostgroup/cluster warning and critical histories, and those can be crunched to find critical alert statistics, but I’m not aware of any comprehensive crunching. Of course, until I find one, we’re just building our own.

Thoughts, lazyweb?

High availability

Having redundant infrastructure is WebOps 101. For my kids’ most prized possessions, their sleeping  ‘loveys’ there is no reason to have a SPOF, under any circumstances. We have at least 4 backups for each on any trip that we go on, as well as a couple of trusted stuffed animals who might meet unfortunate fates.

Capacity planning

This applies to both disposable diapers (a.k.a. consumable capacity) and episodes of the few TV shows we allow them to watch, on the Tivo. My daughter, at 3 and a half, knows every detail from every of the 49 episodes of The Backyardigans. Having some of them on ipods and iphones can make a 6 hour drive to L.A. feel like 4, not 12.

Documentation

Since I’m already used to writing down observations and techniques learned ‘in the field’, then I was totally prepared:

Allspaw Baby Soothing Method, v1

and in case I ever forgot what my most successful swaddling method was:

Architecture and design

It’s unfortunate that I was so sleep-deprived that I never got a photo of the RadioShack remote-control truck that I turned into a cam-driven Moses basket automatic rocker mechanism. But you understand what I’m talking about.

There is one other thing that I learned from working at Flickr which turned out to be useful new parent advice: expect the unexpected, and never rely on past behaviors as an indication of what can happen in the future. They’re kids, not applications.

One of the cooler papers is “Automatic exploration of datacenter performance regimes” in which the smart folks over at the RAD Lab at UCB tackle the idea of:

1. Gathering up real usage metrics in production
2. Taking that data to feed a resource allocation (”auto-scaling”) controller

The bits about coming up with an exploration policy is where the juicy stuff comes in, building in safety factors driven by external SLAs. You should read the whole thing to see how thoughtful their method was, which includes taking into account effects such as cold ramping, which you almost never see accounted for in simulated situations.  Rock on, RAD Lab: this is the stuff that brings the academia smarts to the real world. Kudos.

FYI: I’m not just saying the paper is cool because they cite my book as a resource in it.

With all of the great things being talked about surrounding ‘Automated Infrastructure’, I’ll point to something insanely cool: 1,190 nodes installed from bare metal to all done in 15 minutes.

That’s One Thousand One Hundred and Ninety nodes. Completely installed in: Fifteen. Fucking. Minutes.

ah, the mighty service level agreement! the tooth and claw by which the wily customer brings the vendor to heel. get the SLA right and you, the customer, can sit back and relax, safe in the knowledge that should there be an outage, you are covered. your business is protected from harm by the warm, experienced embrace of a big, stable telco. pinch me, i must be dreaming.

go read the whole thing.

That was a blast! I had never done a ‘duet’ talk before. Here are the slides:

“cloud computing means getting rid of ops”

If by “ops” you mean “people in data centers racking servers, installing OSes, running cables, replacing broken hardware, etc.” then sure, cloud computing aims to relieve you of those burdens. If you really think ‘ops’ is just that, then you really should put down your Nick Carr book and pay attention to the real world for a change.

The reality is, if your ops team is spending a lot of time doing that, then you’re either:

1. Too big to use someone *else’s* cloud, because you basically have your own (Yahoo, Amazon, Google, etc.)
2. Stuck in 1999.

If you deal with any of these things:

• handling site issues/incidents
• building and maintaining tools to monitor and gather systems and application-level metrics
• program abilities to adapt infrastructure to changing system or application-level conditions (usage, failure, degradation, etc.)
• implements, and maintains deployment systems (code, config management, etc.)
• capacity planning (no, really)

then you’re doing “ops”, by my definition. In some environments, these things are done by “developers”. But my definition says those devs are performing ops functions.

Cloud computing isn’t going to make ‘ops’ go away, it’s relieving of ops (and dev) of a bunch of pain-in-the-ass things so they can focus on the real work needed. Namely: your application.

Last I checked, clouds don’t perform the tasks listed above, because those things (done right) are application-specific. And while cloud computing enables (in an excellent way) the efficient resource allocation (or de-allocation) for an application, it doesn’t get rid of the need to do the above things.

1. When did this problem start?
2. What changes, if any, (software, hardware, usage, environmental, etc.) were made just previous to the start of the problem?

The context surrounding these problem events are pretty damn critical to figuring out what the hell is going on.
Most monitoring systems are based around the idea that you want to know if a particular metric is above (or sometimes below) a certain threshold, and have ‘warning’ or ‘critical’ values that represent what is going bad or already bad. When these alarms go off, knowing how and when they got there is really important your troubleshooting approach. This context is paramount in figuring out where to spend your time and focus.

For example: an alarm goes off because a monitor has detected that some metric has reached a critical state. Something that goes critical instantly can be quite different than something that edged into critical after being in a warning state for some time.

Check it out:

Metric passing thru warning and critical thresholds.

Almost instantaneous critical, no time spent in warning.

For this discussion, the actual metric here isn’t that important. It could be CPU on a webserver, it could be latency on a cache hit or miss on memcached/squid/varnish/etc, or it could be network bandwidth on a rack switch.  The values you set for warning and critical are normally informed by how much tolerance the system can withstand being in warning mode, and given ‘normal’ failure modes, and allow enough wall-clock time for recovery actions to take place before it reaches critical.

Most people would approach these two scenarios quite differently, because of the context that time lends to the issue.

In the book, I give an example of how valuable this context is in troubleshooting interconnected systems. When metrics from different clusters or systems are laid right next to each other, significant changes in usage can be put into the right context. Cascading failures can be pretty hard to track down to begin with. Tracking them down without the big picture of the system is impossible. That graph you’re using for troubleshooting: is it showing you a cause, or symptom?

Because context is so important, I’m a huge fan of overlaying higher-level application statistics with lower-level systems ones. This guy has a great example of it over on the Web Ops Visualization group pool:

He’s not just measuring the webserver CPU, he’s also measuring the ratio of requests per second to total CPU. This is context that can be hugely valuable. If any of the underlying resources change (faster CPUs, more caching on the back-end, application optimizations, etc.) he’ll be able to tell quickly how much benefit he’ll gain (or lose) by tracking this bit.

At the Velocity Summit, Theo mentioned that since OmniTI started throwing metrics for all their clients into reconnoiter, they almost always plot their business metrics on top of their system metrics, because why the hell not? Even if there’s no immediate correlation, it gives their system statistics the context needed for the bigger picture, which is:

How is my infrastructure actually enabling my business?

I’ll say that gathering metrics is pretty key to running a tight ship, but seeing them in context is invaluable.