Terse Systems

Fixing X.509 Certificates

wsargent@tersesystems.com (Will Sargent) — Thu, 20 Mar 2014 09:32:00 -0700

This is a continuation in a series of posts about how to correctly configure a TLS client using JSSE, using The Most Dangerous Code in the World as a guide. This post is about X.509 certificates in TLS, and has some videos to show both what the vulnerabilities are, and how to fix them. I highly recommend the videos, as they do an excellent job of describing problems that TLS faces in general.

Also, JDK 1.8 just came out and has much better encryption. Now would be a good time to upgrade.

Part One: we talk about how to correctly use and verify X.509 certificates.

What X.509 Certificates Do
Understanding Chain of Trust
Understanding Certificate Signature Forgery
Understanding Signature Public Key Cracking

Part Two: We discuss how to check X.509 certificates.

Validating a X.509 Certificate in JSSE
Validating Key Size and Signature Algorithm

What X.509 Certificates Do

The previous post talked about using secure ciphers and algorithms. This alone is enough to set up a secure connection, but there’s no guarantee that you are talking to the server that you think you are talking to.

Without some means to verify the identity of a remote server, an attacker could still present itself as the remote server and then forward the secure connection onto the remote server.

This is the problem that Netscape had. As it turned out, some directory services that needed authentication had a system that looked like it might solve the problem.

The ITU-T had a system of public key certificates in a format called X.509 in a binary encoding known as ASN.1 DER. The entire system was copied wholesale for use in SSL, and X.509 certificates became the way to verify the identity of a server.

The best way to think about public key certificates is as a passport system. Certificates are used to establish information about the bearer of that information in a way that is difficult to forge. This is why certificate verification is so important: accepting any certificate means that an attacker’s certificate will be blindly accepted.

X.509 certificates contain a public key (typically RSA based), and a digest algorithm (typically in the SHA-2 family, i.e. SHA512) which provides a cryptographic hash. Together these are known as the signature algorithm (i.e. “RSAWithSHA512”). One certificate can sign another certificate by taking all the DER encoded bits of a new certificate (basically everything except “SignatureAlgorithm”) and passing it through the digest algorithm to create a cryptographic hash. That hash is then signed by the private key of the organization owning the issuing certificate, and the result is stuck onto the end of the new certificate in a “SignatureValue” field. Because the issuer’s public key is available, and the hash could have only been generated by the certificate that was given as input, we can treat it as “signed” by the issuer.

So far, so good. Unfortunately, X.509 certificates are complex. Very few people understand (or agree on) the various fields that can be involved in X.509 certificates, and even fewer understand ASN.1 DER, the binary format that X.509 is encoded in (which has led to some interesting attacks on the format). So much of the original X.509 specification was vague that PKIX was created to nail down some of the extensions. Currently, these seem to be the important ones:

The “basic fields” that every certificate has.
subjectAltName, where ‘dNSName’ is the official hostname of the server.
basicConstraints used to establish chain of trust.
keyUsage. Used to define a CA certificate.

There are other fields in X.509, but in practice, X.509 compatibility is so broken that few of them matter. For example, nameConstraints is considered near useless and policyConstraints has been misunderstood and exploited.

So if you want to do the minimum amount of work, all you need is some approximation to a DN, maybe a basicConstraints, and if you’re feeling really enthusiastic, keyUsage (although this is often ignored by implementations, see the part 2a slides for examples. Even basicConstraints, the single most fundamental extension in a certificate, and in most cases just a single boolean value, was widely ignored until not too long ago).

— Peter Gutmann

Peter Gutmann is an excellent resource on X.509 certificates (although he does have a tendency to rant. Read the X.509 Style Guide, check out the X.509 bits of Godzilla Crypto Tutorial, and buy Engineering Security when it comes out of draft.

If you’re not up for plowing through hundreds of pages of lovingly (or not so lovingly) described X.509 brokenness, the best overall reference is Zytrax’s SSL Survival Guide.

And for an entertaining overview of X.509 brokenness, watch this video on CCC 26’s “Black Ops of PKI” by Dan Kaminsky:

Understanding Chain of Trust

In TLS, the server not only sends its own certificate (known as an “end entity certificate” or EE), but also sends a chain of certificates that lead up to (but not including) a root CA certificate issued by a certificate authority (CA for short). Each of these certificates is signed by the one above them, so that they are known to be authentic. Certificate validation in TLS goes through a specific algorithm to validate each individual certificate, then match signatures with each one in the chain to establish a chain of trust.

Bad things can happen if the chain of trust only checks the signature, and does not also check the keyUsage and the basicConstraints fields in X.509. Moxie Marlinspike has an excellent presentation at DEFCON 17 on defeating TLS, starting off with subverting the chain of trust:

Understanding Certificate Signature Forgery

As previously mentioned, certificates are needed because they can say “this certificate is good because it has been signed by someone I trust.” If you can forge a signature, then you can represent yourself as a certificate authority:

Since the original paper, an MD5 based attack like this has been seen in the wild. A virus called Flame forged a signature (jumping through a series of extremely difficult technical hurdles), and used it to hijack the Windows Update mechanism used by Microsoft to patch machines, completely compromising almost 200 servers.

MD2 was broken in this paper, and is no longer considered a secure hash algorithm. MD4 is out. As shown in the video, MD5 is out, and the current advice is to avoid using the MD5 algorithm in any capacity. Mozilla is even more explicit about not using MD5 as a hash algorithm for intermediate and end entity certificates.

SHA1 has not been completely broken yet, but it is starting to look very weak. The current advice is to stop using SHA-1 as soon as practical and it has been deprecated by Microsoft. Using SHA-1 is still allowed by NIST on existing certificates though.

Federal agencies may use SHA-1 for the following applications: verifying old digital signatures and time stamps, generating and verifying hash-based message authentication codes (HMACs), key derivation functions (KDFs), and random bit/number generation. Further guidance on the use of SHA-1 is provided in SP 800-131A.

NIST’s Policy on hash functions, September 28, 2012

Even the JSSE documentation itself says that SHA-2 is required, although it leaves this as an exercise for the reader:

“The strict profile suggest all certificates should be signed with SHA-2 or stronger hash functions. In JSSE, the processes to choose a certificate for the remote peer and validate the certificate received from remote peer are controlled by KeyManager/X509KeyManager and TrustManager/X509TrustManager. By default, the SunJSSE provider does not set any limit on the certificate’s hash functions. Considering the above strict profile, the coder should customize the KeyManager and TrustManager, and limit that only those certificate signed with SHA-2 or stronger hash functions are available or trusted.”

The short version is that certificates should be signed with an algorithm from the SHA-2 library (i.e. at least SHA-256). And indeed, most public certificates (over 95%) are signed this way.

Understanding Signature Public Key Cracking

An X.509 certificate has an embedded public key, almost universally RSA. RSA has a modulus component (also known as key size or key length), which is intended to be difficult to factor out. Some of these public keys were created at a time when computers were smaller and weaker than they are now: their key size is too small. Those public keys may still be valid, but the security they provide doesn’t provide adequate protection against today’s technology.

The Mozilla Wiki brings the point home in three paragraphs:

The other concern that needs to be addressed is that of RSA1024 being too small a modulus to be robust against faster computers. Unlike a signature algorithm, where only intermediate and end-entity certificates are impacted, fast math means we have to disable or remove all instances of 1024-bit moduli, including the root certificates.

The NIST recommendation is to discontinue 1024-bit RSA certificates by December 31, 2010. Therefore, CAs have been advised that they should not sign any more certificates under their 1024-bit roots by the end of this year.

The date for disabling/removing 1024-bit root certificates will be dependent on the state of the art in public key cryptography, but under no circumstances should any party expect continued support for this modulus size past December 31, 2013. As mentioned above, this date could get moved up substantially if new attacks are discovered. We recommend all parties involved in secure transactions on the web move away from 1024-bit moduli as soon as possible.

Dates for Phasing out MD5-based signatures and 1024-bit moduli

This needs the all caps treatment:

KEY SIZE MUST BE CHECKED ON EVERY SIGNATURE IN THE CERTIFICATE, INCLUDING THE ROOT CERTIFICATE.

and:

UNDER NO CIRCUMSTANCES SHOULD ANY PARTY EXPECT SUPPORT FOR 1024 BIT RSA KEYS IN 2014.

1024 bit certificates are dead, dead, dead. They cannot be considered secure. NIST has recommended at least 2048 bits in 2013, there’s a website entirely devoted to appropriate key lengths and it’s covered extensively in key management solutions The certificate authorities have stopped issuing them for a while, and over 95% of trusted leaf certificates`1qa and 95% of trusted signing certificates use NIST recommended key sizes.

The same caveats apply to DSA and ECC key sizes: keylength.com has the details.

Part Two: Implementation

The relevant documentation is the Certificate Path Programmer Guide, also known as Java PKI API Programmer’s Guide:

Despite listing problems in verification above, I’m going to assume that JSSE checks certificates and certificate chains correctly, and doesn’t have horrible bugs in the implementation. I am concerned that JSSE may have vulnerabilities, but part of the problem is knowing exactly what the correct behaviour should be and TLS does not come with a reference implementation or a reference suite. As far as I know, JSSE has not been subject to the X.509 test suite from CPNI, and CPNI doesn’t release their test suite to the public. I am also unaware of any publically available X.509 certificate fuzzing tools.

With that in mind, the most I can do is make sure that the existing code is at least being called, and that the bits that should be configured and tweaked are indeed tweaked out of the box.

Validating a Certificate in JSSE

Validating a certificate is easy. Certificate validation is done by java.security.cert and basic certificate validation (including expiration checking) is done using X509Certificate :

certificate.checkValidity()

An interesting side note is that although a trust store contains certificates, the fact that they are X.509 certificates is a detail — trust anchors are just subject distinguished name and public key bindings. This means they don’t have to be signed, and don’t really have an expiration date. This tripped me up initially (and a few others), but RFC 3280 and RFC 5280 are quite clear that expiration doesn’t apply to trust anchors or trust stores.

Unfortunately, the chain validation is far more complex.

Validating Key Size and Signature Algorithm

We need to make sure that JSSE is not accepting weak certificates. In particular, we want to check that the X.509 certificates have a decent signature algorithm and a decent key size.

Now, there is a jdk.certpath.disabledAlgorithms feature in JDK 1.7 that looks very close to doing what we want. (There’s also a jdk.tls.disabledAlgorithms security setting which handles server handshakes and is covered elsewhere)

But jdk.certpath.disabledAlgorithms is only in 1.7 and is global across the JVM. We need to support JDK 1.6 and make it local to the SSLContext. We can do better.

Here’s what an example configuration looks like:

ws.ssl {
  disabledSignatureAlgorithms = "MD2, MD4, MD5"
  disabledKeyAlgorithms = "RSA keySize <= 1024, DSA keySize <= 1024, EC < 224"
}

I’ll skip over the details of how parsing and algorithm decomposition is done, except to say Scala contains a parser combinator library which makes writing small parsers very easy. On configuration, each of the statements parses out into an AlgorithmConstraint that is checks to see if the certificate’s key size or algorithm matches.

There’s an AlgorithmChecker that checks for signature and key algorithms (note that the signature algorithm skips the root):

class AlgorithmChecker(val signatureConstraints: Set[AlgorithmConstraint], val keyConstraints: Set[AlgorithmConstraint]) extends PKIXCertPathChecker {
  …
  def check(cert: Certificate, unresolvedCritExts: java.util.Collection[String]) {
    cert match {
      case x509Cert: X509Certificate =>

        val commonName = getCommonName(x509Cert)
        val subAltNames = x509Cert.getSubjectAlternativeNames
        logger.debug(s"check: checking certificate commonName = $commonName, subjAltName = $subAltNames")

        if (isRootCert) {
          logger.debug(s"checkSignatureAlgorithms: skipping signature checks on trusted root certificate $commonName")
          isRootCert = false
        } else {
          checkSignatureAlgorithms(x509Cert)
        }

        checkKeyAlgorithms(x509Cert)
      case _ =>
        throw new UnsupportedOperationException("check only works with x509 certificates!")
    }
  }
  …
}

and finally:

class AlgorithmChecker(val signatureConstraints: Set[AlgorithmConstraint], val keyConstraints: Set[AlgorithmConstraint]) extends PKIXCertPathChecker {
  …
  def checkSignatureAlgorithms(x509Cert: X509Certificate): Unit = {
    val sigAlgName = x509Cert.getSigAlgName
    val sigAlgorithms = Algorithms.decomposes(sigAlgName)

    logger.debug(s"checkSignatureAlgorithms: sigAlgName = $sigAlgName, sigAlgName = $sigAlgName, sigAlgorithms = $sigAlgorithms")

    for (a <- sigAlgorithms) {
      findSignatureConstraint(a).map {
        constraint =>
          if (constraint.matches(a)) {
            logger.debug(s"checkSignatureAlgorithms: x509Cert = $x509Cert failed on constraint $constraint")
            val msg = s"Certificate failed: $a matched constraint $constraint"
            throw new CertPathValidatorException(msg)
          }
      }
    }
  }
  …
}

Now we have an algorithm checker, we need to put it into the chain.

Note that are two ways of validating a chain in JSSE. The first is using CertPathValidator, which validates a certificate chain according to RFC 3280. The second is CertPathBuilder, which “builds” a certificate chain according to RFC 4158. I’ve been told by informed experts that CertPathBuilder is actually closer to the behavior of modern browsers, but in this case, we’re just adding onto the chain of PKIXCertPathChecker. There are several layers of configuration to go through, but eventually we pass this through to the TrustManager.

To do this, we have to create a PKIXBuilderParameters object and then attach the AlgorithmChecker to it, then stick that inside ANOTHER parameters object called CertPathTrustManagerParameters and then pass that into the factory.init method:

class ConfigSSLContextBuilder {
  def buildTrustManagerParameters(trustStore: KeyStore,
    signatureConstraints: Set[AlgorithmConstraint],
    keyConstraints: Set[AlgorithmConstraint]): CertPathTrustManagerParameters = {
    import scala.collection.JavaConverters._

    val certSelect: X509CertSelector = new X509CertSelector
    val pkixParameters = new PKIXBuilderParameters(trustStore, certSelect)

    // Add the algorithm checker in here…
    val checkers: Seq[PKIXCertPathChecker] = Seq(
      new AlgorithmChecker(signatureConstraints, keyConstraints)
    )

    // Use the custom cert path checkers we defined…
    pkixParameters.setCertPathCheckers(checkers.asJava)
    new CertPathTrustManagerParameters(pkixParameters)
  }

  def buildTrustManager(tsc: TrustStoreConfig,
    signatureConstraints: Set[AlgorithmConstraint],
    keyConstraints: Set[AlgorithmConstraint]): X509TrustManager = {

    val factory = trustManagerFactory
    val trustStore = trustStoreBuilder(tsc).build()
    val trustManagerParameters = buildTrustManagerParameters(
      trustStore,
      signatureConstraints,
      keyConstraints
    )

    factory.init(trustManagerParameters)
    val trustManagers = factory.getTrustManagers
    if (trustManagers == null) {
      val msg = s"Cannot create trust manager with configuration $tsc"
      throw new IllegalStateException(msg)
    }

    // The JSSE implementation only sends back ONE trust manager, X509TrustManager
    trustManagers.head.asInstanceOf[X509TrustManager]
  }
}

And we’re done. Now we can check for bad X.509 algorithms out of the box, and have it be local to the SSLContext.

X.509 certificates are one of the moving pieces of TLS that have many, many ways of going wrong. Be prepared to find out of order certificates, missing intermediate certificates, and other things you can’t identify.

Certificate path debugging can be turned on using the -Djava.security.debug=certpath and -Djavax.net.debug="ssl trustmanager" settings. How to analyze Java SSL errors is a good example of tracking down bugs, and you may also like Portecle, a GUI tool for certificates.

Certificate Revocation!

Monkeypatching Java Classes

wsargent@tersesystems.com (Will Sargent) — Sun, 02 Mar 2014 14:56:00 -0800

So, remember that thing I said last post about adding some debug features to JSSE? Turns out that didn’t work.

Sometimes debugging would work. Sometimes it wouldn’t. I couldn’t get it to work reliably.

The debugging feature in JSSE is defined by calling -Djavax.net.debug=ssl on the command line. The class that reads from the system property is sun.security.ssl.Debug and reading it explained everything:

public class Debug {

    private String prefix;
    private static String args;

    static {
        args = java.security.AccessController.doPrivileged(
            new GetPropertyAction("javax.net.debug", ""));
        args = args.toLowerCase(Locale.ENGLISH);
        if (args.equals("help")) {
            Help();
        }
    }

    public static Debug getInstance(String option)
    {
        return getInstance(option, option);
    }

    public static Debug getInstance(String option, String prefix)
    {
        if (isOn(option)) {
            Debug d = new Debug();
            d.prefix = prefix;
            return d;
        } else {
            return null;
        }
    }

    public static boolean isOn(String option)
    {
        if (args == null) {
            return false;
        } else {
            int n = 0;
            option = option.toLowerCase(Locale.ENGLISH);

            if (args.indexOf("all") != -1) {
                return true;
            } else if ((n = args.indexOf("ssl")) != -1) {
                if (args.indexOf("sslctx", n) == -1) {
                    // don't enable data and plaintext options by default
                    if (!(option.equals("data")
                        || option.equals("packet")
                        || option.equals("plaintext"))) {
                        return true;
                    }
                }
            }
            return (args.indexOf(option) != -1);
        }
    }
}

This explained why I was seeing problems with my debug code. The args field of Debug was being set in a static initialization block, when the class was first loaded into the JVM. Due to some race conditions, my code could call System.setProperty("java.net.debug", options) before Debug was loaded, but there was no way of ensuring that. And of course, the args file was marked as private static final so there was no way to modify it outside of the class after that point.

But it was even worse than that. The static methods isOn and getInstance were used by the JSSE internal classes to determine whether debug information should be logged or not, and those fields were also marked private static final, i.e. in sun.security.ssl.SSLContextImpl:

public abstract class SSLContextImpl extends SSLContextSpi {

    private static final Debug debug = Debug.getInstance("ssl");

    private X509ExtendedKeyManager chooseKeyManager(KeyManager[] kms)
            throws KeyManagementException {
        for (int i = 0; kms != null && i < kms.length; i++) {
            if (debug != null && Debug.isOn("sslctx")) {
                System.out.println(
                    "X509KeyManager passed to " +
                    "SSLContext.init():  need an " +
                    "X509ExtendedKeyManager for SSLEngine use");
            }
            return new AbstractKeyManagerWrapper((X509KeyManager)km);
        }
        return DummyX509KeyManager.INSTANCE;
    }
}

In order to turn debugging on, I not only needed to change the args field in sun.security.ssl.Debug after it had been initialized, but I also needed to change every class that used private static final Debug debug = Debug.getInstance("ssl"); so that it would no longer be null.

The first idea I came up with was to change Debug.getInstance() and Debug.isOn. Now, there are ways to change method definitions in the JVM. The package java.lang.instrument defines java agents that can swap out code at run time (“hot reloading”), there’s “HotSwap” the JVM debug option, and there’s the disposable classloader option. None of them are really applicable in this case — JSSE is a system level package, Play does not require a java agent, and HotSwap is… well, it would probably work fine. But we don’t actually need to change the method definition. We just need to swap out private static final field references at run time.

The language tells you that you can’t touch final fields and it tells you that you absolutely can’t touch private fields from outside the class. It’s lying. If you own the JVM, you can monkey patch any field reference.

The key is a method called sun.misc.Unsafe#putObject. It looks like this:

trait MonkeyPatcher {

  private val unsafe: sun.misc.Unsafe = {
    val field = Class.forName("sun.misc.Unsafe").getDeclaredField("theUnsafe")
    field.setAccessible(true)
    field.get(null).asInstanceOf[sun.misc.Unsafe]
  }

  /**
   * Monkeypatches any given static field.
   *
   * @param field the field to change
   * @param newObject the new object to place in the field.
   */
  def monkeyPatchField(field: Field, newObject: AnyRef) {
    val base = unsafe.staticFieldBase(field)
    val offset = unsafe.staticFieldOffset(field)
    unsafe.putObject(base, offset, newObject)
  }
}

So, we’ve got the means to swap out any given field. Now we need to find all the classes that have a Debug field defined. This is a little trickier, as the class loader only knows about the fields in classes that have already been loaded. We need to go through the JAR file, load all the classes in that JAR into memory, and then quiz them about whether they have that field or not.

So, we add a class finder trait:

trait ClassFinder {
  def logger: org.slf4j.Logger
  def initialResource: String
  def isValidClass(className: String): Boolean

  def findClasses: Set[Class[_]] = {
    logger.debug(s"findClasses: using initialResource = ${initialResource}")

    val classSet = scala.collection.mutable.Set[Class[_]]()
    val classLoader: ClassLoader = Thread.currentThread.getContextClassLoader

    val urlToSource: URL = this.getClass.getResource(initialResource)
    logger.debug(s"findClasses: urlToSource = ${urlToSource}")

    val parts: Array[String] = urlToSource.toString.split("!")
    val jarURLString: String = parts(0).replace("jar:", "")

    logger.debug(s"findClasses: Loading from ${jarURLString}")

    val jar: URL = new URL(jarURLString)
    val jarConnection: URLConnection = jar.openConnection
    val jis: JarInputStream = new JarInputStream(jarConnection.getInputStream)
    try {
      var je: JarEntry = jis.getNextJarEntry
      while (je != null) {
        if (!je.isDirectory) {
          var className: String = je.getName.substring(0, je.getName.length - 6)
          className = className.replace('/', '.')
          if (isValidClass(className)) {
            logger.debug(s"findClasses: adding valid class ${className}")

            val c: Class[_] = classLoader.loadClass(className)
            classSet.add(c)
          }
        }
        je = jis.getNextJarEntry
      }
    } finally {
      jis.close()
    }
    classSet.toSet
  }
}

And then we’re going to set up something that will do Debug and args swapping specifically. There are a couple of wrinkles to this: we need to have AccessController give us privileged conditions, and between 1.6 and 1.7 the package name of Debug changed from com.sun.net.ssl.internal.ssl.Debug to sun.security.ssl.Debug, so we have to get the class we want at runtime as well.

abstract class FixLoggingAction extends PrivilegedExceptionAction[Unit] with MonkeyPatcher with ClassFinder {

  def newOptions: String

  def isValidField(field: Field, definedType: Class[_]): Boolean = {
    import java.lang.reflect.Modifier._
    val modifiers: Int = field.getModifiers
    field.getType == definedType && isStatic(modifiers) && isFinal(modifiers)
  }
}

object FixInternalDebugLogging {
  class MonkeyPatchInternalSslDebugAction(val newOptions: String) extends FixLoggingAction {

    def initialResource = "/javax/net/ssl/SSLContext.class"

    def isValidClass(className: String): Boolean = {
      className.startsWith("com.sun.net.ssl.internal.ssl") || className.startsWith("sun.security.ssl")
    }

    def run() {
      val debugType: Class[_] = {
        val debugClassName = foldVersion(
          run16 = "com.sun.net.ssl.internal.ssl.Debug",
          runHigher = "sun.security.ssl.Debug"
        )
        Thread.currentThread().getContextClassLoader.loadClass(debugClassName)
      }

      val newDebug: AnyRef = debugType.newInstance().asInstanceOf[AnyRef]

      // Switch out all the classes with a static final field
      for (debugClass <- findClasses) {
        for (debugField <- debugClass.getDeclaredFields) {
          if (isValidField(debugField, debugType)) {
            monkeyPatchField(debugField, newDebug)
          }
        }
      }

      // Switch out the Debug.args field.
      val argsField = debugType.getDeclaredField("args")
      monkeyPatchField(argsField, newOptions)
    }
  }

  def apply(newOptions: String) {
    try {
      val action = new MonkeyPatchInternalSslDebugAction(newOptions)
      AccessController.doPrivileged(action)
    } catch {
      case NonFatal(e) =>
        throw new IllegalStateException("InternalDebug configuration error", e)
    }
  }
}

That’s it! Just call FixInternalDebugLogging("ssl") and you can turn on debug information in TLS dynamically, at runtime. This is tremendously useful when you are in the Play console or running a couple of tests to a specific client.

This is a technique that works best on initialization code where the original codebase just needs to be tweaked a bit, but monkey patching can be applied to any accessible field. It’s obviously unsafe, but it’s effective. Evil, sure. But effective.

Mua ha ha.

Fixing The Most Dangerous Code In The World

wsargent@tersesystems.com (Will Sargent) — Mon, 13 Jan 2014 14:44:00 -0800

TL;DR

Most non-browser HTTP clients do SSL / TLS wrong. Part of why clients do TLS wrong is because crypto libraries have unintuitive APIs. In this post, I’m going to write about my experience extending an HTTP client to configure Java’s Secure Socket library correctly, and what to look for when implementing your own client.

Introduction

I volunteered to implement a configurable TLS solution for Play’s web services client (aka WS). WS is a Scala based wrapper on top of AsyncHttpClient that provides asynchronous mechanisms like Future and Iteratee on top of AsyncHttpClient and allows a developer to make GET and POST calls to a web service in just a couple of lines of code.

However, WS did not contain any way to configure TLS. It was technically possible to configure TLS through the use of system properties (i.e. “javax.net.ssl.keyStore”) but that brought up more messiness — what if you wanted more than one keystore? What if you needed clients with different ciphers? Sadly, WS isn’t alone in this: most frameworks don’t provide configuration for the finer points of TLS.

Added to that was the awareness that SSL client libraries have been dubbed The Most Dangerous Code in the World (FAQ). The problem is real and serious, and I want to fix it.

There is also a long and well known gulf between the security community and the developer community about the level of knowledge about TLS and the current state of the HTTPS certificate ecosystem. I want to fix that as well, and this blog post should be a good start.

So. Here’s what I did.

For the sake of readability (i.e. avoiding TL;DR), I’m breaking this across several blog posts.

The pull request is on Github and you are invited to review the code and comment as you see fit.

In this blog post, I’m just going to cover the setup.

First, the problems that make TLS necessary.

The First Problem: Programmers do not get security
The Second Problem: Wifi / Ethernet is not secure
The Third Problem: Man In the Middle

Then, the implementation in WS.

The Use Cases for WS
Understanding TLS
Understanding JSSE
Configuring a client
Debugging a client
Choosing a protocol
Choosing a cipher suite

Future posts will discuss certificates in more detail, but this gives us somewhere to start.

Problems

The First Problem: Programmers do not get security

The first problem is the assumption that TLS is overkill, built by researchers to protect against an abstract threat.

Unfortunately, this is not the case. TLS has real attacks against it, and they exist in the wild. Even worse, there are very serious, real world implications to breaking a TLS connection. Some people trust TLS in situations which could mean imprisonment or death.

But. Programmers work with bugs. Programmers get bugs. Programmers do not get security.

Programmers understand how bad input can ruin a programmer’s day. Programmers understand how corrupt data can completely ruin any hope of a functioning program. Programmers know that working with concurrency is so dangerous that it should only be done with special concurrency primitives and rules. Human users may be incompetent, but they are mostly benevolent: the forces working against the programmer are entropy and loose requirements.

Programmers don’t usually write programs that have to defend against an attacker. Most programmers have never even seen an attacker. Even the concept of a human deliberately trying to break or subvert a program is foreign. QA usually tests for successful cases, and maybe for some negative test cases… but QA typically doesn’t submit specially crafted XML documents that poke at the filesystem or chew up gigabytes of memory with character entities.

If programming is like driving a car, then the difference between working with QA versus working against an attacker is the difference between driving in rush hour versus driving with someone determined to run you off the road. TLS, and people using TLS based clients, have to assume that someone is going to try to run them off the road.

It also helps to see what an attack is like. For most programmers, an attack is theoretical, even a bad joke in poor taste. It doesn’t really become real until an actual attack is demonstrated by a security researcher in front of the programmer in question.

With that in mind, I’ve included several videos from real live security professionals in this blog post. You don’t have to watch them all at once, but you should watch them all eventually and see how they think.

The Second Problem: Wifi / Ethernet is not secure

TLS has a specific problem that most programmers do not have to deal with. TLS has to assume an attacker has access to the TCP/IP stream between client and server. This is commonly called packet snooping.

It is trivial to snoop on other computers in your network using tools like Wireshark. This is doubly true when using wireless networks, such as a coffee shop. People are often surprised that every single website they visit is being broadcast in the clear, in the same way as radio, but it’s true, and it’s easy to pick up those radio transmissions.

But don’t take my word for it. Here’s the Wifi Pineapple:

You can buy a Wifi Pineapple for $100 plus shipping.

It picks up all traffic sent over a wifi network. It’s so good at intercepting traffic that people have turned it on and started intercepting traffic accidentally.

You can plug it into an ethernet adapter, or there’s an ultra bundle that provides a huge antenna and an elite version that lets it run off a battery for 72 hours.

Don’t think that WPA protects you from this. WPA2 is vulnerable to bruteforcing and most people choose passwords with extremely low entropy, partly because wifi passwords are shared so often.

The Third Problem: Man In the Middle

Not only can an attacker sniff packets on the network, but the attacker can also substitute traffic. Here’s a video of Cain & Abel at work:

Note that it takes less than 20 seconds to impersonate the server, after which the attacker can modify any URL coming from the server to point somewhere else. This is why rendering a login page in HTTP is essentially no protection at all: by the time the page is rendered, the attacker can make the HTML form send to a completely different URL.

This attack is called Man in the Middle. (For an in depth demonstration of MITM attacks, checkout the presentation Hey, I just middled you, and this is crazy.)

This is not a theoretical attack. It has been automated to the point where rewriting web pages on the fly is fairly trivial — if you go down to your local hackerspace and browse the Internet without using a VPN, at some point you will find all your images URLs are pointing to Goatse.

Google has been subject to a host of attacks, from Iran. Note that the attack happened at the backbone, not at a particular coffee shop. Advanced persistent threats (APT) can include large nation states as well as script kiddies.

Encrypting data ensures that an attacker cannot read plaintext over the network, using public key encryption. However, the client still doesn’t know the identity of the server it’s trying to connect to. This is a problem. If you don’t verify the identity of the machine you’re talking to, then you could be talking to anyone.

The Use Cases

So that’s the threat model. Now the use cases for WS.

WS is a web services client, intended for asynchronous, non-blocking programmatic access to services using HTTP. Most clients will be RESTful with either a small (4k) XML or JSON payload or continously streaming data. Clients will only connect to a few well-known servers. Use of WS for general browsing or indexing a website is possible, but not the focus.

Client connects to internal WS service

In this use case, the client is talking to a service which is not publically available. The client and server will use private certificates (the “moxie option”), use a PKI management solution like DigiCert or OpenCA, or use an internal root CA.

The client may use mTLS / client authentication to connect to the internal service as an additional security measure.

The server will most likely support TLSv1.0. TLSv1.2 support is unlikely given that it does not come out of the box with ngnix and other clients. It is likely that the server supports RC4 ciphers.

Client connects to external WS service

In this use case, the client is talking to an external WS service, which is not owned by the organization and exists on the public Internet. The server may have a self-signed certificate, but is more likely to have a public certificate signed by a certificate authority.

Public facing “webscale” services using HTTPS are likely to support TLSv1.2 and support good ECC ciphers.

Client connects to public internets

In this use case, the client is calling up random URLs given to it by the web service and storing the content. This is a behavior of RSS feed web applications, which require connections to scrape and process data but do not typically analyse the contents.

The server could be anything. This is not the primary use case, and so the defaults will not be tuned for maximum compatbility with unknown or misconfigured servers.

Understanding TLS

This is going to be extremely abbreviated, but let’s give a refresher anyway. Adapted from Zytrax’s SSL Survival Guide (which also has some excellent sections on X.509 certificates):

TLS has four components: authentication, message integrity, key negotiation and encryption.

The client and the server begin a handshake.

The server sends a certificate and the chain of certificates leading back to a root certificate authority. The client should perform certificate and chain validation, making sure the chain terminates to a root CA trusted by the client.

In mutual TLS or client authentication, the client also sends a certificate to the server. This is rare, and most communication just authenticates the server to the client.

Because a server hands out the public key certificate and has the private key certificate, the client can encrypt all HTTP information using the public key and the server can decrypt it using the private key.

Thomas Porrin’s explanation of how SSL works is also excellent. For details, Wikipedia is comprehensive as always.

So far, so good. Next is JSSE.

Understanding JSSE

JSSE is complex. The reference guide and the crypto spec are surprisingly helpful (once I started to understand it), but it wasn’t until I had the source code handy and could look at the internal Sun JSSE classes that I felt I had a handle on it:

JSSE 1.7: Reference Guide, Crypto Spec, Sun Providers, Cert Path, Source code
JSSE 1.6: Reference Guide, Crypto Spec, Sun Providers, Cert Path, Source code

In addition, the following best practices guides were very helpful:

It’s important to note that Play supports JDK 1.6. and 1.6 came out in December 2006. That’s over eight years ago. Since then, TLS (and the attacks on TLS) have evolved. Where possible, I wanted to bring 1.6 up to the 1.7 level of functionality, or at least note where it lags.

There are several parts to the JSSE, but it all starts with SSLContext(source, impl). Without a correctly configured SSLContext, you have nothing.

val sslContext = SSLContext.getInstance(protocol)
sslContext.init(keyManagers, trustManagers, secureRandom)
return sslContext // correctly configured ssl context.

Wait, what? What’s a TrustManager? What’s a KeyManager? Well, from the JSSE Reference Guide (which is the first, last and frequently only word on the subject):

TrustManager: Determines whether the remote authentication credentials (and thus the connection) should be trusted.
KeyManager: Determines which authentication credentials to send to the remote host.

Most of the time, you’ll be working with a trust manager. You only need to worry about a key manager if you’re doing client authentication.

The interesting thing with this API, right off the bat, is that init() takes null parameters for defaults, and it takes an array of managers.

sslContext.init(null, null, null)

What the JSSE Reference Guide says is “installed security providers will be searched for the highest priority implementation of the appropriate factory”. What actually happens is that you get an empty key manager and a default X509TrustManagerImpl that points to cacerts.

Likewise, the API takes an array of key managers, so you would expect this to work:

sslContext.init(Array(keyManager1, keyManager2), null, null) // what could go wrong?

The problem here is that init() doesn’t compose or aggregate managers together. As the Javadoc says, “only the first instance of a particular key and/or trust manager implementation type in the array is used. (For example, only the first javax.net.ssl.X509KeyManager in the array will be used.)”

There is similar fineprint and tricksy assumptions throughout the JSSE API. If you don’t have the source code available, you will be utterly confused.

If you’re not using SSLContext, then changes are done by setting system properties. This isn’t a bad way per se, but it’s global and opaque to the API.

Direct unit testing is painful as half the classes are defined as final, or use static methods. The internal logic is frustratingly and needlessly tightly coupled.

Despite all of this, and despite having interfaces temptingly near, you should NEVER replace the underlying JSSE functionality. Augment it, sure. Subclass away. Filter out weak points. But doing a straight up rewrite is a mistake. As per Moxie Marlinspike:

If you’re interested in writing a more restrictive TrustManager implementation for Android, my recommendation is to have your implementation call through to the system’s default TrustManager implementation as the very first thing it does. . That way you can ensure you at least won’t be doing any worse than the default, even if there are vulnerabilities in the additional checks you do.

—Guardian’s StrongTrustManager Vulnerabilities

Having read through TLS and JSSE, we’re now ready to check out how to configure the client.

Configuring a client

So, with the source code in hand, the first question was: how is WS?

It turns out that WS does the right thing. If you want to disable certificate validation, you have explicitly set the following in application.conf:

ws.acceptAnyCertificate = true

which will let you accept a self-signed certificate that has not been added to your trust store.

However, the way ws.acceptAnyCertificate is interesting. In Play 2.2.x, it looks like this:

if (!playConfig.flatMap(_.getBoolean("ws.acceptAnyCertificate")).getOrElse(false)) {
  asyncHttpConfig.setSSLContext(SSLContext.getDefault)
}

That’s it. There’s no other logic that involves telling AsyncHttpClient to accept any certificate anywhere else in Play.

It turns out that accepting any certificate is the default behavior in AsyncHttpClient. If you are making HTTPS calls in Java using AsyncHTTPClient 1.7.x directly, you are vulnerable to a MITM attack.

The SSLContext class is central to the SSL implementation in Java in general and in AsyncHttpClient in particular. The default SSLContext for AsyncHttpClient is dependent on whether the javax.net.ssl.keyStore system property is set. If this property is set, AsyncHttpClient will create a TLS SSLContext with a KeyManager based on the specified key store (and configured based on the values of many other javax.net.ssl properties as described in the JSSE Reference Guide linked above). Otherwise, it will create a TLS SSLContext with no KeyManager and a TrustManager which accepts everything. In effect, if javax.net.ssl.keyStore is unspecified, any ol’ SSL certificate will do.

— SSL Certificate Verification in Dispatch and AsyncHttpClient

The first step in implementing HTTPS is to set up certificate verification to avoid issue 352. This, in itself, is fairly easy: just create an SSLContext instance, then init with null values.

val builder = new AsyncHttpConfig.Builder()
val sslContext = SSLContext.getInstance(protocol)
sslContext.init(null, null, null)
builder.setSSLContext(sslContext)
val asyncHttpClientConfig = builder.build()

But, of course, that was only the beginning.

The essential problem with ws.acceptAnyCertificate is while it’s wrong, it’s also a one line configuration setting. It’s obvious what it does. Meanwhile, the experience of adding a self signed certificate to the trust manager is downright painful. By default, the root CA certificates are in $JAVA_HOME/lib/security/cacerts and so if you want to add an extra certificate (rather than replace all the existing CA certs), you have to know the exact keystore command for it:

keytool -import -trustcacerts -file /path/to/ca/ca.pem -alias CA_ALIAS -keystore $JAVA_HOME/jre/lib/security/cacerts

Just from a deployment and maintenance perspective, this is a huge hassle. And it’s not like trust stores and keystores are all that complicated: there’s an list of certificates tied to aliases, with an optional password attached.

The simplest thing to do, from a programmer perspective, would be to have a list of stores that were pulled into a single manager. Then, instead of having to run a keytool command, you could just add a line saying where your store was, and you’d be done.

This involved creating a key manager and a trust manager that could take multiple stores. After looking through the source code, I determined that there was no API problem with using multiple stores inside a single manager… but then ran into the implementation again. In the X.509 implementation of JSSE, there’s a one to one correspondence between a manager and a store. I ended creating managers from the factories and then using a composite manager pattern based off Cody A. Ray’s blog post, and using the X509TrustManagerImpl implementation and the X509ExtendedTrustManager example as references.

The composite trust manager has a list of X509TrustManagerImpl, and iterates through each one until it finds one that doesn’t throw an exception. If all of them throw exceptions, then it rethrows the exception with the entire list (so that no exceptions are swallowed), otherwise, it returns the first good result. This extends the TrustManager functionality while safely keeping all of the existing logic in place.

def checkServerTrusted(chain: Array[X509Certificate], authType: String): Unit = {
  var trusted = false
  val exceptionList = withTrustManagers {
    trustManager =>
      trustManager.checkServerTrusted(chain, authType)
      trusted = true
  }

  if (!trusted) {
    val msg = s"No trust manager was able to validate this certificate chain: # of exceptions = ${exceptionList.size}"
    throw new CompositeCertificateException(msg, exceptionList.toArray)
  }
}

private def withTrustManagers(block: (X509TrustManager => Unit)): Seq[Throwable] = {
  val exceptionList = ArrayBuffer[Throwable]()
  trustManagers.foreach {
    trustManager =>
      try {
        block(trustManager)
      } catch {
        case e: CertPathBuilderException =>
          logger.debug("No path found to certificate: this usually means the CA is not in the trust store", e)
          exceptionList.append(e)
        case e: GeneralSecurityException =>
          logger.debug("General security exception", e)
          exceptionList.append(e)
        case NonFatal(e) =>
          logger.debug("Unexpected exception!", e)
          exceptionList.append(e)
      }
  }
  exceptionList
}

Now you can now configure multiple key stores and trust stores directly in application.conf:

ws.ssl {
  keyManager = {
    stores = [
      { type: "PKCS12", path: "keys/client.p12", password: "changeit2" },
      { type: "PEM", path: "keys/client.pem" }
    ]
  }

  trustManager = {
    stores = [
      { path: "keys/mystore.jks" },
      { path: ${java.home}/lib/security/cacerts }
    ]
  }
}

and end up with a properly configured key manager and trust manager that contain all the keys from the various stores.

There’s a lot more to key stores and trust stores than I’ve mentioned here. For more details (including how to resolve improperly configured certificate chains), see:

And to muck with certificates inside a keystore, I recommend Portacle, a GUI key management system.

Configuring multiple clients

So now we have a configuration. But there’s another problem. There’s only one application.conf file, and all the WS methods are on the companion object:

WS.url("https://google.com")

This meant that if you have several web services, say “secure.com” and “loose.com”, you cannot set up different configuration profiles for them, or set up a client dynamically, or do isolated testing. Everything had to be handled when the Play configuration loads.

I broke apart the WS.client and added a WSClient trait that could call url in the same way. Now you can do this:

import com.typesafe.config.ConfigFactory
import play.api.libs.ws._
import play.api.libs.ws.ning._

val configuration = play.api.Configuration(ConfigFactory.parseString(
  """
    |ws.ssl.trustManager = …
  """.stripMargin))
val parser = new DefaultWSConfigParser(configuration)
val builder = new NingAsyncHttpClientConfigBuilder(parser.parse())
val secureClient : WSClient = new NingWSClient(builder.build())
val response = secureClient.url("https://secure.com").get()

and have much finer grained control over the TLS configuration.

Unfortunately, getting a client passed as an implicit parameter to WS.url is harder. I added a magnet pattern so you can do this:

object PairMagnet {
  implicit def fromPair(pair: Pair[WSClient, java.net.URL]) =
    new WSRequestHolderMagnet {
      def apply(): WSRequestHolder = {
        val (client, netUrl) = pair
        client.url(netUrl.toString)
      }
   }
}
import scala.language.implicitConversions
val client = WS.client
val exampleURL = new java.net.URL("http://example.com")
WS.url(client -> exampleURL).get()

and added another method WS.clientUrl that takes an implicit client:

implicit val sslClient = new play.api.libs.ws.ning.NingWSClient(sslBuilder.build())
WS.clientUrl("http://example.com/feed")

Debugging a client

While I was going through the client, I figured I may as well make it easier to turn on and off debugging as well.

Debugging is done by setting a system property, i.e. -Djavax.net.debug="ssl". Debugging output is written directly to System.out.println(), and the recommended way to change this is to change System.out. I can only hope this changes in JDK 1.8 — at the very least it should use java.util.logging — but it’s what there is for now.

I had the JSSE debug page and the debug section of the reference guide handy, so it was fairly simple to provide that in configuration rather than futz with system properties. I added certpath and “ocsp” (an undocumented debug property) as well, while I was checking for certificate validation.

ws.ssl.debug = [
 "certpath", "ocsp",

 # "all "  # defines all of the below
 "ssl",
 "defaultctx",
 "handshake",
   "verbose",
   "data",
 "keygen",
 "keymanager",
 "pluggability",
 "record",
   "packet",
   "plaintext",
 "session",
 "sessioncache",
 "sslctx",
 "trustmanager"
]

This is not a perfect solution, because system properties are global across all clients. But it’s better.

ADDENDUM: this only worked intermittently and eventually I figured out why and fixed it. The more sensitive among you may wish to avoid this link.

Next, it was time to figure out what went into the client. The most important thing is the protocol.

Choosing a protocol

TLS comes in different versions. In JSSE, the list is available here.

There are two calls that refer directly to the protocol names, the getInstance call:

val sslContext = SSLContext.getInstance("TLS") // or "TLSv1.2"

and the enabledProtocols list, which shows what the SSL context is willing to accept:

val enabledProtocols = sslContext.getDefaultParameters().getEnabledProtocols()

SSLv2 and SSLv2Hello (there is no v1) are obsolete and usage in the field is down to 25% on the public Internet. SSLv3 is known to have security issues and is still out in the field with 100% support. Virtually all HTTPS servers support it, and Mozilla Firefox still uses SSLv3 by default. They have a number of security issues compared to TLS.

TLSv1.2 is the current version, but early implementations of TLS 1.2 were prone to misconfiguration, which resulted in TLS 1.2 being disabled for the client by default in 1.7. Mozilla Firefox also has TLSv1.2 disabled, as of January 2014, and is only enabling it in the next version.

However, virtually all servers support TLS v1.0, and given our use cases, we expect that web services will have TLSv1.2 configured correctly. TLS 1.0 has been described as broken from the BEAST attack, but the attack seems to apply only to CBC ciphers, which we’re not obliged to use.

We want people to use the highest possible version of TLS. So we specify “TLSv1.2”, “TLSv1.1”, “TLSv1” in that order for JDK 1.7. For JDK 1.6, only “TLSv1” is available, so that’s what we have. We throw an exception on “SSLv3”, “SSLv2” and “SSLv2Hello”. If you don’t want that, then you have to explicitly set a ws.ssl.loose.allowWeakProtocols flag.

You can also specify the default protocol and the protocols list explicitly, i.e. if you want to configure JSSE for Suite B:

ws.ssl.protocol = "TLSv1.2" // passed into SSLContext.getInstance()

// passed into sslContext.getDefaultParameters().setEnabledProtocols()
ws.ssl.enabledProtocols = [
  "TLSv1.2"
]

Next, it’s time to figure out what cipher suite to pick.

Choosing a cipher suite

A cipher suite is really four different ciphers in one, describing the key exchange, bulk encryption, message authentication and random number function. In this particular case, we’re focusing on the bulk encryption cipher.

The JSSE list of cipher suites is here and there is an extensive comparison list. There’s a number of different ciphers available, and the list has changed substantially between JDK 1.7 and JDK 1.6.

In 1.7, the default cipher list is reportedly pretty good.

In 1.6, the default list is out of order — some of the weaker ciphers show up before the stronger ciphers do. Not only that, but 1.6 has no support for Elliptic Curve cryptography (ECC) ciphers, which are much stronger and allow for perfect forward secrecy.

Now, the client doesn’t control what cipher will eventually be used. The server does. As a client, there are two things that you can do:

You can present a list of ciphers which you are willing to accept.
You can refuse a cipher which you know to be weak.

In 1.7, we use the default cipher list.

For 1.6, the client provides a truncated cipher list based off Brian Smith’s list, with the ECC ciphers taken out and the 3DES cipher removed. Roughly 55% of the Internet uses RC4, and given that WS is a web services client, it will probably be talking to only a few services which are current.

val java16RecommendedCiphers: Seq[String] = Seq(
  "TLS_DHE_RSA_WITH_AES_256_CBC_SHA",
  "TLS_DHE_RSA_WITH_AES_128_CBC_SHA",
  "TLS_DHE_DSS_WITH_AES_128_CBC_SHA",
  "TLS_RSA_WITH_AES_256_CBC_SHA",
  "TLS_RSA_WITH_AES_128_CBC_SHA",
  "SSL_RSA_WITH_RC4_128_SHA",
  "SSL_RSA_WITH_RC4_128_MD5",
  "TLS_EMPTY_RENEGOTIATION_INFO_SCSV" // per RFC 5746
)

There are some ciphers which everyone agrees are bad though, and should never be used. NULL. Export Suite. DES. Anon. They are disabled by default, and the JSSE team says You are NOT supposed to use these cipher suites.

This brings up the next question: should WS consider RC4 and MD5 based ciphers to be weak? Surprisingly, probably not.

If you are setting up a server, you shouldn’t use RC4 and MD5 in your cipher suites, certainly. But if you’re a client, you’re probably fine talking to a server that is using RC4 or MD5.

In the case of RC4:

RC4 is horribly broken, and is horribly broken in ways that are meaningful to TLS. But the magnitude of RC4’s brokenness wasn’t appreciated until last year, and up until then, RC4 was a common recommendation for resolving both the SSL3/TLS1.0 BEAST attack and the TLS “Lucky 13” M-t-E attack. That’s because RC4 is the only widely-supported stream cipher in TLS. Moreover, RC4 was considered the most computationally efficient way to get TLS deployed, which 5-6 years ago might have been make-or-break for some TLS deployments. You should worry about RC4 in TLS —– but not that much: the attack is noisy and extremely time consuming. You should not be alarmed by MD5 in TLS, although getting rid of it is one of many good reasons to drive adoption of TLS 1.2.

— Thomas H. Ptacek

and:

The best, known attack against using RC4 with HTTPS involves causing a browser to transmit many HTTP requests — each with the same cookie — and exploiting known biases in RC4 to build an increasingly precise probability distribution for each byte in a cookie. However, the attack needs to see on the order of 10 billion copies of the cookie in order to make a good guess. This involves the browser sending ~7TB of data. In ideal situations, this requires nearly three months to complete.

— A roster of TLS cipher suites weaknesses

The case of MD5:

The MD5 hash function is broken, that is true. However, TLS doesn’t use MD5 in its raw form; it uses variants of HMAC-MD5, which applies the hash function twice, with two different padding constants with high Hamming distances (put differently, it tries to synthesize two distinct hash functions, MD5-IPAD and MD5-OPAD, and apply them both). Nobody would recommend HMAC-MD5 for use in a new system, but it has not been broken.

— Thomas H. Ptacek

and:

The attacks on HMAC-MD5 do not seem to indicate a practical vulnerability when used as a message authentication code.

— RFC 6151, section 2.3

Note that we are talking about use of MD5 in a cipher here — as a client, accepting an MD5 signed certificate is a different kettle of fish.

The jdk.tls.disabledAlgorithms constraint in 1.7 works fine to exclude bad or weak ciphers, and can also check small key sizes. In particular, before 1.8, ephemeral DH parameters were limited to 1024 bits, which is considered weak these days. The solution is to set jdk.tls.disabledAlgorithms="DHE keySize <= 1024, ECDHE keySize <= 1024". You may also need to do this if you’re on 1.7, as there’s a nasty bug in 1.7 that causes connections to fail 0.05% of the time — although frankly, upgrading to 1.8 is a much better solution.

Note that jdk.tls.disabledAlgorithms is a security property, not a system property, and it’s null by default:

scala> java.security.Security.getProperty("jdk.tls.disabledAlgorithms")
res9: String = null

Also note that, as the author says, “There is no such handy approach in JDK 6 and 5.0”.

Play supports 1.6, so we make our own “handy approach”: we check the cipher list at configuration time, and throw an exception if we find that there is a weak cipher in the list. If you want to turn off the check, you have to configure the ws.ssl.loose.acceptWeakCiphers flag.

I don’t think that ciphers need to be checked at run time, as I don’t think that the client will accept a cipher from the server that is not already in the client list. However, if you’re going to be paranoid, you probably want to set jdk.tls.disabledAlgorithms anyway.

As with protocols, you can configure the cipher list by hand:

ws.ssl.enabledCiphers = [
  "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384"
  "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"
]  

And that about wraps things up for cipher suites.

X.509 Certificates!

Building a Development Environment with Docker

wsargent@tersesystems.com (Will Sargent) — Wed, 20 Nov 2013 22:22:00 -0800

TL;DR

I’ve written a cheat sheet for Docker, and I have a github project for it. Here’s the thinking that went into why Docker, and how best to use it.

The problem

You want to build your own development environment from scratch, and you want it to be as close to a production environment as possible.

Solutions

Development environments usually just… evolve. There’s a bunch of tries at producing a consistent development environment, even between developers. Eventually, through trial and error, a common set of configuration files and install instructions turns into something that resembles a scaled down and testable version of the production environment, managed through version control and a set of bash scripts.

But even when it gets to that point, it’s not over, because modern environments can involve dozens of different components, all with their own configuration, often times communicating with each other through TCP/IP or even worse, talking to a third party API like S3. To replicate the production environment, these lines of communication must be drawn — but they can’t be squashed into one single machine. Something has to give.

Solution #1: Shared dev environment

The first solution is to set up a environment with exactly the same machines in the same way as production, only scaled down for development. Then, everyone uses it.

This works only if there is no conflict between developers, and resource use and contention is not a problem. Oh, and you don’t want to swap out one of those components for a particular team.

If you need to access the environment from outside the office, you’ll need a VPN. And if you’re on a flaky network or on a plane, you’re out of luck.

Solution #2: Virtual Machines

The second solution is to put as much of the environment as possible onto the developer’s laptop.

Virtual Machines such as VirtualBox will allow you to create an isolated dev environment. You can package VMs into boxes with Vagrant, and create fresh VMs from template as needed. They each have their own IP address, and you can get them to share filesystems.

However, VMs are not small. You can chew up gigabytes very easily providing the OS and packages for each VM, and those VMs do not share CPU or memory when running together. If you have a complex environment, you will run into a point where you either run out of disk space or memory, or you break down and start packaging multiple components inside a single VM, producing an environment which may not reflect production and is far more fragile and prone to complexities.

Solution #3: Docker

Docker solves the isolation problem. Docker provides (consistent, reproducible, disposable) containers that make components appear to be running on different machines, while sharing CPU and memory underneath, and provides TCP/IP forwarding and filesystems that can be shared between containers.

So, here’s how you build a development environment in Docker.

Docker Best Practices

Build from Dockerfile

The only sane way to put together a dev environment in Docker is to use raw Dockerfile and a private repository. Pull from the central docker registry only if you must, and keep everything local.

Chef recipes are slow

You might think to yourself, “self, I don’t feel like reinventing the wheel. Let’s just use chef recipes for everything.”

The problem is that creating new containers is something that you’ll do lots. Every time you create a container, seconds will count, and minutes will be totally unacceptable. It turns out that calling apt-get update is a great way to watch nothing happen for a while.

Use raw Dockerfile

Docker uses a versioned file system called AUFS, which identifies commands it can run from layers (aka cached fs) and pulls out the appropriate version. You want to keep the cache happy. You want to put all the mutable stuff at the very end of the Dockerfile, so you can leverage cache as much as possible. Chef recipes are a black box to Docker.

The way this breaks down is:

Cache wins.
Chef, ansible, etc, does not use cache.
Raw Dockerfile uses cache.
Raw Dockerfile wins.

There’s another way to leverage Docker, and that’s to use an image that doesn’t start off from ubuntu or basebox. You can use your own base image.

The Basics

Install a internal docker registry

Install an internal registry (the fast way) and run it as a daemon:

docker run -name internal_registry -d -p 5000:5000 samalba/docker-registry

Alias server to localhost:

echo "127.0.0.1      internal_registry" >> /etc/hosts

Check internal_registry exists and is running on port 5000:

apt-get install -y curl
curl –get –verbose http://internal_registry:5000/v1/_ping

Install Shipyard

Shipyard is a web application that provides an easy to use interface for seeing what Docker is doing.

Open up a port in your Vagrantfile:

config.vm.network :forwarded_port, :host => 8005, :guest => 8005

Install shipyard from the central index:

SHIPYARD=$(docker run \
    -name shipyard \
  -p 8005:8000 \
  -d \
  shipyard/shipyard)

You will also need to replace /etc/init/docker.conf with the following:

description "Docker daemon"

start on filesystem and started lxc-net
stop on runlevel [!2345]

respawn

script
        /usr/bin/docker -d -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock
end script

THen reboot the VM.

Once the server has rebooted and you’ve waited for a bit, you should have shipyard up. The credentials are “shipyard/admin”.

Go to http://localhost:8005/hosts/ to see Shipyard’s hosts.
In the vagrant VM, ifconfig eth0 and look for “inet addr:10.0.2.15” — enter the IP address.

Create base image

Create a Dockerfile with initialization code such as `apt-get update / apt-get install’ etc: this is your base.
Build your base image, then push it to the internal registry with docker build -t internal_registry:5000/base .

Build from your base image

Build all of your other Dockerfile pull from “base” instead of ubuntu.

Keep playing around until you have your images working.

Push your images

Push all of your images into the internal registry.

Save off your registry

if you need to blow away your Vagrant or set someone else up, it’s much faster to do it with all the images still intact:

docker export internal_registry > internal_registry.tar
gzip internal_registry.tar
mv internal_registry.tar.gz /vagrant

Tips

docker add blows away the cache, don’t use it (bug, possibly fixed).
There’s a limit to the number of layers you can have, pack your apt-get onto a single line.
Keep common instructions at the top of the Dockerfile to leverage the cache as long as possible.
Use tags when building (Always pass the -t option to docker build).
Never map the public port in a Dockerfile.

Exposing Services

If you are running a bunch of services in Docker and want to expose them through Virtualbox to the host OS, you need to do something like this in your Vagrant:

(49000..49900).each do |port|
  config.vm.network :forwarded_port, :host => port, :guest => port
end

Let’s start up Redis:

docker pull johncosta/redis
docker run -p 6379 -d johncosta/redis

Then find the port:

docker ps
docker port <redis_container_id> 6379

Then connect to the 49xxx port that Virtualbox exposes.

Cleanup

docker ps -a | grep 'weeks ago' | awk '{print $1}' | xargs docker rm

eliminate:

docker rm `docker ps -a -q`

Running from an existing volume

docker run -i -t -volumes-from 5ad9f1d9d6dc mytag /bin/bash

Sources

Phusion Base Image
Dockerfile Best Practices
How Mailgun uses Docker (video is more complete than the blog post).
Docker Dev Environment in 24 Hours

Play in Practice

wsargent@tersesystems.com (Will Sargent) — Sat, 20 Apr 2013 14:20:00 -0700

I gave a talk on Play in Practice at the SF Scala meetup recently. Thanks to Stackmob for hosting us and providing pizza.

I went into describing how to implementing CQRS in Play, but there was a fairly long question and answer section about Play as well. I couldn’t go into detail on some of the answers and missed some others, so I’ll fill in the details here.

Video

Slides

Core API

The core API is Action, which take in a Request and return a Result. The Request is immutable, but you can wrap it with extra information, which you’ll typically do with action composition. 2.1.1 introduced EssentialAction, which uses (RequestHeader => Iteratee[Array[Byte], Result]) instead of Action’s (Request => Result) and makes building Filters easier.

Again, Play’s core is simple. About as simple as you can get.

Streaming

Streaming is handled by Iteratees, which can be a confusing topic for many people. There are good writeups here and here. lila is the best application to look at for streaming, especially for sockets and hubs.

Having good streaming primitives is something that I didn’t get into that much in the talk, but is still vitally important to “real time web” stuff.

Filters

If you want to do anything that you’d consider as part of a “servlet pipeline”, you use Filters, which are designed to work with streams.

An example of a good Filter is to automatically uncompress an asset — here’s an example that uses an Enumeratee:

class GunzipFilter extends EssentialFilter {
  def apply(next: EssentialAction) = new EssentialAction {
    def apply(request: RequestHeader) = {
      if (request.headers.get("Content-Encoding").exists(_ == "gzip")) {
        Gzip.gunzip() &>> next(request)
      } else {
        next(request)
      }
    }
  }
}

Note that this only does uncompression: Automatic streaming gzip compression of templates is not available “out of the box” in 2.1.2, but it should be available in Play 2.2.

Templating

Play comes packaged with its own template language, Twirl, but you’re not required to use it. There is an integration into Scalate that gives you Mustache, Jade, Scaml and SSP. There’s also an example project that shows how to integrate Play with Freemarker.

One thing that Play doesn’t address directly is how to set up a structure for page layouts. Play provides you with index.scala.html and main.scala.html, but doesn’t provide you with any more structure than that. If you set up a header and footer and allow for subdirectories to use their own templates, you can minimize the amount of confusion in the views.

There’s an example in RememberMe, and this is the approach that lila takes as well.

Another thing is that Play’s default project template is intentionally minimal. If you use Backbone and HTML5 templates, then a custom giter8 template like mprihoda/play-scala may suit you better.

JSON

Play’s JSON API is very well done, and is a great way to pass data around without getting into the weeds or having to resort to XML. It goes very well with case classes.

The documentation isn’t bad, but Pascal Voitot (the author of play-json) has a series of blog posts that go the extra mile: reading JSON with JsPath, writing JSON formats, transforming JSON, and even defining JSON macros.

Forms

Form handling is one of those things that is never intuitive for me. The documentation helps, but really if you want to know how to do validation, using the sample forms application is the best way to pick things up. There are many useful nuggets that aren’t explicitly discussed in the documentation. In particular, the ability to make custom constraints is extremely useful.

Routing

There’s only one routing API replacement that I know of, Play Navigator, a routing DSL for REST services. However, you can use custom data types in the routing table using QueryStringBindable and PathBindable, and save yourself some “string2foo” conversion.

Asynchronous Operation

Talking about Akka (and the other async code) in Play is tricky for a couple of reasons.

The first reason is that “async” involves a number of different concepts, all of which are complex and worthy of blog posts in themselves. Sadek Drobi gives a nice overview, and there’s an exhaustive mailing list discussion about the asynchronous code in Play works.

The second bit of trickiness is that Play 2.0 and Play 2.1 async features do not work in the quite the same way.

Play 2.0 uses Akka for almost everything internally.

Play 2.1 does not use Akka to handle incoming requests, or iteratees, or internal code. It uses scala.concurrent.Future instead with its own thread pools.

Play 2.1 also uses a default thread pool, which is Akka backed — ActorSystem("play") — and is used for the application code, i.e. the stuff inside Action.

This is important, because blog posts like James Ward’s Optimizing Play 2 for Database Driven Apps are only applicable to Play 2.0, not 2.1. For 2.1, use the thread pools documentation.

In addition to the “play” actor system, there’s a Play Akka plugin. The Akka plugin is actually packaged with Play itself, and you can find it under play.api.libs.concurrent.Akka.

So, if Play already uses Akka under the hood, then why define an Akka plugin?

I believe it’s because the Akka plugin defines a distinct ActorSystem("application") that can be used for backend tasks like sending email, and can be configured without impacting the “play” ActorSystem. The Akka plugin provides a useful default and enforces seperation between Play’s actors and the application’s actors.

CQRS

Given that most of the CQRS talks I’ve read have been from the enterprise perspective, it was nice to talk about CQRS in the context of functional programming and statelessness.

Message passing is something that is typically mentioned in inter process communication, or in message oriented middleware. Akka — a message passing architecture on the thread level — allows us to build “zero coupling” systems . As message passing patterns, CQRS and DDD are a good set of idioms to think about domain logic together, especially since they already assume eventual consistency and indeterminate time.

Authentication

If you’re using Scala, there are two good authentication options, RememberMe (ahem) and SecureSocial. SecureSocial has better documentation and has been around longer, but RememberMe has better security resistance to some attacks. I’m working to integrate RememberMe’s functionality into SecureSocial, but you’ll want to check out both of them.

There’s also a pure Java authentication option: Play Authenticate. I haven’t used this, but the code looks reasonable.

If you’d rather go it alone or need a basic starter application, you may find Play20StartApp useful (password reset, account confirmation, etc.)

Authorization

Deadbolt 2 is the best known authorization framework. You can use things like Shiro, but you’re better off with something specifically designed for Play.

Security

Play does fairly well on security compared to other frameworks. For example, it will set a CORS header to protect against clickjacking, will sign the session cookie with an HMAC to protect against broken authentication, supports SSL, etc.

However, there are some things that Play doesn’t do.

Play doesn’t encrypt the session cookie, so you shouldn’t store any sensitive information in there.

Play won’t protect you from replay attacks, as Play is stateless by default. You can specify a nonce or request counter to counteract this, and RememberMe uses a token based approach for persistent login cookies.

Play won’t protect you against injection attacks. You can specify value classes to validate your input against raw strings.

Play won’t protect you against security misconfiguration. You should have a release checklist.

Play won’t protect you from insecure cryptography practices. Education helps, but there’s a lot of misinformation out there as well; watch this video (and slides) and be wary of things you read on Stack Overflow and Hacker News.

Play won’t protect you from failure to restrict URL access; that’s up to the authorization framework.

Play does have cross site request forgery protection, but it will only be effective if you enable the filter and explicitly pass the CSRF helper function in through every single form. There is an authenticity token approach as well, though I haven’t used it.

Most importantly, Play won’t tell you about how web application security fails. I recommend The Tangled Web as an excellent overview on how web applications are stitched together out of different technologies, and how to secure them.

Logging

The underlying logger for Play is Logback. Logback is one of the few hardcoded dependencies in Play, which has caused some issues. Fortunately, Play uses Logback through the SLF4J logging API, but there’s no option built into Play to allow Logback to be swapped out easily. There are reports of people swapping out Logback for other logging frameworks, but I haven’t tried them.

There have also been issues with the logging configuration conflicting in places or being unclear. One thing that has tripped people up repeatedly is that all the logging configuration must be done in one place. You can’t have some logging configuration in application.conf and some configuration in logger.xml.

While Play uses SLF4J under the hood, it doesn’t expose SLF4J functionality in play.api.Logger. In fact, there are only two method signatures for logging:

  def error(message: => String) : Unit
  def error(message: => String, error: => Throwable) : Unit

This doesn’t really cover the way I like to log, and it doesn’t provide even the features that are available in SLF4J, such as parameterized logging. My own answer was to ignore the Play logging API entirely and write a Logging wrapper directly against SLF4J (with kestrel combinators, natch), but you may want to use something out of the box.

For example, Typesafe Logging, uses SLF4J and provides you with this:

  def error(message: String): Unit
  def error(message: String, params: AnyRef*): Unit
  def error(message: String, t: Throwable): Unit
  def error(marker: Marker, message: String): Unit
  def error(marker: Marker, message: String, params: AnyRef*): Unit
  def error(marker: Marker, message: String, t: Throwable): Unit

Or you can use loglady, which uses the Python API style with printf syntax:

  def error(message: String,  params: Any*) : Unit
  def error(thrown: Throwable, message: String,  params: Any*) : Unit

WAR packaging

I said in the Q&A that I didn’t think you could package Play 2 applications as WAR files. Well, it turns out that there is a plugin available, and it works with Servlet 3.0 and 2.5 containers (Tomcat 6/7, Jetty 7/8/9, JBoss 5/6/7, etc). You may need to tweak the logger to work in the container correctly.

I don’t know how Play’s performance is affected by running inside a servlet container; let me know if it works for you.

Asset Packaging

Javascript assets in Play are minified using Google Closure — this happens automatically on play dist. They also can be gzipped using a custom SBT script.

This makes a good enough solution for most people. If you are really intent on minimizing your asset overhead, you should consider putting your assets on a static file server backed by HAProxy, or putting them on CDN.

Email

Email is one of those things that I think should be divorced as much as possible from Play. It’s backend and async by nature, and this makes it something that is best handled through Akka.

akka-email is available on Github and gives you a starting place to build up a message passing infrastructure for email.

Metrics

Instrumenting applications is important. Sadly, every metrics solution has its own API, so you can’t easily switch between them. However, there’s no shortage of options.

New Relic recently came out with support for Play 2.
Ostrich, the Twitter metrics library.
Metrics, with the metrics-scala from Erik Van Oosten, cross-compiled for multiple versions. This is what I use.
Pillage, which has a Scala option (I have not tried this).
statsd module for Play 2.

The Typesafe Console is the best monitoring tool to use if you are using Akka, but that depends on having a Typesafe subscription if you want to use it in production.

Load and Stress Testing

Determining a load plan is hard, and involves some amount of educated guessing. Fortunately, most applications simply don’t get that much load, even ones you’d think would be busy.

Gatling and wrk are good ways of stressing a system, but they don’t reflect normal user behavior. Apache JMeter is very good at modelling random user behavior, but is clunky. A good and arguably the most realistic load test is to hire a couple of hundred users from Mechanical Turk to pound on the site at once, but this may not be very convenient.

Deployment

There are a number of different ways to deploy Play projects. Using play dist gets you most of the way, but you may want to deploy with Ansible or Chef or Fabric. Or you can use upstart or even git hooks.

If you just want to push changes to a staging server as they happen, you can do this with rsync -avz –delete -e ssh $deployed_code staging:/opt/play-app, although this isn’t so great for production.

Java Support

The Java and Scala APIs are very similar. However, there are a couple of notable differences, which come out of Java’s lack of closure support:

The Java API does not support Iteratees.
The Java API does not have an implicit execution context.

The play.libs.F library goes a fair way to providing Scala’s functional programming constructs in Java.

More?

If you have suggestions or want to point something out, please email me at will.sargent@gmail.com, and I’ll fill out this post with more details.

Error Handling in Scala

wsargent@tersesystems.com (Will Sargent) — Thu, 27 Dec 2012 15:38:00 -0800

The previous post was mostly about programming “in the small” where the primary concern is making sure the body of code in the method does what it’s supposed to and doesn’t do anything else. This blog post is about what to do when code doesn’t work — how Scala signals failure and how to recover from it, based on some insightful discussions.

First, let’s define what we mean by failure.

Unexpected internal failure: the operation fails as the result of an unfulfilled expectation, such as a null pointer reference, violated assertions, or simply bad state.
Expected internal failure: the operation fails deliberately as a result of internal state, i.e. a blacklist or circuit breaker.
Expected external failure: the operation fails because it is told to process some raw input, and will fail if the raw input cannot be processed.
Unexpected external failure: the operation fails because a resource that the system depends on is not there: there’s a loose file handle, the database connection fails, or the network is down.

Java has one explicit construct for handling failure: Exception. There’s some difference of usage in Java throughout the years — IO and JDBC use checked exceptions throughout, while other API like org.w3c.dom rely on unchecked exceptions. According to Clean Code, the best practice is to use unchecked exceptions in preference to checked exceptions, but there’s still debate over whether unchecked exceptions are always appropriate.

Exceptions

Scala makes “checked vs unchecked” very simple: it doesn’t have checked exceptions. All exceptions are unchecked in Scala, even SQLException and IOException.

The way you catch an exception in Scala is by defining a PartialFunction on it:

val input = new BufferedReader(new FileReader(file))
try {
  try {
    for (line <- Iterator.continually(input.readLine()).takeWhile(_ != null)) {
      Console.println(line)
    }
  } finally {
    input.close()
  }
} catch {
  case e:IOException => errorHandler(e)
}

Or you can use control.Exception, which provides some interesting building blocks. The docs say “focuses on composing exception handlers”, which means that this set of classes supplies most of the logic you would put into a catch or finally block.

Exception.handling(classOf[RuntimeException], classOf[IOException]) by println apply {
  throw new IOException("foo")
}

Using the control.Exception methods is fun and you can string together exception handling logic to create automatic resource management, or an automated exception logger. On the other hand, it’s full of sharp things like allCatch. Leave it alone unless you really need it.

Another important caveat is to make sure that you are catching the exceptions that you think you’re catching. A common mistake (mentioned in Effective Scala) is to use a default case in the partial function:

try {
  operation()
} catch {
  case _ => errorHandler(e)
}

This will catch absolutely everything, including OutOfMemoryError and other errors that would normally terminate the JVM.

If you want to catch “everything” that would normally happen, then use NonFatal:

import scala.util.control.NonFatal

try {
  operation()
} catch {
  case NonFatal(exc) => errorHandler(e)
}

Exceptions don’t get mentioned very much in Scala, but they’re still the bedrock for dealing with unexpected failure. For unexpected internal failure, there’s a set of assertion methods called require, assert, and assume, which all use throwables under the hood.

Option

Option represents optional values, returning an instance of Some(A) if A exists, or None if it does not. It’s ubiquitous in Scala code, to the point where it fades into invisibility. The cheat sheet is the best way to get a handle on it.

It’s almost impossible to use Option incorrectly, but there is one caveat: Some(null) is valid. If you have code that returns null, wrap it in Option() to convert it:

val optionResult = Option(null) // optionResult is None.

Either

Either is a disjoint union construct. It returns either an instance of Left[L] or an instance of Right[R]. It’s commonly used for error handling, where by convention Left is used to represent failure and Right is used to represent success. It’s perfect for dealing with expected external failures such as parsing or validation.

case class FailResult(reason:String)

def parse(input:String) : Either[FailResult, String] = {
  val r = new StringTokenizer(input)
  if (r.countTokens() == 1) {
    Right(r.nextToken())
  } else {
    Left(FailResult("Could not parse string: " + input))
  }
}

Either is like Option in that it makes an abstract idea explicit by introducing an intermediate object. Unlike Option, it does not have a flatMap method, so you can’t use it in for comprehensions — not safely at any rate. You can use a left or right projection if you’re not interested in handling failure:

val rightFoo = for (outputFoo <- parse(input).right) yield outputFoo

More typically, you’ll use fold:

parse(input).fold(
  error => errorHandler(error),
  success => { … }
)

You’re not limited to using Either for parsing or validation, of course. You can use it for CQRS.

case class UserFault
case class UserCreatedEvent

def createUser(user:User) : Either[UserFault, UserCreatedEvent]

or arbitary binary choices:

def whatShape(shape:Shape) : Either[Square, Circle]

Either is powerful, but it’s trickier than Option. In particular, it can lead to deeply nested code. It can also be misunderstood. Take the following Java lookup method:

public Foo lookup(String id) throws FooException // throw if not found or db exception

Scala has Option, so we can use that. But what if the database goes down? Using the error reporting convention of Either might suggest the following:

def lookup() : Either[FooException,Option[Foo]]

But this is awkward. If you return Either because something might fail unexpectedly, then immediately half your API becomes littered with Either[Throwable, T].

Ah, but what if you’re modifying a new object?

def modify(inputFoo:Foo) : Either[FooException,Foo]

If you’re dealing with expected failure and there’s good odds that the operation will fail, then returning Either is fine: create a case class representing failure FailResult and use Either[FailResult,Foo].

Don’t return exceptions through Either. If you want a construct to return exceptions, use Try.

Try

Try is similar to Either, but instead of returning any class in a Left or Right wrapper, it returns Failure[Throwable] or Success[T]. It’s an analogue for the try-catch block: it replaces try-catch’s stack based error handling with heap based error handling. Instead of having an exception thrown and having to deal with it immediately in the same thread, it disconnects the error handling and recovery.

Try can be used in for comprehensions: unlike Either, it implements flatMap. This means you can do the following:

val sumTry = for {
  int1 <- Try(Integer.parseInt("1"))
  int2 <- Try(Integer.parseInt("2"))
} yield {
  int1 + int2
}

and if there’s an exception returned from the first Try, then the for comprehension will terminate early and return the Failure.

You can get access to the exception through pattern matching:

sumTry match {
  case Failure(thrown) => {
    Console.println("Failure: " + thrown)
  }
  case Success(s) => {
    Console.println(s)
  }
}

Or through failed:

if (sumTry.isFailure) {
  val thrown = sumTry.failed.get
}

Try will let you recover from exceptions at any point in the chain, so you can defer recovery to the end:

val sum = for {
  int1 <- Try(Integer.parseInt("one"))
  int2 <- Try(Integer.parseInt("two"))
} yield {
  int1 + int2
} recover {
  case e => 0
}

Or recover in the middle:

val sum = for {
  int1 <- Try(Integer.parseInt("one")).recover { case e => 0 }
  int2 <- Try(Integer.parseInt("two"))
} yield {
  int1 + int2
}

There’s also a recoverWith method that will let you swap out a Failure:

val sum = for {
  int1 <- Try(Integer.parseInt("one")).recoverWith {
    case e: NumberFormatException => Failure(new IllegalArgumentException("Try 1 next time"))
  }
  int2 <- Try(Integer.parseInt("2"))
} yield {
  int1 + int2
}

You can mix Either and Try together to coerce methods that throw exceptions internally:

val either : Either[String, Int] = Try(Integer.parseInt("1")).transform({ i => Success(Right(i)) }, { e => Success(Left("FAIL")) }).get
Console.println("either is " + either.fold(l => l, r => r))

Try isn’t always appropriate. If we go back to the first exception example, this is the Try analogue:

val input = new BufferedReader(new FileReader(file))
val results = Seq(
  Try {
    for (line <- Iterator.continually(input.readLine()).takeWhile(_ != null)) {
      Console.println(line)
    }
  },
  Try(input.close())
)

results.foreach { result =>
  result.recover {
    case e:IOException => errorHandler(e)
  }
}

Note the kludge to get around the lack of a finally block to close the stream. Victor Klang and Som Snytt suggested using a value class and transform to pimp Try:

implicit class TryOps[T](val t: Try[T]) extends AnyVal {
  def eventually[Ignore](effect: => Ignore): Try[T] = {
    val ignoring = (_: Any) => { effect; t }
    t transform (ignoring, ignoring)
  }
}

Try(1 / 0).map(_ + 1) eventually { println("Oppa Gangnam Style") }

Which is cleaner, at the cost of some magic.

Try was originally invented at Twitter to solve a specific problem: when using Future, the exception may be thrown on a different thread than the caller, and so can’t be returned through the stack. By returning an exception instead of throwing it, the system is able to reify the bottom type and let it cross thread boundaries to the calling context.

Try is new enough that people are still getting comfortable with it. I think that it’s a useful addition when try-catch blocks aren’t flexible enough, but it does have a snag: returning Try in a public API means exceptions must be dealt with by the caller. Using Try also implies to the caller that the method has captured all non fatal exceptions itself. If you’re doing this in your trait:

def modify(foo:Foo) : Try[Foo]

Then Try should be at the top to ensure exception capture:

def modify(foo:Foo) : Try[Foo] = Try {
  Foo()
}

Because exceptions must be dealt with the caller, you are placing more trust in the caller to handle or delegate a failure appropriately. With try-catch blocks, doing nothing means that the exception can pass up the stack to a top level exception handler. With Try, exceptions must be either returned or handled by each method in the chain, just like checked exceptions.

To pass the exception along, use map:

def fooToString(foo:Foo) : Try[String] = {
  modify(foo).map { outFoo =>
   outFoo.toString()
  }
}

Or to rethrow the exception up the stack if the return type is Unit:

def doStuff : Unit = {
  val modifiedFoo = modify(foo).get // throws the exception if failure
}

And you want to avoid this:

modify(foo) match {
  case Failure(f) => {
    // database failure?  don't care, swallow exception.
  }
  case Success(s) => {
    …
  }
}

If you have a system that needs specific error logging or error recovery, it’s probably safer to stick to unchecked exceptions.

TL;DR

Throw Exception to signal unexpected failure in purely functional code.
Use Option to return optional values.
Use Option(possiblyNull) to avoid instances of Some(null).
Use Either to report expected failure.
Use Try rather than Either to return exceptions.
Use Try rather than a catch block for handling unexpected failure.
Use Try when working with Future.
Exposing Try in a public API has a similiar effect as a checked exception. Consider using exceptions instead.

Problems Scala Fixes

wsargent@tersesystems.com (Will Sargent) — Sun, 16 Dec 2012 14:47:00 -0800

When I tell people I write code in Scala, a typical question is well, why? When it comes to writing code, most of my work is straightforward: SQL database on the backend, some architectural glue, CRUD, some exception handling, transactions handlers and an HTML or JSON front end. The tools have changed, but the problems are usually the same: you could get a website up in 5 minutes with Rails or Dropwizard. So why pick Scala?

It’s a tough question to answer off the bat. If I point to the language features, it doesn’t get the experience across. It’s like explaining why I like English by reading from a grammar book. I don’t like Scala because of its functional aspects or its higher kinded type system. I like Scala because it solves practical, real world problems for me.

You can think of Scala as Java with all the rough edges filed off, with new features that make it easier to write correct code and harder to create bugs. Scala is not a purist’s language — it goes out of its way to make it easy for Java programmers to dip their toes in the pool. You can literally take your Java code and hit a key to create working Scala code.

So what problems does Scala solve?

Let’s start with the single biggest problem in programming, the design flaw that’s caused more errors than anything else combined. Null references.

Solving for Null

Scala avoids null pointer references by providing a special type called Option. Methods that return Option[A] (where A is the type that you want, i.e. Option[String]) will give you an object that is either a wrapper object called ‘Some’ around your type, or None. There are a number of different ways you can use Option, but I’ll just mention the ones I use most. You can chain Options together in Scala using for comprehensions:

  for {
     foo <- request.params('foo')
     bar <- request.params('bar')
  } yield myService.process(foo, bar)

or through a map:

  request.params('foo').map { foo => logger.debug(foo) }

or through pattern matching.

  request.params('foo') match {
    case Some(foo) => { logger.debug(foo) }
    case None => { logger.debug('no foo :-(') }
  }

Not only is this easy, but it’s also safer. You can flirt with NPE saying myOption.get, but if you do that, you deserve what you get. Not having to deal with NPE is a pleasure.

Right Type in the Right Place

What’s the second biggest problem in programming? It’s a huge issue in security and in proving program correctness: invalid, unchecked input.

Take the humble String. The work of manipulating strings is one of the biggest hairballs in programming — they’re pulled in from the environment or embedded in the code itself, and then programs try to figure out how best to deal with them. In one case, a string is displayed to the user and it’s done. In another case, an SQL query is embedded as a query parameter on a web page and passed straight through to the database. To the compiler, they’re just strings and there is no difference between them. But there are some types of strings that are suitable to pass to databases, and some which are not. Ideally, we’d like to tell the compiler that SQL and query parameters have different types. Scala makes this easy.

With the Shapeless library, you can add distinguishing type information to objects and ensure that you can’t pass random input in:

import shapeless.TypeOperators._
type SqlString = Newtype[String, Any]
val x: SqlString = newtype("SELECT * FROM USER")

I’ve called out strings because it’s a good example, but you can also do this for repository IDs. No more this:

  case class User(id: Int, firstName:String)
  def lookup(id:Int) : User

When you can have this:

  case class User(id: Id[User], firstName:String)
  def lookup(id:Id[User]) : User

You can also use this to validate input on the front end. One of the big problems with regular expressions is that when you parse a random string for certain kinds of input, you get back… more strings. You may be validating a string as a username (no spaces, no odd characters), but what you’ve got at the end is a string that says it’s a username.

val rawInput = request.params('foo')
if (isUsername(rawInput)) {
  val username = rawInput
}

You can replace that with something nicer.

   val email : Option[Username] = parseUsername(rawInput)

This embeds the constraint in the type itself. You can design your API to accept Username instead of String, and so enforce a kind of whitelisting.

Can you do this in Java? Yes, but it’s inconvenient. Scala’s type system makes it easy for you, and in 2.10 there will be Value Classes, which will provide this functionality in the core language itself.

Doing the gruntwork for you

The previous example can be improved though. Really, we just want a Username at the end — we don’t want to have to call parseUsername on it. Fortunately, Scala rewards the lazy with implicit conversions. If you define a method like this and use the implicit keyword:

  implicit def string2username(input : String) : Option[Username] = parseUsername(input)

And do this:

  val email : Option[Username] = rawInput;

Then the compiler is smart enough to see that a String isn’t an Option[Username], and looks through any implicit methods available to do the conversion.

There is an element of ‘magic’ to implicit conversions, especially when you’re reading someone else’s code and trying to figure out where the conversion is happening. You can find the appropriate implicit through the REPL, or through IDEA.

Providing Context

There are many cases in programming where everything depends on a Context object in some way: either you’re using a database connection, or you rely on a security principal, or you’re resolving objects from a request or JAAS / LDAP / Spring context… the list goes on. Whatever it is, it’s passed in by the system, it’s absolutely essential, and you can count on most of your API to depend on it in some way. A typical Java way to deal with this is to make it part of the parameter list, or try to ignore it and make it a ThreadLocal object.

   public void doStuff(Context context);

Scala has a better way to deal with this: you can specify implicit parameters on a method.

   def doStuff(implicit context:Context)

which means that anything marked as implicit that is in scope will be applied:

   implicit val context = new Context()
   doStuff  // uses val context automatically.

This is all handled by the compiler: just set up the implicits and Scala will do the rest.

A place for everything

So now you have a number of implicit methods, value classes and type definitions and wotnot. In Scala, there’s a place to keep all this stuff that is so intuitive, you may not think of it as a place at all. It’s the package object.

Package objects are supremely useful. You define a file called package.scala, then in the file you put

package object mypackagename {
  implicit def string2username(input : String) : Option[Username] = parseUsername(input)
}

and after that point, anything with ‘import mypackagename._’ will import the package object as well. One less thing to think about.

Free Data Transfer Objects

Case classes. So called because they’re used in case statements (see below).

case class Data(propertyOne:String, propertyTwo:Int)

Immutable, convenient, and packed with functionality. They make creating data types or DTOs trivial. They’re cool.

Free Range (Organic) Checking

Scala contains a powerful pattern matching feature. You can think of it as a switch statement on steroids.

a match {
   case Something => doThis
   case SomethingElse => doThat
}

There are so many things that feed into pattern matching — extractor objects, aliases, matching on types, regular expressions and wildcards — it’s the ‘regexp’ of Scala. It takes in an object as input, filters it, and manipulates it in exactly the way you want.

But the thing I really like about pattern matching is what it doesn’t let you do. It doesn’t let you miss something.

There’s a feature called sealed classes which lets you define all the valid types in a file. If you define a trait with the sealed keyword inside a file, then any classes you define inside that file that extend that trait are the ONLY classes that will extend that trait.

sealed trait Message { def msg: String }
case class Success(msg:String) extends Message
case class Failure(msg:String) extends Message

The compiler knows this, and so when you write use pattern matching against that trait, it knows that it must be one of the case classes defined. If not all of the case classes are defined in the match, it will print out a warning method saying that you don’t have an exhaustive match.

def log(msg: Message) = msg match {
  case Success(str) => println("Success: " + str)
  case Failure(str) => println("Failure: " + str)
}

And More

But that’s enough for now. I hope this gives you an idea of why I like Scala. If you have any features dear to your heart, add them to the comments and let me know what makes you happy.

Remember Me Cookies for Play 2.0

wsargent@tersesystems.com (Will Sargent) — Sat, 07 Jul 2012 15:19:00 -0700

I’ve been working with Play 2.0 for a while now, and in many ways it’s the ideal web framework for me: it’s a light framework that gets a request, puts together a result (either at once or in chunks using an iteree pattern), and provides some HTML templates and form processors for ease of use. It lets you change code and templates while the server is running, and gives you an asset pipeline for compressing LESS and Coffeescript into minified CSS and Javascript out of the box.

That being said, it’s a new web framework, and the biggest issue right now is all the boring infrastructure that goes on top of it to make a framework deal with authentication, authorization, and even boring things like resetting a password.

On the Java side, Yvonnick Esnault has a good starter application (disclaimer; I contributed some code), or you can use Play Authenticate.

On the Scala side, play20-auth is a good starting point for an authentication system. However, it didn’t do token based authentication, aka “Remember Me” cookies. Adding this feature turns out to be tricky if you’re new to Scala, because extending the request pipeline in Play 2.0 Scala requires that you know a functional style of programming called “action composition”.

So here’s a boilerplate project play20-rememberme that does authentication with remember me functionality (although it doesn’t have the password reset or confirm features added to Play20StartApp).

UPDATE: Now works with Play 2.1.

The User Illusion

wsargent@tersesystems.com (Will Sargent) — Sat, 26 May 2012 15:51:00 -0700

Slides from the May 17th Five Minutes of Fame. This one is on consciousness and a book called The User Illusion. I read the book and thought it interesting, but it was only after reading Blindsight and following Peter Watts’s blog that it clicked as something that happened outside of science experiments.

That, and I’d really been hankering for a good science talk. There’s nothing quite like science; even when it comes to something as wishy-washy as consciousness, it can still give you surprising answers.

And video!

The short version of the slides:

When you see, you see what’s already been processed and filtered. Illusions are when the system doesn’t work; you don’t see when it does work. In other words, we see “car accident” as presented to our consciousness — we don’t consciously put it together from our visual input. I’ll spare you the customary link to the “You wouldn’t know if a Gorilla showed up” study, but it’s fairly clear the brain only passes on the Cliff Notes version to the executive layer.

Consciousness lags well behind. When scientists measure the movement of a finger, the electrical potential rises a full second before the finger moves. But we report making the decision to move half a second before the finger moves (Libet). We become aware of making the decision after it’s already happened.

Some scientists conjecture that consciousness may simply be unnecessary (Rosenthal). Others think that consciousness may be a result of conflicting subconscious systems (Morsella, Hofstader, Metzinger), and the Watts commentary points out that consciousness seems to be strongly associated with inner conflict and/or pain, although I’m not spoiling his punchline.

Despite what Heph says, I don’t think the talk is depressing. When you think about consciousness, you assume that it’s a good thing, but realistically we’re far happier and productive in flow, without that nagging voice inside our heads. Rather than life being suffering, suffering itself is the act of consciousness.

The talk itself went down well, with the coveted seal of approval. The 5MoF itself was surprisingly wide ranging — Eclair Bandersnatch showed up in a barbie mask and wig to talk about art, Danny O’Brien gave a talk on The Cosmopolitan Anarchist and recapped the news on Byron Sonne, Liz Henry read poetry from her new book, and Josh Juran presented FORGE, a GUI based on manipulating what appeared to be symbolic files on the filesystem — a programming paradigm that apparently came from Plan 9 and hurts my brain every time I think about it.

We’re doing the same thing next month, and I’ll probably be talking about Transcranial Direct Current Stimulation (if I’m not, y’know, drooling in a corner). So! If you have a thought that’s been burning a hole in some mental sidepocket, you should sign up.

Systemantics

wsargent@tersesystems.com (Will Sargent) — Sat, 17 Mar 2012 12:29:00 -0700

New Five Minutes of Fame presentation. This one’s a presentation about a little known book called Systemantics (a.k.a. The Systems Bible).

Systemantics

This is a hard book to get hold of, but a worthwhile one. Systemantics doesn’t make a lot of sense without the context of Systems Theory, which is responsible for the word “cybernetics” and a whole bunch else, mostly talking about systems in the context of the complex feedback loop of a nuclear power plant.

Systemantics is a little bit different: it talks about the feedback loop involved in organizations, and how the system has an independent life (and will to live) outside of any of its participants. It’s a book about how systems actually behave, and how what an observer may consider to be a bug looks like appropriate behavior to the system. It’s about the system as you know it at 2 am, the system complete unto itself in all its ineffable complexity.

That being said, much of it is applicable to complex computer systems as well — in fact I’d say that Systems Theory is far more applicable to my day job than most CS Theory is, and if there’s ever going to be a Software Engineering curriculum then I’d want it to include this book.

Interviews without Puzzles

wsargent@tersesystems.com (Will Sargent) — Wed, 22 Feb 2012 19:12:00 -0800

Technical interviews have their own particular lore, and their own history. Over the years, there are some interview practices that have sunk into the group subconscious of engineers, to the point where they’re used so commonly we don’t even question why. Puzzles, for example.

There are a few famous interview puzzles out there. Microsoft has “Why are Manhole Covers Round?” Google has “You are shrunk to the height of a nickel and your mass is proportionally reduced so as to maintain your original density. You are then thrown into an empty glass blender. The blades will start moving in 60 seconds. What do you do?”

The standard answer to the first puzzle is “So they don’t fall in.” The answer to the second is that assuming you have the same proportionate strength, you can jump out. (There’s an inverse square root effect between muscle and body length, so density is not a factor.)

But here’s the thing. The first answer is incorrect. Manhole covers are round because they can be round. They could be square or triangular just as easily.

So as an interviewer, if you pick random interview puzzles out of a book and you think you know the “right answer” to the puzzle, you run the chance of not hiring someone because the answer he gave was actually the correct one.

A more common problem is that a clever interview puzzle is usually well-known. As soon as you figure it out, you’ll tell all your friends, and they tell their friends. Eventually, you can google for the answer.

Back in the day, Zen Schools had the same problem. They were looking for insight and flashes of realization, initially, and wanted a way to test for this. Some people thought up excellent questions that could test the subtle understanding of self, reality and perception required of Zen students. More and more, the koans were used as the standard by which students’ understanding could be measured. Eventually, someone had the bright idea to put together a book of koans — complete with “acceptable responses” — and through years of formalization, the age old question “What is the sound of one hand clapping” turned into a meaningless ritual.

Even if you don’t use well known puzzles, interviewing with logic puzzles in the long run optimizes for them. Through discussion, shared experiences and research, people will generally know that they should study for a general class of logic puzzle. And the company will start getting more people that will do well at those puzzles… but that doesn’t mean they know programming any better.

Fermi questions, those “How many elevators are in New York City?” questions that are popular in interviews, have a clearer structural weakness. They have no right answer at all, and can be completely circumvented with the right training. Once you know the rules, you can come up with completely the wrong answer and still be “correct” according to the law of the game.

I also have a philosophical problem with Fermi questions. Yes, they’re pointless, but it’s not just that they’re pointless. Asking a Fermi question says that you don’t really care what the answer is. Asking a Fermi question tells your candidate that you want them to guess.

Engineers are trained out of guessing. Engineers are trained to nail down as much as they can before solving any problem, because invalid assumptions and requirements are dangerous and expensive. But that’s not why engineers don’t like to guess.

Why is because most engineers remember vividly what happened the last time someone came up to them and said “When do you think we can go live? Just make a guess.” When it comes to Fermi questions, a cagey and hesitant engineer isn’t a bad candidate, but an experienced one.

So what do I recommend? This book is good to get a broader sense of what interviews are supposed to do, and you can ask spot questions about language and system knowledge, then rate them on the Programmer Competency Matrix. But the best way to figure out how someone attacks code is to bring out some buggy code and ask for help debugging it, or bring out a design and talk about how you’d implement it, talk about the engineer’s background, print out the engineer’s github project and ask about it. They’ll appreciate it, and so will you.

Failing with Passwords

wsargent@tersesystems.com (Will Sargent) — Fri, 17 Feb 2012 11:08:00 -0800

Did a talk about implementing password security right last night at Five Minutes of Fame.

If you don’t want to go through the slides, here’s the TL;DR version:

TL;DR User Security

Use a password manager like LastPass or 1Password (with Dropbox) and use their password generation.
If no manager available (routers, OS logins, etc), use pass phrases with non-English words or acronyms (see xkcd)
Assume sites get compromised all the time and you never hear about it. NEVER reuse a password.
If you’re at a coffee shop or hackerspace, use a public VPN service.
OAuth / Twitter / Facebook based authentication is putting your auth credentials in their hands.

TL;DR Encryption Security

Use bcrypt. Bounce up the factor every few years.
Do not limit password field length. (bcrypt takes up to 55 bytes of input.)
Run a JS password tester to reject weak passwords.
Run a password cracker regularly to test your security.
Suggest to your users that they use passphrases with acronyms, punctuation or LOLspeak.
Generate random passwords for your users.
Consider removing password masking.

TL;DR Operational Security

Use HTTPS for both rendering and submitting login page.
Show Cain and Abel video to everyone you work with.
Use HSTS headers with HTTPS.
Use Synchronizer Token to prevent CSRF attacks (or use a decent web framework).
Use a captcha / throttle on password attempts.
Use double validation for registering accounts (register sends email, clicking email link heads back to site).
Use one time use password reset links.
Send email notifications on password change attempts.

Extra Credit

Add Honeypot Logins.
Use login token IDs with hidden check bits and math invariants that indicate tampering.
Implement a secret in the session management system to keep state on the client and verify it on server interaction for better session authentication.

OWASP also has cheat sheets which look useful if you’re putting a site together. It still disturbs me how freaking MANUAL so much of this is, but I suppose web frameworks can’t do everything for you. There are some options if you’re on Rails.

It was a surprisingly tough talk to give. At first I was like, “lol, look at all the companies with crappy security”, but it’s a murky field in general. For example, the XKCD cartoon about passphrases is missing the problem that most people type passphrases in standard English, and only use about two thousand words in general conversation. It may look like there’s more entropy generated, but if your attackers know that your customers use passphrases, you may have just made their jobs much easier.

Also, brute force cracking is surprisingly effective. MD5 and the SHA-* algorithms are inappropriate because GPUs chew through them very quickly, but the newer FPGA chips can do a reasonable implementation of bcrypt in hardware. It’s an issue that computers are fast, but a bigger problem is that they just keep getting faster.

The biggest thing has to be to not let your users pick crappy passwords. Even if you have bcrypt with all the factors, if your users are entering “12345” as the password, it’s not going to make a difference.

Heuristics in Mate Search

wsargent@tersesystems.com (Will Sargent) — Tue, 17 Jan 2012 15:00:00 -0800

Five Minutes of Fame talk about dating heuristics. This went much better than expected because the pictures and subject matter helped balance out the math.

Although there were a number of people afterwards who were like “too unrealistic” and I was like “yeah, this works better for interviews and college placement but whatchagonnado.”

Five Minutes of Web Frameworks

wsargent@tersesystems.com (Will Sargent) — Fri, 18 Nov 2011 15:00:00 -0800

New 5MOF presentation, wherein I talk about why web applications are complicated. In five minutes.

I really need to write this up as an essay, as I think presentation objects vs domain objects are a much bigger detail than we realize.

How We Make Decisions

wsargent@tersesystems.com (Will Sargent) — Fri, 21 Oct 2011 14:04:00 -0700

Five Minutes of Fame presentation on how we make decisions. This one was a lot more dry and technical, but it was a nice change of pace after bronies melted my brain.

Part of what makes this so fascinating to me is that you can actually see the algorithm that tells us “hey, we should do more of this.” That’s a huge step in knowing our blind spots.

Happiness Lecture at Noisebridge

wsargent@tersesystems.com (Will Sargent) — Fri, 15 Jul 2011 15:11:00 -0700

Ty put up a video of me presenting Happiness @ Noisebridge.

It has kittens.

Updated: Now with slides!

The Core of Agile

wsargent@tersesystems.com (Will Sargent) — Sun, 12 Jun 2011 00:00:00 -0700

I've been thinking lately about Agile. Again.

The first thing I've been thinking about is the people who say "You're doing Agile Wrong."

There's always been a dichotomy for me between the theory of Agile, and the practice. It's a common problem with any dream; it's always cleaner, brighter, simpler, better than the reality. Reality is messy. It is imprecise. It is never seen directly, always filtered through recollections to make each participant the protagonist of their own play.

If you try pair programming, then you're going to find that "you should never pair program 100% of the time." Or that "you should only pair program between people with equal skill sets." Or that "you should practice pair programming ping pong". There will always be a special case. There will always be something that works for you that doesn't work for someone else. There will always be something that doesn't work for you that works for someone else.

Not only do we do Agile "wrong", but we will always do Agile "wrong". We won't ever do anything "right" – we will do imperfect jobs, come home to imperfect relationships, have imperfect children and live imperfect lives. This is what happens when you measure yourself against an ideal.

But why believe in Agile then, if the only way you can do it is wrong? Something that came to mind about that statement.

If you're trying to do Agile, and it's not working for you… then you're doing it wrong.

Another way of phrasing that statement is that Agile is Doing It Right.

In fact, almost by definition, Agile is Doing It Right.

"Agile teams produce a continuous stream of value, at a sustainable pace, while adapting to the changing needs of business." – Elizabeth Hendrickson.

"Agile development uses feedback to make constant adjustments in a highly collaborative environment." – Practices of an Agile Developer.

"Agile has no definition. […] There's no standards board, there's no test, there's no approved workbook, there's no checklist. […] It's based on three things: 1) principles not practices, 2) attention to people, and 3) always be adapting." – Daniel Markham.

Three definitions of Agile. Nothing about practices, or even methodology. What they agree on is a feedback cycle that can respond to changing input and produce useful output.

It's Dorner's model of problem solving. Or Deming's PDCA cycle. Or the Military's OODA cycle. Or the Scientific Method. Or Kaizen. It's continous process improvement, in all its forms.

If you're following a "best practice" and that "best practice" isn't working for you, then it's not a case of "You're Doing Agile Wrong." You're doing something that isn't providing a benefit for you. By following that "best practice", you're not doing Agile at all. Agile is the ability to plan something new, throw out something old, and challenge preconceived beliefs.

That's the core of Agile for me: it's not about Wrong or Right. It's the idea of saying "We can do better." And then doing it.

The Logic of Failure

wsargent@tersesystems.com (Will Sargent) — Fri, 10 Jun 2011 00:00:00 -0700

Another talk, this time on The Logic of Failure: Recognizing and Avoiding Error in Complex Situations, a book by Dietrich Dorner.

The book’s been a favorite of mine for years, not just for the set up, but for the detailed, unsparing look it provides on how human beings fail to get things right. Too often in psychology, there’s an emphasis on either seeing how people feel about a situation, or how well or how poorly they perform at a given task. Dorner goes further, and tries to understand not just how, but why they fail.

The Slides

The Setup

The setup was simple. Dorner set up a computer simulation of an African village called Tanaland. This book was written in 1990, and so Sim City was not widely known, but it’s the same concept. The players were given dictatorial powers, given the goal to “improve the wellbeing of the people” and had six opportunities over 10 years to review (and possibly change) their policies.

The Experiment

Given the tools the players had at hand, they went to improving what they could. They improved the food supply (using artifical fertilizer) and increased medical care. There were more children and fewer deaths, and lif expectancy was higher. For the first three sessions, everything went well. But unknown to the players, they’d set up an unsustainable situation.

Famine typically broke out in the 88th month. The agarian population dropped dramatically, below what they had been initially. Sheep, goats and cows died off in their herds, and the land was left barren by the end. Given a free hand, most players engineered a wasteland.

One player, by the end of the simulation, had a stable population and had significantly better quality of life for the villagers. Failure was the rule, but somehow he had found an exception.

The Breakdown

The litany of possible errors was a long one, and so immediately recognisable that it's hard to suppress a wince of empathy on reading.

The players who did badly tended not to ask "why" things happened. They tended to jump from one subject to another, switching aimlessly, without focus. They proposed hypotheses without testing them. If they did test their hypotheses, they did so on an adhoc basis, testing success cases without testing possible failure cases. In some cases they had tunnel vision: focussing on irrelevancies at the expence of the larger picture. In other cases, they attempted to "delegate" intractable problems to the villagers themselves or refused to deal with the issue at all. Finally, and most tellingly, most players dealt with the problems that they saw "on the spot" without thinking of the larger, longer term problems that they were setting up with that immediate short term solution.

These results were not a surprise. They were just what Dorner's team was looking for. Where many scientists would have looked at the successes and determined the optimal "working strategy" – Dorner was just as interested in the range of failures in the experiment. Dorner's team had specifically designed the simulation so that most people would fail at it, precisely aiming at the weak points of human decision making.

Not all players failed in the same way. Even amongst the players who failed the same way, many players had different reasons for their particular mode of failure. And yet, there were strong commonalities among the failing players, both in their reactions to incipient failure, and in their attempts at recovery.

The Reasons

The reason why most people failed was that they did not understand the nature of Tanaland. Despite being a simulation, Tanaland was no game, and Dorner's team programmed in as accurate a simulation of an African village as the hardware would allow. The watertable under the village had a limited amount of water available. The population grew at an exponential rate given the available food and healthcare. Even the topsoil was modelled accurately, so that overgrazing caused by massive herds would erode the topsoil over time. All of this data was available to the players – had they thought to look. But most players didn't. The experiment ended in three predictable failure modes: either the cattle starved and died, or the groundwater was exhausted, or the population exceeded the available food. Far from being a bundle of independent subsystems, all of Tanaland was deeply intertwingled.

The Weaknesses

The deeper reason why Tanaland was so successful at bamboozling players is partly due to the incredible success of the human brain's pattern recognition system. Human beings are capable of driving in heavy traffic, understanding language, and recognizing patterns in almost random data, feats far beyond most computers. But there are some problems which defeat human intuition.

Linear extrapolation. Human beings have a tendency to assume change itself is static. Even when shown exponential growth, we're not good at internalizing that knowledge. This may be why calculus is so hard for many people, because we don't think about the rate of growth itself growing.

Delayed Feedback. Human beings tend to assume that an action will yield a response immediately, or not at all. This is the way that we interact with the world on a daily basis, and we can become very confused when there's a significant delay in the system's response. Even when we recognize intellectually that a change is "in the pipeline", we may struggle against the instinct to do more and oversteer rather than correctly "sitting on our hands."

Contradicting goals. Part of the failure of players was inherent in the vague goals that they had. What, exactly does "improve the wellbeing of the people" really mean? Does it mean providing the best quality of life to all the villagers? Growing the village as a whole to be more prosperous? In many cases, the top level goal ended up being broken down into goals that conflicted with each other. In other cases, players tried to find concrete problems to fix. One player, deciding that the village needed irrigation, set out building an irrigation system and quickly became fixated on that one problem, becoming "addicted" to his experience of flow. In such cases, players were unable to clearly form goals at all.

Priorities. Even when the players had clearly defined goals, they had another problem to contend with: they would be stymied by cases where actions which furthered one goal would thwart another. The complex interdependencies in the system did not allow for a full optimization of every variable, and players would either flail uselessly or be paralysed by their inability to cover every base.

Information overload. In many cases, the players used too little information to know how to make the best decisions. However, some players had the opposite problem; given access to all data of a complex system, they tried to see the entire system at once. These players found themselves paralysed by complexity, and unable to interpret the results of the data. Interestingly, the problem the players had was not that they did not see the correct chart. They were literally unable to recognize the charts and the changes in data as relevant – having looked at all the data available, their abilities to see a pattern was exhausted well before they stumbled on the correct chart.

Reductive Hypotheses. By far the worst problem that players had, above all others, was that the first hypotheses they formed about the system were not changed in response to the data. If anything, the players were apt to be the most sure in their beliefs when the hypotheses were completely wrong. Part of this came from uncertainty and cognitive dissonance. Uncertainty produced fear and doubt. Asserting the hypothesis helped quell this fear and doubt. Over time, the players learned that the more they believed in the hypothesis, the better they felt. This reduced their ability to develop new hypotheses, as they were already "wedded" to their existing ideas.

The Successes

There were also commonalities amongst the players who did well. The players who did well were the ones who could tolerate uncertainty. They defined clear goals and priorities. They made many small decisions in different areas, and followed up on the expected vs actual results of most, if not all of those decisions. They kept an eye on the overall processes of the system, and did not succumb to flow experiences.

Adding to this, Dorner's team ran an experiment with two groups of fifteen players each. One group was drawn from the student population. The other group was made of senior managers from large industrial and commercial firms. The managers did significantly better than the students on every possible metric; given several different challenges, they responded appropriately to each one. Dorner's team was unable to determine if this was innate talent or the benefit of years of experience.

A Decision Making Model

So what is the right thing to do when faced with a complex situation? Dorner presents a possible schema for problem solving, intended more as a helpful aid than as a representation of how people actually solve problems.

Formulation of Goals - deciding what it is that needed fixing, and putting priorities on those goals.
Formulation of Models - determining the internal workings of the system.
Prediction and Extrapolation - determining the eventual output of the system.
Planning of Actions; decision making, and execution of actions - feeding input into the system.
Review of effects of actions and revision of strategy - determining the expected model vs the actual model.

Interestingly, this is very close to the Plan/Do/Check/Act cycle proposed by Deming – it assumes incomplete knowledge of a complex system and tries to improve understanding of the underlying model through repeated iterations of the cycle.

Dorner also notes that being simply told of a decision making strategy did no good at all to players; when given instruction on dealing with complex systems, the players thought that they had been helped and were better able to discuss their failures with better terminology… but their actual performance was the same as the control group. What really helped players, overall, was repeated exposure to complex systems. Showing was not enough; they had to experience their own reactions and build up their tolerance to decision making in the face of uncertainty and emotional stress.

Life in Fifty Years

wsargent@tersesystems.com (Will Sargent) — Sat, 23 Apr 2011 00:00:00 -0700

Another presentation at Five Minutes of Fame. Christina took video of the talk, so I might be able to put that up as well.

This one took me a while to go through. I picked up a number of books on the subject when Borders shut down, and the talk gave me the impetus to crank through them. The most relevant books were Plan B 4.0, The Ecotechnic Future, and Hot: Living Through The Next Fifty Years. I also recommend The Windup Girl or World Made By Hand for a much better sense of what the future may feel like.

Part of the reason I went through the books was to see how likely the Singularity is. As far as I can make out, it's very dependent on how much free energy is available in fifty years. With a strong global economy, all the parts and dependencies sorted and the sheer power requirements it would take to jam a human-equivalent neural network down through silicon pathways, it's technically possible to have AI. But then you've still got the supply chain management to work through before you can develop more advanced AI, and the overall "cost friction" means that, in practice, AI is only going to be as intelligent as is economically reasonable. Fundamentally, silicon AI is a luxury for the rich. As such, it doesn't show up much in the slides, but I didn't have time to do it justice. (This doesn't go into the biological AI depicted in Starfish, but that's another talk for another time.)

The reaction from Noisebridge wasn't quite what I'd hoped with this: most people found it really depressing. I was a bit surprised, because I had made a concerted effort to scale down some of the more apocalyptic predictions and pointed out that the US makes out way better than most other countries (pretty simple reason: the countries that get hit the worst are the poorest).

Still, it was worth it to do the research, and I got a couple of compliments and a fun conversation afterwards. The future is a large and complicated subject, so I'm going to be going back to the presentation and filling out bits as I find out more.

EDIT: The videos of Hot: The Next 50 Years and A Really Inconvenient Truth are worth watching as well.

Responses to “Where Pair Programming Failed For Me”

wsargent@tersesystems.com (Will Sargent) — Thu, 30 Dec 2010 00:00:00 -0800

I'd like to thank everyone for reading the blog post. It got far more attention than I was anticipating, and while I'm still processing everything, I'm touched by how much people "got" what I was saying and responded positively to it. There are a number of people out there that have had the same experience (or close to it) that I had, and while it's not exactly fun to admit vulnerability, I'm happy I did it just to hear from you guys.

So, on with the show. There were a number of responses either in blog comments, or on Twitter (holy god Twitter exploded, thanks @buzz and @tenderlove), or on Hacker News and Reddit. I thought about responding to comments individually, but it makes more sense to write it up on the blog itself. In no particular order:

“You weren’t really doing pair programming”

This is an easy statement to make, and a lazy one. There were a number of engineers there, all of them very competent. Pivotal Labs had gone through the space on at least two occasions. They had several Pivotal Labs alums on staff, and the bulk of the programming library was Agile and XP books. They all believed this was pair programming. If that wasn’t pair programming, I don’t know what is.

I anticipate the response to be "well, it wasn't done the way they say to do it in the books." Yes. That's because nothing is done the way it's done in the books. Ever.

EDIT: And as far as I can tell, they were working directly from the book in that "all significant development is done in pairs" and "If you prefer to work alone, that's fine. You just can't work with us." People who are saying pairing is a part time or optional activity – are they also doing it wrong, or just tailoring it to their team? Different practices are agreed on for different teams, and this was the practice they chose.

“You walked into a situation where you didn’t know any of the tools. What were you thinking?”

I was thinking that pair programming would be an excellent way to be mentored on the appropriate use of those tools. I was upfront about my lack of experience with those tools and front end programming in general, and was looking forward to learning about it. Bear in mind I wasn’t going in cold: I know Ruby very well and knew at least the principles of jQuery, while knowing enough CSS and Javascript to get by. But you’re correct: it was a large part of the problem initially.

Something I touched on briefly in “No high notes”: the best way for me to get the feel of a framework is to sit down and work with it. I’ll read through the manual, build a couple of example applications and push the limits of what it can do. I’ll read through the source code, sign up to the mailing lists, find a couple of embarrassing bugs, and I’ll know the shape of it in my head. And that knowledge is solid: I can tell you some fine details of a framework even if I haven’t touched it in five years.

I couldn’t do that when pair programming. Lectures and presentations tell me nothing about what the first hand experience is: it’s like thinking you know how to ride a bicycle because you saw it done on Youtube. Time after time, my partner would happily dash out a complex jQuery statement pulling out an element by its div and class attributes, and I’d then struggle to do something similar while I was still figuring out the syntax. Every day, I’d get more and more of an overall larger picture of the system, but it was through the corner of my eye as we were whizzing past to our destination.

“You should have spoken up more.”

That's a fair question. Why didn’t I speak up more? I asked co-workers about my general experience and raised some concerns, but I didn’t push nearly as hard as I could have.

The best answer I could give is that I expected it to be hard at first, and then to get progressively easier. At the point where it wasn’t getting easier I was concerned, but I wasn’t going to tell them they were Doing It Wrong. Remember that I was coming into pair programming with a very different background, and was there to learn. I did speak up when it came to the design and refactoring issues I saw, but when it came to XP I had no basis for comparison.

I did tell some of my partners to please stop grabbing the keyboard and suggested that I could tackle problems that didn’t require a pair, but XP is fairly clear that pair programming is the rule, not the exception. XP is also clear that this way of working makes programmers happier and more productive. Ultimately, I wanted to believe it. I told myself my own experience was wrong and that the hundreds of books, blogs and comments were right. I kept quiet.

“You knew pair programming was what they did and you had an interview where you pair programmed: how did you not expect this?”

Also a fair question. The truth is that I didn’t know what to expect, and nothing that I read in the literature lead me to expect that I would dislike it as much as I did. The initial interview was, by necessity, done in a couple of hours with a single person in another room. The following day of pair programming had to be cut short due to a misunderstanding with the recruiter: we’d only scheduled an hour. The problem that was supposed to have taken a day was mostly fixed inside of that hour, and we all enjoyed the experience.

“Have you considered autism / you must suffer from social anxiety.”

No, I suffer from introversion. (Although, some might say I suffer from extroverts.) Put simply, interaction with people tires me and drains me of energy. Being in a quiet room with one person is fine. Keeping an active conversation with background noise or a television is a noticeable drain. Keeping a conversation up with several people at a party where loud background conversations are happening is a larger drain. And about thirty minutes in IKEA is enough to turn me into a zombie.

I can certainly behave in autistic fashion when I’m low energy, but it’s not my default state. And social anxiety is something that everyone is prone to, especially when things don’t seem to “work” but it’s unclear exactly what to fix.

“Pride in ownership is a bad thing”

“Ownership” wasn’t the best term. Pride in good craftsmanship would have been better. I have absolutely no problem with people refactoring code – just because I wrote something and it worked for a particular problem at a particular place and time doesn’t mean it can’t be made better. Seeing how people looked at my code and did one better is actually one of the perks of the job – the day I stop learning is the day I hang my hat up as a programmer.

“It was a dysfunctional / deathmarch environment.”

Actually, no. While it was a dysfunctional environment for me personally, most of the people there certainly seemed to be very happy. It was about as far away from a death march as you can imagine. Bear in mind that there were 10 programmers on the team, and we switched daily. It’s not like I had one bad partner: it didn’t gel with any of them.

They were doing a UI redesign of the entire front end, and had a hard deadline based on the engineers they had at the time. They had evidently spent much time determining the tools and wireframes involved in the redesign. For many of the engineers there, this was a long overdue opportunity to sit down and really crank. Some of them were amazingly intelligent and ambitious people, as good as any I’ve worked with. Pairing with me must have been a frustrating experience, especially for the younger developers who were not used to being mentors.

For my part, I was frustrated by my own lack of skill working on the front end, and was keenly aware that I was slowing people down every time I asked a question. Rightly or wrongly, we had a job to do. No matter how patient and giving they were, ultimately we were being tracked on our points delivered.

And… saying that there was something wrong with the environment is a lazy response. It’s the moral equivalent of having a user file a bug, assuming user error, and then closing the bug as WORKSFORME without actually looking at the code.

Bear in mind I am not saying that their approach to development was flawed in principle or even in practice – I was but one engineer out of many and they had to consider the entire team. Bear in mind that this company followed XP down to a very fine detail and prided itself on its software process. I joined this company specifically because I wanted to work in a company that did Agile. I would have left immediately if it had been obviously dysfunctional or a deathmarch. It worked for them – it didn’t work for me.

“Why were they pairing if they had a fixed deadline and needed to crank out work?”

Good question. Given the context and the relatively straightforward work that they were doing, I believe that the project would have been completed faster if they had not been pairing for the duration. It may have been slightly lower quality, but it could have worked. It may have been that they judged the benefit of always pairing to be more valuable that any short term gain.

But honestly, my guess is that the question never occurred to them. Pair programming is a core practice of XP. If you’re an XP shop, that’s what you do.

“Pair programming can produce amazing work”

This is something I don’t dispute. Almost everyone I know, myself included, has done pair programming at some point and had positive results. Even Dijkstra pair programmed and found some use in it.

However, that’s very different from saying that long term pair programming is flat out better, which is what XP says. Most studies find little appreciable difference between the work that a pair does, and the same work done individually. Anecdotal evidence is not reliable in this field because humans are subject to all manner of cognitive bias; Hacknot (an under-appreciated website that is now an ebook) does a wonderful dissection in Anecdotal Evidence And Fairy Taies. I’m not saying it’s a worse practice in general. But a meta analysis shows that there is no statistical evidence that this is a better practice overall.

Leaving that aside, there’s the small problem that programmers are not fungible. The best programmers are up to 28 times better than the worst programmers. Even if pair programming were proven to help the average programmer, it won't work for some edge cases.

Tying a good programmer to Guy Steele or Richard Stallman will make the good programmer slightly better, but will drastically impede Steele or Stallman. Even tying Steele and Stallman together (or even better, Stallman and Gosling – they're both Emacs fans, right?) wouldn't produce the output of a Voltron-like superprogrammer (EDIT: I am wrong. They did, and it did.). And while I know some people who would love to be paired with Zed Shaw, Jamie Zawinski or Hani Suleiman, I’m fairly certain there are others who wouldn’t survive the experience.

EDIT: “Pair Programming done “right” works.”

This is true, as far as it goes. Pair programming is excellent at covering gaps and edge cases, and it makes doing test driven development much easier as your pair will write tests to verify your code, and vice versa. For stupid “immediately obvious to the observer” bugs, it’s great.

But there’s a problem. You can’t stop to reflect. Without that, you are unlikely to see the larger picture of how the system will work in a larger context, or any more subtle bugs under the hood. Even with an understanding of how the system may work in production, things like transaction management or concurrency bugs may slip past you.

But that’s just my opinion. Again, I’m not saying it doesn’t work, just that it doesn’t work for me.

The Point

This is my point: One size does not fit all. Individuals and Interactions over processes and tools. And if you’re considering implementing a process, think about your team and consider what works for them. As Elisabeth says, introverts need attention too.

Where Pair Programming Fails For Me

wsargent@tersesystems.com (Will Sargent) — Wed, 29 Dec 2010 00:00:00 -0800

It’s been almost a year, but I think I have enough perspective now to write about my experience pair programming.

The short story is that pair programming doesn’t work for me as the main way of developing software. I can pair program for a day, or maybe a week, especially if we’re focused on a particular problem. But after that? I’m done. Toast. I don’t want to see anyone, talk to anyone, and I need at least a couple of days in a cave until I’m fit for human company again.

It’s a sad story, but the funny thing is that I’m so much happier now with how it ended. I’m happily employed on a contract where I work from home or from a coffee shop, and I’ve made new friends and explored more of San Francisco than I ever thought possible. I have a bicycle and a laptop, and as long as I meet my deadlines and check in code regularly, my time is my own.

I’ll list the big problems I have with pair programming up front and give you the detail and anecdotes later.

Split focus.
No experimentation.
No high notes.
No pride in ownership.
No escape.

Background

My background is as a backend programmer. I’ve done inventory systems, credit card processing and fulfillment, transcoding and digital asset management. In non-agile teams, I was used to being handed the largest, hairiest problem in a project, pulling out the domain and putting something solid together inside the deadline.

I came into the company at the tail end of a long period of discussion where they had decided to rewrite the entire front end UI. They were making heavy use of jQuery (a Javascript AJAX library), SASS (a CSS variant) and a library called Erector, which produced HTML markup from Ruby classes. I knew none of these things; I knew some Javascript and CSS only from implementing them on my blog. Because they were focusing on the UI and they were on a tight deadline, they were going to do an eight week sprint, and rewrite as much of the UI as they possibly code in that timeframe.

So, my first day.

I was watching a man looking at a page – what the finished product should look like. He was telling me they’d discussed what the features should be already, and we just had to bang this out inside a day.

He had opened up a text editor, and we were looking at Ruby code. The ruby class was actually a page composed of Erector methods – ruby code that evaluated to HTML – but it had been refactored into extremely small methods which called each other, so there was no way to ‘see’ the HTML and how that corresponded to the class – you simply had to know it. In addition, they had tests attached to each section of UI code. If you refactored the UI, you had to rewrite all the tests.

Sitting with another man, trying to work together to produce working code. I was shocked at how unnatural and unintuitive it felt. Far from being an easy practice, it felt both clumsy and shockingly intimate; both too personal and impersonal.

If I asked a question, then work stopped.

If I asked about the background of the code or the design, then work stopped.

If I tried to follow along, then I found that I was trying to guess at what various methods and frameworks did. I was being bounced between the domain classes, the Erector classes, the Javascript and the various tests, and I didn’t have a grasp on any of them.

I understood the HTML code, but even there, I didn’t even know what to say – did I point out the typos? The two methods that were almost exactly the same except for the innermost loop? Was I nitpicking by pointing these things out, or signalling my involvement?

And then, he was done. It was time for me to write some code. I opened up some files, looked around at the methods. Tried to figure out what was going on, and write some exploratory code to see what was happening… And then my pair started typing the method I was trying to write, handing it back for me to test. I found that this would happen if I paused too long, or seemed to be typing the wrong thing – my pair had another keyboard, and he would type over me to try to “finish the sentence” rather than talk to me.

At the end of my first day, I had a massive headache from trying to absorb everything. I was told I’d done very well, and that it was a lot to take in on the first day. I didn’t feel like I’d been very productive, but I hoped that we could settle down into some kind of flow.

Split Focus

Split focus happens when I try to do two things at once. Human beings are not good at multitasking in general, and I’m worse than most. Pair programming is a balancing act between managing code and managing the pair that you’re working with. Go too far in one direction or another, and you both fall off and lose effectiveness until you restore balance again.

I found that in order to pair, I had to act as if I was in a continuous meeting. I had to not just listen to my pair, but appear to be listening; I had to nod in the right places, repeat back what my pair said in active listening fashion. I had to pick the right moment to interject. I tried to model my partner’s mental state in my head so I could see his viewpoint better.

While I was doing this, I was trying to see the code that he was writing, and the design that he was trying to make the code fit. If there was a failing test, I was trying to figure out the test and the test framework at the same time.

And if I was writing code, I found something very interesting: I don’t think in English when I write code. I think in code first, and I can translate written code to English. When I was trying to talk to people when I was writing code, I found that I’d have to write pseudocode and then talk to them about it – and when my pair wanted me to talk about code, he wanted me to stop typing. But if I stopped typing, I couldn’t describe what I was doing. Every time I tried to write code and talk to my partner at the same time, I could feel the lurch between what I could feel – the complex shape of it in my head – and what I was having to say.

So pair programming split my focus not just in one way. It split it in three or four different ways, and kept it split. No wonder I had a headache.

No experimentation

The large part of pair programming was doing things in “The Simplest Possible Way That Could Work.” In practice, this meant doing it the way that it had been done previously, with the least possible amount of change.

The way that the Erector toolkit was used, combined with pair programming and test driven development, ensured that two Ruby programmers were required to change any and all elements of the UI. Tests were written to check for every single element in the UI, and with good reason; because elements of the UI had been abstracted out into subclasses and views, there was no clear way to see what code would do until it had been fully executed. Sometimes the simplest possible thing to do had been to hack one of the classes for a special-case logic. Then more code had been put on top, until a refactoring was just too expensive. As long as all the tests were passing, the code was in a known, “good” state – and we had to keep it that way.

The simplest thing that could possibly work as far as I was concerned was to put together an ERB page. A straight HTML page with some embedded Ruby was a known, easy solution that would have enabled straight web designers to work on the UI without involving the full programming team, or at least have allowed us to quickly compare the mockup with a page. But that ran into another problem; there is no facility in pair programming to change tracks, or try anything different. Spikes only happen at the beginning of an iteration, and this iteration was budgeted at 8 weeks – we were locked into our current way of doing things.

Not that it mattered; there was simply no leverage to explore the framework or try different approaches. Any code changes that weren’t immediately applicable to changing the current page were “going off track”. I couldn’t see the parameters of the current system, or feel for the fragile points if it didn’t match up with what my pair thought was important. Every single moment I was typing, I was being watched.

No high notes

When I got to the point where I understood some of the framework, I saw that there were some bugs inherent in the existing design which would make the system act unpredictably in production. In other situations, the design could improved by eliminating some classes and moving some other classes to have a single responsibility.

To me, this was “refactor mercilessly” and “once and only once.” But to my partner, this was a violation of “do the simplest thing that could possibly work.” It also went against the daily switching of partners. Whatever you start when you pair has to be finished, and checked in, by the end of the day. If you’re only looking at getting the story done by the end of the day, you’re not interested in hypothetical bugs or refactoring – you don’t have time for them and they’re not budgeted for.

The only way I could move discussion along was if I could write a test for it. If I couldn’t write an effective test, or I described it in unfamiliar terms (usually from books or blogs), he wouldn’t hear “solves a problem that is difficult to test but is real.” He would hear “introduces needless complexity and complicates design” and point to the simplest solution that makes sense to him. If I tried writing a test and it took too long, I lost there as well.

Eventually, I realized that many of my partners had never worked outside a test driven development process: they didn’t have the same idea of design or architecture as patterns distinct from code. Trying to argue for encapsulation or adherence to SOLID principles is pointless if your partner has no background, let alone hands on experience, with what you’re saying. This goes double when you’re talking about hard-won domain expertise: the more I knew about a subject, the less I could say.

I started picking my battles.

How you pick your battles is, you plan to lose in the best way possible. You try to leave an opening so that the bug that you know, from past experience, is going to hit, is not going to result in any lost data and can be easily fixed later. It’s not great, but at least you can recover from it.

There are no high notes, no thought out design. There’s only the design that could have tests written for it, in the time you had to write the tests. The classes look reasonable at first glance, and the methods are seemingly bug free. But the cracks are there, if you know where to look.

No pride in ownership

There is no ownership in pair programming. You’re working on a different piece every day, with a different partner. Everyone owns all the code equally. No-one really has responsibility for a single piece of it. Pair programming views this as a strength.

I had no pride in ownership. The code that ended up on the ground every day wasn’t something I felt I had input into. It didn’t feel right to me. Doing a good job matters to me. My idea of a good job: years after I’d moved on from one project, someone I’d never met wrote me a thank you note for taking the time to make my code clear and easy to work with.

The part of my brain that works though code will chew through things. Impede the flow, and that part of my brain starts to itch. Block it, and it starts pounding, chewing away at anything that comes close. Without pride in ownership, there was no meaning to any of the work that I was doing. I couldn’t even identify it as mine.

It was about this time that my physical health started to pack up. Winter set in, and it started raining constantly. I got a cold that got into my lungs. I couldn’t climb the stairs without resting. I couldn’t write code. I couldn’t think. The rain beat against my window. I couldn’t sleep.

No escape

The worst part of pair programming was getting up in the morning and realizing I had to do it all over again. No matter how drained I was, no matter how much I wanted to get away from people: I couldn’t. As an introvert, I yearned to be left alone with a large problem and the freedom to slice it into pieces. But there was no escape.

Even at the best of times, there were people. First, a fifteen minute standup meeting. Then sitting down at a work station with engineers on either side of me, two to a table. Talking. An engineer sitting next to me in my personal space. Every moment I was sitting there watching him, I was trying to think of something to say. Whenever I was typing I was worried that I was too slow, or that I was taking too much time to think, wasn’t saying enough, wasn’t being clear. Was I being anti-social? Talking too much? Too much smalltalk? There was no way to know.

Even when pairing worked, it was the slow, clumsy communication of two people trying to move a large sofa down a staircase. There was no flow. There were no spots where there was no conscious movement between thinking something and having it appear on the screen. I missed it horribly. Even the small discussions: was having “as_currency” method on an ActiveRecord Model the correct thing to do? Or was it something that should belong to the UI, as a helper method? Just how much did I really care?

And then a group lunch, where there was food, small talk and team bonding experiences. I started stealing moments of solitude whenever I could.

I asked my co-workers if they saw what I saw, if I was missing something, anything – I didn’t see how this could work, how people could keep doing this. They said I was doing fine, that it just took time to settle in and adjust. That it was hard for everyone at first.

Eventually, I retreated into myself. Between the blinding headaches, the insomnia, and the pounding, unmet need to write code, I stopped responding to input. I could stare at a screen and not see anything. Someone could talk to me unexpectedly and I wouldn’t hear them. I was fulfilling the rote requirements of my job, but I wasn’t there. I’d used up everything I had just showing up for the day. I started checking my iPhone when my other partner was typing.

Finally – just shy of three months later, and for the first time ever – I was fired for not being a team fit when pair programming.

Not Alone

I wrote this not just to understand it, but also to be able to talk about it. There’s been a presumption that pair programming works for most people and is much easier and faster than programming solo would be. This may or may not be the case, but as a long term practice, pair programming doesn’t work for me. There are many other people that pair programming doesn’t work for either. We matter too.

Mark Wilden had a similar experience at Pivotal Labs. His discussion of design rings very true.
Nick Carroll says that pair programming is not for him. He thinks peer review is more useful.
William Pietri has a list of anti-social practices, many of which I saw and/or practiced.
Software Reality has more to say about the XP approach to pair programming and design. He also has a seperate p ost addressing pair programming specifically.
A CodeRanch discussion also discusses issues with pair programming, including Damon Black’s experience.
More generally, Daniel Markham talks about some gaps in agile as it is practiced and has made an interesting list of the responses.

EDIT: Also, Kathy Sierra writes about her experiences with pair programming in Pair Programming is NOT a choice, complete with pre-filled in responses.

EDIT: Another programmer describes his experience and gets a follow up response here.

EDIT: Another article: Nobody Expects The Agile Imposition.

EDIT: Alex Ruiz discusses his mixed feelings about pair programming.

EDIT: A story from Xebia about Chris, the tester.

EDIT: Jay Fields talks about why he doesn’t pair program any more.

EDIT: Adam Logic discusses his dislike of pair programming and compares it to Design By Committee.

EDIT: Buzz Andersen: ”I’ve never been more sure of my absolute distaste for anything in my professional life.”

Also, it’s important to note that I still believe that Agile development can lead to better business solutions with less wasted effort. Pair programming is part of a single agile process called XP, and there are many other Agile processes such as Crystal Clear which do not mandate pair programming as a daily practice.

EDIT: I’ve read through the responses and replied here.

Best Practices Crypto Algorithms

wsargent@tersesystems.com (Will Sargent) — Fri, 08 Oct 2010 00:00:00 -0700

I’ve always wanted just a nice simple list of what crypto algorithms go where. Finally, from Web Security 2.0:

AES256 in CBC mode for encryption
HMAC-SHA512 for integrity protection
SHA-256 or SHA-512 with salt for hashing
PBKDF2 or bcrypt for passwords, see here for a comparison between the two.
/dev/urandom or RtlGenRandom/CryptGenRandom for random numbers

Simplest Possible Acceptance Test

wsargent@tersesystems.com (Will Sargent) — Tue, 05 Oct 2010 00:00:00 -0700

If you are fed up with walking through the same flow over and over again and want to have some code drive the browser around (what is sometimes called is known as acceptance/system/end-to-end testing) and don't know quite how: this is for you.

I remember when I first wanted to automate my tests. It was painful. Not only did I have to read through lots and lots of documentation, but so much of it was out of date, or badly written, or just incomplete. So let me say up front: I've read through a bunch of different options. The best option right now (October 2010) is Selenium Webdriver.

There are a variety of options for setting up Webdriver. If you want to do it using raw Java, then the 5 minute guide is your best option.

The way I do it normally is to use Capybara and Steak, and tie them together with rspec – I find writing tests easier this way, both because Capybara tries very hard to make the API intuitive, and because Ruby tends to be very concise. What this looks like in practice is:

# spec/acceptance/checkout_spec.rb
require File.dirname(__FILE__) + '/acceptance_helper'

# You may find the following useful:
#
# http://rspec.info/documentation/
# http://jeffkreeftmeijer.com/2010/steak-because-cucumber-is-for-vegetarians/
# http://richardconroy.blogspot.com/2010/08/capybara-reference.html
#
feature "Feature name", %q{
  In order to check out
  As a user
  I want to submit an order
} do

  scenario "Checkout" do
    email = Time.now.to_i.to_s + '@example.com'
    
    visit "/orders"
  
    within("//form[@id='commitOrderForm']") do
      fill_in "login-email", :with => email
      fill_in "login-pass", :with => 'password1'
      fill_in "login-pass-confirm", :with => 'password1'
      
      find("//input[@type='submit']").click
    end
    
    page.should have_content("Your order is complete")
  end

end

There is some stuff in the acceptance_helper.rb you have to worry about:

# spec/acceptance/acceptance_helper.rb
require File.dirname(__FILE__) + "/../spec_helper"

require "steak"
require 'capybara'
require 'capybara/dsl'

Capybara.default_driver = :selenium                # Uses webdriver by default.
Capybara.app_host = 'http://localhost:8080'        # used for relative URLs.
Capybara.run_server = false                        # turn off the internal rack server.
Capybara.default_wait_time = 5

Spec::Runner.configure do |config|
  config.include Capybara 
end

# Put your acceptance spec helpers inside /spec/acceptance/support
Dir["#{File.dirname(__FILE__)}/support/**/*.rb"].each {|f| require f}

And then there's the spec helper itself:

# spec/spec_helper.rb
require 'spec'

# Requires supporting files with custom matchers and macros, etc,
# in ./support/ and its subdirectories.
Dir[File.expand_path(File.join(File.dirname(__FILE__),'support','**','*.rb'))].each {|f| require f}

Spec::Runner.configure do |config|

end

That's pretty much it as far as Ruby code goes. Note that you will need to set up all the gems first. Here's what should be in the Gemfile:

# Gemfile
gem 'rspec'
gem 'steak'
gem 'capybara'

And, well… you'll need to install Ruby. This can be a pain, but it's at least a fairly well known one and there are many guides that will show you the way. Assuming that you're using MacOS (If you are using Linux or another Unix based system, you do not need to install MacPorts and disable the OOTB Ruby):

Disable the OOTB Ruby by moving:

  mv /usr/bin/ruby /usr/bin/ruby.bak
  mv /usr/bin/gem /usr/bin/gem.bak

Install MacPorts and add it to your PATH environment variable. (Note, this may require Xcode Tools and X11 as prerequisites, please see http://guide.macports.org/#installing)

  sudo port selfupdate

Install the MacPorts Ruby:

 sudo port install ruby

Make sure that rubygems is up to date:

  sudo port install rb-rubygems # or download rubygems from rubyforge – they seem to break their update method every odd number of releases
  sudo gem update –system

Install a library that Capybara uses:

  sudo port install libffi

And then finally, we can install bundler, and then the bundle:

  sudo gem install bundler
  cd mytests # assume you have Gemfile here
  bundle install

Then to run the spec:

  spec spec/acceptance/checkout_spec.rb

And finally: I know how frustrating it can be to get everything set up even for the "simplest" solution, so I've tried my best not to leave out any steps here. If you've gone through these steps and they don't work for you, don't stew: send me an email at will.sargent@gmail.com. Send me a stack trace and I'll see if I can help.

Information Radiator with Toodledo

wsargent@tersesystems.com (Will Sargent) — Mon, 04 Oct 2010 00:00:00 -0700

In something that has been fairly two years in the making, I wrote a Toodledo gem for Ruby, bought a USB Betabrite Machine, and finally got the two talking to one another.

Here is what goes into Toodledo: http://twitpic.com/2uh58x

Here is what comes out of the Betabrite: http://twitpic.com/2uh5ft

It’s based on a Thoughtworks project called Radiator (which, in turn, uses some of Aaron Patterson’s code, the inspiration for this): I forked the code and then pounded on it until it did what I wanted. The fork is here: http://github.com/wsargent/radiator and I’ve tried to include as clear step by step instructions as I could. I’ve tried to write it so I can add other plugins as well (twitter, itunes, bugs, build failures) that I can add without changing the core rendering server.

As a management tool, it is surprisingly effective and unobtrusive. If I’m working on something and it scrolls by, it’s a good feeling. If I’m doing something else and it scrolls by, then it’s a context shift rather than an alarm — I can choose to do something else, but I am at least reminded of it. And if I’m in a space where I can easily do it, then it’s a win.

Why you should be using PostgreSQL with Rails

wsargent@tersesystems.com (Will Sargent) — Thu, 30 Sep 2010 00:00:00 -0700

Rails has a problem. The standard way of dealing with scale in Rails is to scale horizontally: add more servers. This is commonly called “shared nothing”, although it’s really more “shared database” – the usual pattern is that no state is kept in individual application servers.

That’s not the case in a stateful application that uses the database directly. As soon as you have several servers running in parallel against the database, you are essentially running concurrently. There are many subtle problems in dealing with concurrency, but it turns out that there are some very nasty bugs that only show up with multiple servers: depending on the libraries you’re using, your collections can get out of sync.

Xavier Shay has done an excellent job of documenting this, so I’ll wait while you go read the following links:

You have some choices when you have concurrency, and you need to maintain state immediately. You can use a central lock server. Or you can use the locking that comes into the database in the form of transactions or optimistic locking. And, in fact, the “quick fixes” for using acts as list or acts as state machine involve wrapping it in a SERIALIZABLE transaction.

So. Why don’t Rails developers use transactions? More broadly, why the aversion to databases in general? Why do Rails developers tend to treat the database as a raw, dumb datastore and shun even simple database constraints?

Frankly, I think the reason is MySQL.

From what I can tell, MySQL got where it was mostly by having better documentation than PostgreSQL when it really mattered, working on Windows with IIS, and better marketing. It came in by default by Perl and PHP websites, and never left. For a long time, MySQL didn’t support transactions, let alone sub selects — and since it was the first database that most developers had been introduced to, they didn’t know it worked any other way.

Now MySQL has transactions. Sort of. Most of the time, they work. But sometimes, they deadlock.

To a developer, it can seem like deadlock can happen for pretty much any reason at all.

A single insert into an InnoDB table can cause a deadlock, because they use row level locks internally.
Foreign key references use row level locks internally.
You can also get silent deadlocks — you’ll just get lock wait timeouts when you have row level locking between an InnoDB table and a table level lock MyISAM table, and it won’t trigger the InnoDB deadlock detection.

The documentation mentions that you should be prepared to reissue a transaction if it fails.

“Always be prepared to re-issue a transaction if it fails due to deadlock. Deadlocks are not dangerous. Just try again.”

This is not something you can do when you’ve submitted a credit card authorization to the payment gateway.

But MySQL doesn’t stop there. It also locks the entire table when you add or remove a column or an index. It does this because, under the hood, it creates a new table with the extra column, and then copies all the data from one table to the other. I have seen this take hours. This gets even worse when you consider that the quick fix solution, creating auxiliary tables and joining between them, will only work up to the point where it gets you into serious trouble. Add this to the naive query optimizer that MySQL uses under the hood, and you’re screwed. (Let’s not even talk about the bugs.)

My theory is that Rails developers treat the database as a raw, dumb datastore with no data integrity because that is all MySQL is good for. Every time they tried to follow “best DBA practice” and do useful with MySQL, they got bit, and the more they started hating touching (or even thinking about) databases. Over time, MySQL has trained developers into avoiding the bits that don’t work effectively. It’s now to the point where it’s controversial to use any database features at all.

TL;DR

Use PostgreSQL. It’s a much better choice if you’re in any kind of startup. Not only does it have better transaction support, it allows you to rollback DDL operations, so you can do complicated schema upgrades atomically. If you’re doing a bunch of database migrations and refactorings, this is really useful. PostgreSQL does have some gotchas (notably, raw COUNT(*) can be slow), but you can always use counter_cache.

If you need to get hold of data really fast, use NoSQL, or plug memcached into CacheMoney. Then start working your way into replication, either using the built in PostgreSQL 9.0 or Tungsten.

Wrap all your controller logic in transaction blocks, or move your ActiveRecord logic into a service / manager class and set up transaction management around that. Either way, transactional behavior should be the default. Use Foreigner to integrate foreign key references into ActiveRecord migrations, and use simple constraints to enforce integrity. You should still use application level validation as you will get better error messages and have finer control, but things like “validates_uniqueness_of” are more trouble than they are worth.

Finally, have Xavier Shay give a demonstration at your company. I’ve never met the man, but his website is right on the money.

What I Believe About Writing Software

wsargent@tersesystems.com (Will Sargent) — Tue, 28 Sep 2010 00:00:00 -0700

So. I left ATG consulting so I could get a better understanding of what different environments are like. I’ve worked at a variety of startups, and written some Ruby, some Scala, even some Flex and Actionscript, and done a fair amount of fixing, writing and rewriting different systems and architectures.

The good news is that all in all, I don’t suck. I’m very much of the “Domain Driven Design” school of development: I think that the domain and the concerns of the business team are a (if not THE) priority and I believe that it’s possible to write code in such a way that makes it hard to write bugs. I’ve tried different styles, and while the style of code I prefer has been charitably described as “verbose and up front,” I think it is clear and leaves little room for ambiguity. If there are bugs, it’s not a mystery where they are – I don’t like metaprogramming and I don’t like using any more tricks than I have to, so I can live with the occasional verbosity. In fact, some of the code is actually better than I remember. It’s a nice feeling to read code and be impressed by the writer… and then realize the writer was you.

It turns out that the biggest problem I have is not a technical problem. It’s that I believe different things than other people. Some people believe different things than me, and are surprised and concerned when they find that I have a different background than them and don’t believe the same things they do.

I’ve been told that some of the things on the list are “not agile” (not XP, technically) and that “you aren’t going to need this.” I don’t believe XP prescribes or proscribes any programming technique: I don’t believe that YouAintGonnaNeedIt is an automatic veto, just as TheSimplestThingThatCouldPossiblyWork isn’t a green light to do what you want: I’ve heard “simplest thing” used as a justification for everything from never using transactions to checking in over a gigabyte of binaries into a project (including both Linux and MacOS binaries of Apache and MySQL) to “simplify” the developer’s build environment.

So. Here’s what I believe.

I believe that loosely coupled, encapsulated systems are the way to go, for many reasons. I believe that they are easier to mock, easier to debug, and easier to use.
I believe that good strong interfaces make for good neighbors.
I believe in separation of concerns. Specifically, I believe in separation of content from presentation.
I believe in the SOLID principles.

I believe that programming languages have their strengths and weaknesses. Arguing which is better is like asking if a hammer is better than a trowel.
I believe that operating systems have their strengths and weaknesses. Given the right toolset, I’m as happy programming in Windows as I am in Linux or MacOS.
I believe that editors have their strengths and weaknesses. On a given day, I’ll use vim, jEdit, Textmate, Eclipse and IDEA to edit files. I’m fine with all of them.
I believe that frameworks have their strengths and weaknesses. J2EE works in its context. Rails works in its context. They both suck outside their context.
I believe that design by contract, program functions and static analysis tools such as ESC/2 are going to be the next wave of programming.
I believe that weak typing is good, because it speeds rapid prototyping and code flexibility.
I believe that strong typing is good, because it limits the number of possible bugs that the programmer can generate and provides a rich abstract syntax tree that can be used by tools.

I believe that simple database constraints are a good thing. Specifically, I believe that adding NOT NULL directly to your tables is a good thing.
I believe that referential integrity can, in some cases, be a good thing. I think that used properly, they reduce the amount of bugs and bad data possible.
I believe that database transactions are a good thing. You may have concurrency bugs when you scale your app if you don’t use them.
I believe that 99% of the time, a decent SQL database with transaction and constraint support (i.e. not MySQL) will serve your needs, and that NoSQL is required only in some exceptional scenarios.

I believe that designing systems using finite state machines and explicit state transition can be a good way to tightly define a system and eliminate bugs.
I believe that validating an object is good, validating a state change is better, and validating a system is best.
I believe in PMD, Checkstyle, and Findbugs. I believe in Flog, Heckle, and Saikuro.
I believe in Release It, in its entirety. I believe in fail fast. I believe in bulkheads. I believe in circuit breakers.

I believe that most of the really nasty bugs come from miscommunication between different parts of a system.
I believe that defensive programming is a good thing, because it flushes out hidden assumptions between the different parts of a system.
I believe that code is not finished until you have thought about how the system responds to invalid input and exceptions.
I believe that, on some level, unit tests, assertions, defensive programming, design by contract and validation logic are all the same thing.
I believe that that a good solid configurable logging framework is a requirement for production code.
I believe that having diagnostic logging statements in production code is a good thing.
I believe that code is not finished until you have thought about how someone else is going to have to debug it.
I believe that comments can be valuable and worthwhile when they augment and not repeat the source code.
I believe that tests are not documentation. Documentation is for a human audience first, and tests have to work first and be readable second.

I believe that some tests are inherently useless, notably when they test the underlying library code and not the system under test.
I believe that any testing done by a programmer, automated or otherwise, is inherently biased in ways that don’t reflect a user.
I believe that automated tests can be useful, but they can only show you the programmer’s intent. Even end to end system tests will only look for specific areas of a page.
I believe that 100% code coverage provides little advantage over 80% code coverage.
I believe that a good integration test suite is better than a great unit test suite.
I believe that there is no substitute for a good QA team.

Finally, I believe that two people pair programming are no more efficient or effective than two people programming alone. The most cited study points to a 15% development time cost, but mentions that the co-workers believed that code was consistently at a higher level of design quality and had fewer defects. This has not been my experience: just looking back at the rate and quality of work produced, I can’t see a substantive improvement over a regular team. To be clear, I don’t think it’s markedly LESS efficient or effective, just that I haven’t seen it take teams to a different level. Which is saddening, but I’m not going to go into it.

Thank you.

Making Avatar Make Sense

wsargent@tersesystems.com (Will Sargent) — Sun, 03 Jan 2010 00:00:00 -0800

Okay, so. I’m going to assume that everyone has now seen Avatar, and hence I’ll assume you’re all up for massive spoilers, etc.

The movie we all just watched was not the movie we thought we watched. And this is not just some George “hahaha, I destroyed the hopes and dreams of a generation” Lucas screwing with the fanbase… this is lurking beneath the surface of Avatar from the beginning. The existence of the Na’vi.

The Na’vi have no common morphology with the rest of the planet. Sig even comments on it in a Youtube article. Why do they exist? How is it that they speak a recognizable language, and have genes close enough to human that it’s possible to MIX IN HUMAN DNA with the Na’vi? How is it that the Na’vi have built in neural interfacing equipment that can instantly domesticate the larger animals and even predators? Wouldn’t evolution make such a thing impossible?

The answer is that the Na’vi aren’t a natural race. Eywa made them. They’re close enough to human that the humans can communicate with them and think they look cute and cuddly (Ewya may have been slightly confused here), and alien enough that they can survive in the local environment. If that wasn’t enough, Eywa provided with some elevated sudo privileges, so they could take advantage of the local fauna without Eywa being directly involved.

The how is easy. Eywa’s a worldmind capable of transferring human neural networks into Na’vi clones on the second try, and it’s entirely feasible to create a race and set it up with false memories. And something that’s smart enough to create room temperature superconductor in bulk (what? You think Eywa runs on plants alone, when there’s massively complex electromagnetic flux happening around the tree and the Na’vi just happening to be sitting on giant deposits of unobtainium?) will have no problem reading our electromagnetic communications. Eywa’s on Alpha Centauri; it’s been listening in since the first radio broadcasts.

The why is a bit harder to explain. Why would a worldmind play dumb?

Well, probably because it has a very good understanding of what happens to things that look like a threat to humanity. If we had any conception of what Eywa is, we’d be terrified, and we’d probably sterilize the entire planet before we even set foot on it. As it is, Eywa looks harmless. It looks beautiful. It’s not exactly friendly, but it’s the kind of environment that keeps humans focused on the trees instead of on the forest. Eywa can afford to watch and wait until it has to act.

It also makes a hell of a way to see human capabilities up close and personal. Eywa is well capable of doing a vacuum cleaner impression on every single EM communication on the planet. And when Eywa saw a human built Na’vi run out the container and disregard orders, it could lay bets that this was someone stupid and romantic enough to provide Eywa with more insight into the military command structure. The cute floaty things aren’t accidental. They are Eywa’s way of saying “hold up, this guy could be useful.”

And Eywa gains massively out of it. Not only does it have a neural imprint of a woman with a massive amount of scientific knowledge, it also got to see a run through of the military. It had to expose itself to a certain extent to get the humans to back off, but you can be certain that the humans will have Eywa bacteria in their digestive tracts when they reach Earth. I wouldn’t be surprised if Eywa had a very specific form of toxoplasmosis all ready to go if needed. (Indeed, one plot treatment specifically mentioned this possibility.) It doesn’t have space travel down, but as long as it stays quiet, it can build up and infiltrate the military before they get to the point they know what they’re dealing with.

And for those romantic souls thinking this was about Jake Sully… dude. THEY LOST. Jake and the Na’vi didn’t stand a chance against the military, they used up most of what they had, and THEN, as soon as it looked like Eywa might actually lose a valuable resource, it acted. It was perfectly happy to use the Na’vi as cannon fodder if it meant it didn’t have to show a card.

And I, for one, welcome our worldmind overlord. Life under Eywa would not be a bad thing for the vast majority of people, and Eywa would make better use of our technology and infrastructure than we could. Eywa keeps the reins loose for the most part, and doesn’t enforce behaviour as much as “convincingly persuade.” That may be the Na’vi, the charming public face presented by the unknowable, near-omniscient power that is Eywa, but even if the Na’vi are rolled up and tossed out like last week’s pizza once we are no longer a threat (which is unlikely — the Na’vi are too damn useful for offworld activity), then humanity, and Earth as a whole, can rest in peace knowing that something much smarter, tougher and biologically engineered to perfection took us out.

What Makes People Happy

wsargent@tersesystems.com (Will Sargent) — Mon, 28 Dec 2009 00:00:00 -0800

If you want to know what makes you happy, you have to be willing to think hard about what happiness is, and pay attention to what makes you happy.

“Happiness comes in small doses, folks. It’s a cigarette butt, or a chocolate chip cookie, or a five second orgasm. You come, you smoke the butt, you eat the cookie, you go to sleep, wake up and go back to fucking work the next morning, THAT’S IT! End of fucking list!” – Denis Leary

So after much time and experience, here’s the list of things that make me happy.

1) Direct sunlight.
2) 8 hours of sleep.
3) Movement outside.
4) Social interaction.
5) Regular meals.
6) Satisfying work.

The upshot from this list is that I’m not all that complicated. Also, what actually makes me happy may not be what I spend most of your time thinking about. I don’t think about sunlight all that much. But I can tell the difference between when I have it and when I don’t. Biking has a huge effect on my mood. Not eating has a huge effect on my mood. And I’m going to take a leap in logic and say that this list is globally applicable to all humans.

The complicated part of this is social interaction and satisfying work. But even the satisfying work is simpler than you might expect.

But hang on a sec. This list doesn’t just apply to humans. Look at dogs, for example.

Dogs need sunlight. Just ask one.
Dogs need sleep. They get cranky if they don’t.
Dogs need walkies.
Dogs need to meet other dogs and sniff each others butts. And humans.
Dogs need regular meals. And they’ll eat anything you give them.
Dogs need to do something. Pointers need to herd, bloodhounds need to sniff.

You break it down and you’ll conclude that human beings are social animals, and have the same needs as social animals. I used to think that the people who get up at 8 am, eat breakfast and then jog for an hour were obnoxiously happy people who were just naturally gifted. They’re not. They’re happy because doing those things will make you a happy human. Likewise, staying up until 3 am, not getting outside, getting crappy sleep and reading existentialist philosophy will make you pretty damn unhappy. It doesn’t matter what your brain thinks about the activities you’re doing — do these things and you’ll look and act like an unhappy person the next day.

Again, this goes back to how to get the most milk out out of cows or the most work out of programmers.

The Dog Hypothesis: Human beings need walkies.

Getting work out of Programmers, Part 2

wsargent@tersesystems.com (Will Sargent) — Mon, 20 Aug 2007 00:00:00 -0700

So here’s how I think you get the most work out of programmers. This is a follow on from part 1.

Morale

Programmers have good morale when they are treated well, and they are given a problem that they know they can solve. Thank programmers whenever they do something. Let them know you not only know how much they work, but that you care. Keep track of morale through weekly one on ones with each and every programmer. Keep track of commitments you make to your programmers (including the verbal ones) and follow through on them. Make it clear that you are looking out for their interests. (Creating a Software Engineering Culture, Chapter 3.)

Money and stock options only have a limited effect on morale, and may have a negative effect (Agile Development, p63). Most often, a gift of money or stock options is a substitute. (Rapid Development, p262). Morale events can be fun, but don’t actually raise morale — they just allow for a different kind of interaction with people. (How to avoid Lame Morale Events).

And there are all kinds of things that can hurt morale. I think the major one is the Broken Window Theory (Pragmatic Programmer, p4). Under the Broken Window Theory, any neglect or rot in a system that is not directly addressed and countered is a drag on morale. People wonder why it is that they have to write good code and do things right, when they’re not allowed to fix the crappy code. Management assertions that “we don’t have time right now” or “we’ll do it later” start to sound empty and hollow as project after project goes by, and the crappy code festers and rots as hack after hack is piled on top of it.

The Psychology of Computer Programming, Chapter 10 deals specifically with Morale and Motivation. Rapid Development, Chapter 11 goes into typical developer motivations. They are both very much worth reading.

Sleep

You can’t make people sleep, and you can’t do much about sleeping arrangements. But you can tell them that you want them to get 8 hours of sleep a night, and you can bring up lack of sleep as an issue. Anyone who looks sleep deprived needs help; either they have been trying to sneak in work late at night, or they’re suffering in other ways. Give them all the help you can. The number of work hours will be an issue there. And if someone loads up on caffeine and junk food late at night… well, I’d point out that there may be a connection (How to Sleep Better).

Exhausted employees are easy to spot. They’re the people who frighten small children and spouses. They are not fit for work. They barely even know they are at work. They should be sent home until they know what they’re doing. Just that simple act of humanity will raise morale.

Focus

The best way to ensure focus is to ensure transparency and feedback. If programmers have a public, physical way to see what needs to be done at a granular level, whether in a todo list on a whiteboard or a series of 3x5 cards on the wall, they can see at a glance what they will be working on not just today, but next week as well. (Agile Development, p97) This works very well for helping out programmers who have a large list, or determining the priorities — you can only have one task on the top of the list, so what you are supposed to do is never in doubt. This is a technique that is used in several agile methodologies, and is known as an information radiator. (Crystal Clear, p32)

Don’t confuse your programmers, or worse, try to multitask them. If you give them two jobs to do at the same time with the same priority, you’re putting them in a situation where they cannot win; no matter what they do, they’ll be working on the wrong thing. And if they try to do them in parallel, they’ll do both jobs more slowly than they would if they did them sequentially. (Joel on Software) (The Multitasking Myth) (Quality Software Management)

Keep the number of goals small in a project. Pick one objective and make it clear that it’s the most important one. (Rapid Development, p257)

But the best thing you can do for programmers is to let them work. Don’t interrupt them. Don’t spring meetings or interviews on them with no warning. Don’t change what they’re working on. Don’t spring last minute high priority projects on them. Don’t file marketing requests to change the font size as priority one critical bugs. Don’t come up and ask when a bug is going to be fixed. Don’t ask them for status updates every five minutes. I’ve seen constant change requests happen at company after company and I can tell you from experience what happens… the programmers roll their eyes, and they stop taking priority changes seriously. Because they know that in an hour’s time, it won’t be a priority any more. (Rapid Development, p259)

If you let your programmers work uninterrupted, something wonderful will happen. There’s a psychological state called flow that has been documented by Mihaly Csikszentmihalyi. In this state, programmers are able to be “in the zone” and become focused to a great extent. Programmers in flow can produce far more code than they would be able to ordinarily. There is a catch though; it takes the average person at least 15 minutes of uninterrupted work to enter flow. If they are interrupted, or even expect to be interrupted, then they’re less lightly to enter flow. (Rapid Development, p506) Some teams implement a policy called “focus time” (Crystal Clear, p33) or the “cone of silence” (Alistair Cockburn) where meetings and interruptions are banned for a portion of the day. Other teams use a red bandanna (Peopleware) or a sign to indicate that they are not to be interrupted, or other methods.

Background

The best way to have programmers with the best background in the problem domain is to cultivate them. I’ve been surprised in the past how little programmers know what the business does. I’ve often thought it would be a useful exercise to have new programmers spend some time with each member of the business team to understand their concerns and priorities. Failing that, it can be a good idea to have documentation (business process management or six sigma documentation) that can bring new programmers up to speed on the organization as a whole. Of course, this only works so far in that it doesn’t track the history of the organization. Legacy code is nettlesome to new programmers, because typically they reflect legacy business processes and legacy business decisions. But ultimately, it comes with time.

Business specific domains, by nature, share little common ground with each other. There are only a couple of books I can recommend here. Domain Driven Design is the single best book I have read about how to effectively model and discuss domains. It is a similar book to Design Patterns, as it not only talks about implementation and common patterns, but it talks about domains as a common language. And Working Effectively with Legacy Code does an excellent job of pointing out useful ways to desnarl and refactor code that is no longer up to snuff.

Experience

Experience comes with work. It doesn’t always come with time. To quote Weinburg, experience is the best teacher, but it doesn’t necessarily teach anything. Experience can be passed on by proxy, through education and mentoring: some of the best experiences I’ve learned from have been the ones other people have had. If you want books that give experience, then Code Complete 2 and Refactoring are the best bets. If you want to read about experience, then Software Craftsmanship and Software Creativity are the best books. But if you want to make gathering experience, then make it available to your programmers. Set aside an education budget. Encourage your employees to attend conferences and seminars. Join a software engineering book club and have programmers pick out reference books for an in-house library. Have regular brown bag sessions and encourage your team to pass techniques around the company. Do this, and you will not only raise the general experience level of your team, but you will raise morale as well. (Rapid Development p257) (Software Craftsmanship, Chapter 19) (Building a Software Engineering Culture, Chapter 4).

Talent

You can hire talented programmers, but I think that’s as much as can be done. I think that talent is not a static quality. I believe talent is a series of mental habits, and that new habits can be learned, just as old habits can be put aside. I believe that some useful habits are solid grasp of systems theory, along with an ability to ask the “right” question. But that’s another essay. And there is another problem: talented people get bored doing things that don’t stretch their capabilities. If you have work that doesn’t require PhDs, you may be better off not hiring them.

Hours Worked

According to decades of economists and management experts: 40 hours. Contrary to popular belief, the standard work week was not invented by the government or the unions. It was pioneered by Henry Ford in 1926. More than 40 hours a week, and the factories didn’t produce as much money; the temporary increase in productivity was more than offset by industrial accidents and mistakes, and after two weeks there was less productivity. (Why Crunch Mode Doesn’t Work) (Can People Really Program 80 Hours a Week?) (When Should You Start Project Overtime?)

This is counter-intuitive, so I’ll say it again: studies prove overtime provides a temporary benefit for a maximum of two weeks, and is worse than useless thereafter. James Shore provides a recent example.

I believe this, not just from the studies, but from my own experience with extended overtime. Programmers will not only do less work, they’ll make mistakes in the work that they do. Then they’ll get irritable and suffer from low morale. Then they’ll burn out completely. I believe (but do not have the studies to prove) that after even after normal work hours are restored, there is a convalescent effect; people will produce less work following the overtime, producing the same amount of work overall. So after the project goes live, they’ll need a long convalescent period before they’re up to snuff, or even worse, they’ll look up from their monitors, take a good hard look at the results of their labor and their (usually meager) rewards, and quit, producing a huge opportunity cost for the company in terms of hiring, maintenance, and reputation. (It’s Not Just Abusive, It’s Stupid)

For every complex problem, there is a solution which is simple, obvious, and wrong. Extended overtime is that solution.

Pressure

Fine; hours worked have no effect. What about directly applied pressure? What happens if we keep the hours, and if we tell the programmers to work harder and produce more work during those 8 hours?

Surprisingly, nothing. Programmers produce work using their brains; the amount of thoughts a programmer can have is fairly constant. Tom DeMarco and Tim Lister researched this and formulated Lister’s Law: People under time pressure don’t think faster. (Slack, p50) They might be more stressed, but that doesn’t help people think faster; stress impedes complex thought, and pushes the brain to a “flight or flight” response. If you’re stressing your programmers, they’ll be at their desks more. But they’re not going to write any more code.

Conclusion

This advice is all simple and straightforward. I don’t see anything in here that comes as a shock, and even my mother said “Well, that’s obvious, isn’t it? It’s like being a farmer and taking care of your cows. If you want your cows producing the most milk, you make sure they’re treated like cows should be treated.”

So treat your programmers well. Keep track of morale through weekly one on ones. Make it clear you care about the health and welfare of your programmers. Make sure programmers know what they should be working on at all times. Don’t change out work that the programmers are doing or abuse the bug tracking system. Keep interruptions to a minimum to allow for flow. Establish a training and education budget, and establish mentoring and brown bag sessions to transfer experience. Keep to 40 hour work weeks, and forgo direct pressure.

Do all of these things, and you’ll get more work out of your programmers. And you’ll probably have programmers beating down your door to work for you.

EDIT: Also see this LinkedIn question that provides some useful advice. Larger monitors have been mentioned in several studies, but I don’t have the references to hand.

Getting work out of Programmers, Part 1

wsargent@tersesystems.com (Will Sargent) — Thu, 16 Aug 2007 00:00:00 -0700

I was recently asked the question: what is the best way to get the most work out of my programmers?

I’ve thought about this for a while. In some ways, it’s the story of my career — every manager I’ve ever had has wrestled with this question. I’ve worked with enough teams and individuals that I think I have a good understanding of what programmers can and can’t do, and how work gets done. And how work doesn’t get done, for that matter. I believe I have an answer, although certainly not the answer.

The first thing to do is to look at the assumptions behind the way the question is phrased. The assumption here is that there are veins of untapped work, hidden inside of programmers somewhere. And the manager’s job is to extract it. I think that this is the wrong place to start from. I believe that most people would rather do a good job than a bad one, and will do the best work they can do given what they have to work with. (Software Creativity 2.0, p22.) The best way to phrase this question would be: “What can I do to help an individual programmer be most productive?”

So, let’s construct a hypothetical employee. What is known to create productivity? Let’s start from the simplest level and work up.

Morale

An employee who feels safe and secure in his position and his ability to raise issues will do more work. You can argue whether this is the cause or the effect, but it’s well recognized that morale is extraordinarily important. Someone who believes in the team, the company and the work he is doing is likely to do more of it. I believe motivation and morale are equivalent, but morale feels like a better word to me.

Sleep

An employee who has slept 8 hours a day will be more productive than one who has slept 5 hours. I have yet to hear anyone argue with this in principle. Studies have been done that compare the effects of sleep deprivation to alcohol intoxication. The outcome: a programmer awake for 21 hours is effectively too drunk to drive. Think about that. There is a second stage of incapacitation, which is exhaustion AKA burnout. Exhausted employees are, by definition, not productive.

Focus

An employee who knows what he is supposed to be working on will be more productive than one who is not sure which programming task takes priority. An employee who knows that he can complete the work he is doing without having his priorities changed will be more productive than one who does not know what’s coming next. Programmers dislike surprises as much as anyone else. But this also covers the programmer’s inherent self-discipline. Given a gap in work, will the programmer ask what he should be working on, or will he create his own scripting language?

Background

An employee who knows the domain, the history and the subtle interplay of forces at work in the codebase will be more productive than a programmer who has no previous background in the work assigned. This factor is often overlooked, but domain expertise is an incredibly valuable asset for a programmer, and anecdotally speaking I’d say it makes an order of magnitude difference in time and quality.

Experience

An experienced programmer — a veteran — is more productive than an inexperienced one. An experienced programmer will understand that this project needs two weeks just to be deployed, that there’s a particular tool that’s perfectly suited to refactoring the database, and when it’s worth building extra flexibility into an area of the system because the business team will almost certainly want customized logic in the next release.

Talent

Some people are just good programmers. It’s not simply intelligence, or interest; I’ve known very smart people who don’t make good programmers, and some of the most interested programmers often did the worst work. A talented programmer may do work that may be simply beyond other programmers, no matter how much experience or background they have.

Hours Worked

A programmer who works 40 hours a week will do more work than one who works 20 hours a week. And a programmer who works no hours a week, will do no work. But I think everyone will agree that a programmer who works 168 hours in a week will not produce 168 hours of work. In fact, I feel fairly safe in saying that a programmer who works 168 hours will produce less usable work than the programmer who works 40 hours, because the programmer’s ability to work will be degraded by lack of sleep long before they stop working. </

Conclusion

I don’t think anyone will argue that the above factors matter. If any one of these goes to zero, your programmers will be effectively useless.

Obviously, these factors are interconnected. Someone with no experience will typically lack background. Someone with no sleep will he low morale very shortly, and will even lose talent. So let’s see what we can do to increase these variables.

Terse Systems

Fixing X.509 Certificates

Table of Contents

What X.509 Certificates Do

Understanding Chain of Trust

Understanding Certificate Signature Forgery

Understanding Signature Public Key Cracking

Part Two: Implementation

Validating a Certificate in JSSE

Validating Key Size and Signature Algorithm

Next

Monkeypatching Java Classes

Fixing The Most Dangerous Code In The World

TL;DR

Introduction

Table of Contents

Problems

The First Problem: Programmers do not get security

The Second Problem: Wifi / Ethernet is not secure

The Third Problem: Man In the Middle

The Use Cases

Client connects to internal WS service

Client connects to external WS service

Client connects to public internets

Understanding TLS

Understanding JSSE

Configuring a client

Configuring multiple clients

Debugging a client

Choosing a protocol

Choosing a cipher suite

Next

Building a Development Environment with Docker

TL;DR

The problem

Solutions

Solution #1: Shared dev environment

Solution #2: Virtual Machines

Solution #3: Docker

Docker Best Practices

Build from Dockerfile

Chef recipes are slow

Use raw Dockerfile

The Basics

Install a internal docker registry

Install Shipyard

Create base image

Build from your base image

Push your images

Save off your registry

Tips

Exposing Services

Cleanup

eliminate:

Running from an existing volume

Sources

Play in Practice

Video

Slides

Core API

Streaming

Filters

Templating

JSON

Forms

Routing

Asynchronous Operation

CQRS

Authentication

Authorization

Security

Logging

WAR packaging

Asset Packaging

Email

Metrics

Load and Stress Testing

Deployment

Java Support

More?