.org-string { color: green; }

.org-comment { color: darkred; }

.org-comment-delimiter { color: darkred; }

.org-keyword, .org-builtin, .org-variable-name { color: blue; }

Lately I’ve been thinking about Clojure programs written in this “threaded” or “pipelined” style:

(defn large-process [input]
  (-> input
      subprocess-one
      subprocess-two
      subprocess-three))

If you saw my talk at Clojure/West (video forthcoming) this should look familiar. The value being “threaded” by the -> macro from one subprocess- function to the next is usually a map, and each subprocess can add, remove, or update keys in the map. A typical subprocess function might look something like this:

(defn subprocess-two [data]
  (let [{:keys [alpha beta]} data]
    (-> data
        (assoc :epsilon (compute-epsilon alpha))
        (update-in [:gamma] merge (compute-gamma beta)))))

Most subprocess functions, therefore, have a similar structure: they begin by destructuring the input map and end by performing updates to that same map.

This style of programming tends to produce slightly longer code than would be obtained by writing larger functions with let bindings for intermediate values, but it has some advantages. The structure is immediately apparent: someone reading the code can get a high-level overview of what the code does simply by looking at the outer-most function, which, due to the single-pass design of Clojure’s compiler, will always be at the bottom of a file. It’s also easy to insert new functions into the process: as long as they accept and return a map with the same structure, they will not interfere with the existing functions.

The only problem with this code from a readability standpoint is the visual clutter of repeatedly destructuring and updating the same map. (It’s possible to move the destructuring into the function argument vector, but it’s still messy.)

(defmacro defpipe [name argv & body]
  `(defn ~name [arg#]
     (let [{:keys ~argv} arg#]
       ~@body)))

(macroexpand-1 '(defpipe foo [a b c] ...))
;;=> (clojure.core/defn foo [arg_47_auto]
;;     (clojure.core/let [{:keys [a b c]} arg_47_auto] ...))

(def ^:private pipe-arg (gensym "pipeline-argument"))

(defmacro defpipe [name argv & body]
  `(defn ~name [~pipe-arg]
     (let [{:keys ~argv} ~pipe-arg]
       ~@body)))

(defn- return-clause [spec]
  (let [[command sym & body] spec]
    (case command
      :update `(update-in [~(keyword (name sym))] ~@body)
      :set    `(assoc ~(keyword (name sym)) ~@body)
      :remove `(dissoc ~(keyword (name sym)) ~@body)
      body)))

(defmacro return [& specs]
  `(-> ~pipe-arg
       ~@(map return-clause specs)))

(defpipe foo [a b]
  (return (:update a + 10)
          (:remove b)
          (:set c a)))

;; which expands to:
(defn foo [input]
  (let [{:keys [a b]} input]
    (-> input
        (update-in [:a] + 10)
        (dissoc :b)
        (assoc :c a))))

(defmacro defpipeline [name & body]
  `(defn ~name [arg#]
     (-> arg# ~@body)))

(defpipeline name
  pipe1
  ;; Map signifies a conditional branch:
  {predicate-a pipe-a
   predicate-b pipe-b
   :else       pipe-c}
  ;; Regular pipeline execution follows:
  pipe2
  pipe3)

(def ^:private pipe-arg (gensym "pipeline-argument"))

(defn- req
  "Required argument"
  [pred spec message]
  (assert (pred (first spec))
          (str message " : " (pr-str (first spec))))
  [(first spec) (rest spec)])

(defn- opt
  "Optional argument"
  [pred spec]
  (if (pred (first spec))
    [(list (first spec)) (rest spec)]
    [nil spec]))

(defmacro defpipeline [name & spec]
  (let [[docstring spec] (opt string? spec)
        [attr-map spec] (opt map? spec)]
    `(defn ~name
       ~@docstring
       ~@attr-map
       [arg#]
       (-> arg# ~@spec))))

(defmacro defpipe
  "Defines a function which takes one argument, a map. The params are
  symbols, which will be bound to values from the map as by :keys
  destructuring. In any tail position of the body, use the 'return'
  macro to update and return the input map."
  [name & spec]
  {:arglists '([name doc-string? attr-map? [params*] & body])}
  (let [[docstring spec] (opt string? spec)
        [attr-map spec] (opt map? spec)
        [argv spec] (req vector? spec "Should be a vector")]
    (assert (every? symbol? argv)
            (str "Should be a vector of symbols : "
                 (pr-str argv)))
    `(defn ~name
       ~@docstring
       ~@attr-map
       [~pipe-arg]
       (let [{:keys ~argv} ~pipe-arg]
         ~@spec))))

(defn- return-clause [spec]
  (let [[command sym & body] spec]
    (case command
      :update `(update-in [~(keyword (name sym))] ~@body)
      :set    `(assoc ~(keyword (name sym)) ~@body)
      :remove `(dissoc ~(keyword (name sym)) ~@body)
      body)))

(defmacro return
  "Within the body of the defpipe macro, returns the input argument of
  the defpipe function. Must be in tail position. The input argument,
  a map, is threaded through exprs as by the -> macro.

  Expressions within the 'return' macro may take one of the following
  forms:

      (:set key value)      ; like (assoc :key value)
      (:remove key)         ; like (dissoc :key)
      (:update key f args*) ; like (update-in [:key] f args*)

  Optionally, any other expression may be used: the input map will be
  inserted as its first argument."
  [& exprs]
  `(-> ~pipe-arg
       ~@(map return-clause exprs)))

(defpipe setup []
  (return  ; imagine these come from a database
   (:set alpha 4)
   (:set beta 3)))

(defpipe compute-step1 [alpha beta]
  (return (:set delta (+ alpha beta))))

(defpipe compute-step2 [delta]
  (return
   (assoc-in [:x :y] 42)  ; ordinary function expression
   (:update delta * 2)
   (:set gamma (+ delta 100))))  ; uses old value of delta

(defpipe respond [alpha beta gamma delta]
  (println " Alpha is" alpha "\n"
           "Beta is" beta "\n"
           "Delta is" delta "\n"
           "Gamma is" gamma)
  (return)) ; not strictly necessary, but a good idea

(defpipeline compute
  compute-step1
  compute-step2)

(defpipeline process-request
  setup
  compute
  respond)

(process-request {})

;; Alpha is 4 
;; Beta is 3 
;; Delta is 14 
;; Gamma is 107

;;=> {:gamma 107, :delta 14, :beta 3, :alpha 4}

further …

further …

further …

This is about three categories of software error. I have given them catchy names for purposes of illustration. The three kinds of error are the Fuck-Up, the Oh, Fuck and the What the Fuck?.

• Clojure 1.3.0 was released, bringing better performance to numeric applications, reader syntax for record types, and other enhancements.

• ClojureScript was unveiled to the world, leading to universal confusion about how to pronounce “Clojure” differently from “Closure.”

• The second Clojure/conj went off without a hitch but plenty of bang.

• Logic programming burst into the mindspace of Clojure developers with core.logic.

• Clojure-contrib was completely restructured to support new libraries, a distributed development model, and an automated release process.

• ClojureCLR got a new home and recognition as a Clojure sub-project.

• David Liebke introduced Avout, an extension of Clojure’s software transactional memory model to distributed computing.

• Clojure project management tools Leiningen and Cake decided to bury the hatchet and join forces for the greater good.

• Clojure/core started blogging more regularly.

• Books! The Joy of Clojure and Clojure in Action (both from Manning) hit the bookshelves. The Second Edition of Programming Clojure (Pragmatic) and Clojure Programming (O’Reilly) went into early release.

• PragPub magazine dedicated their July 2011 issue to Clojure.

• Chas Emerick released Clojure Atlas, an interactive documentation system.

• Clojure creator Rich Hickey spoke at Strange Loop and caused a minor tweet-storm.

• 4Clojure entered the world, giving thousands of programmers a new way to kill time and learn Clojure.

At this point, I’ve been typing and looking up links for an hour, so I’m calling it quits. Needless to say, this was a big year for Clojure, and I’m sure there’s a ton of stuff that I missed on this list. Regarding 2012, all I can say is, You ain’t seen nothing yet.

Christmas Eve will find me
In Windows Update screens,
I’ll be done by New Year’s
If only in my dreams.

I’ll be home for Christmas
Installing Google Chrome,
Removing malware, cruft, and crud,
And backing up your /home.

Christmas Eve will find me
Swearing at the screen,
Replacing it with Linux
If only in my dreams.

How much information can we determine about a program without evaluating it? At the high end, we have the Halting Problem: for any Turing-complete programming language, we cannot determine if a given program written in that language will terminate. At the low end, consider an easier problem: can we determine the names of all of the invokable functions/procedures/methods in a given program? In a language in which functions are not first-class entities, such as C or Java, this is generally easy, because every function must have a specific lexical representation in source code. If we can parse the source code, we can find all the functions. At the other extreme, finding all function names in a language like Ruby is nearly impossible in the face of method_missing, define_method, and other “metaprogramming” tricks.

Again, these decisions have consequences. Ruby code is flexible and concise, but it’s hard to parse: ten thousand lines of YACC at last count. I find that documentation generated from Ruby source code tends to be harder to read — and less useful — than JavaDoc.

Here’s another example: can I determine, without evaluating it, what other functions a given function calls? In Java this is easy. In C it’s much harder, because of the preprocessor. To build a complete call graph of a C function, we need to run the preprocessor first, in effect evaluating some of the code. In Ruby — Ha! Looking at a random piece of Ruby code, I sometimes have trouble distinguishing method calls from local variables. Again because of method_missing, I’m not sure it’s even possible to distinguish the two without evaluating the code.

I do not want to argue that method_missing, macros, and other metaprogramming tools in programming languages are bad. But they do make secondary tooling harder. The idea for this blog post came into my head after observing the following Twitter conversation:

Dave Ray: “seesaw.core is 3500 lines of which probably 2500 are docstrings. I wonder if they could be externalized somehow…”

Chas Emerick: “Put your docs in another file you load at the *end* of seesaw.core, w/ a bunch of (alter-meta! #'f assoc :doc "docs") exprs”

Anthony Grimes: “I hate both of your guts for even thinking about doing this.”

Chas Emerick: “Oh! [Anthony] is miffed b/c Marginalia doesn’t load namespaces it’s generating docs for. What was the rationale for that?”

Michael Fogus: “patches welcomed.”

Chas Emerick: “Heh, sure. I’m just wondering if avoiding loading was intentional, i.e. in case someone has a (launch-missiles) top-level.”

(As an aside, it’s hard to link to a Twitter conversation instead of individual tweets.)

Clojure — or any Lisp, really — exemplifies a tension between tools that operate on source code and tools that operate on the running state of a program. For any code-analysis task one might want to automate in Clojure, there are two possible approaches. One is to read the source as a sequence of data structures and analyze it. After all, in a Lisp, code is data. The other approach is to eval the source and analyze the state it generates.

For example, suppose I wanted to determine the names of all invokable functions in a Clojure program. I could write a program that scanned the source looking for forms beginning with defn, or I could evaluate the code and use introspection functions to search for all Vars in all namespaces whose values are functions. Which method is “correct” depends on the goal.

Michael Fogus wrote Marginalia, a documentation tool for Clojure which takes the first approach. Marginalia was inspired by Literate Programming, which advocates writing source code as a human-readable document, so the unevaluated “text” of the program is what it deals with. In contrast, Tom Faulhaber’s Autodoc for Clojure takes the second approach: loading code and then examining namespaces and Vars. Autodoc has to work this way to produce documentation for the core Clojure API: many functions in core.clj are defined before documentation strings are available, so their doc strings are loaded later from separate files. (As an alternative, those core functions could be rewritten in standard syntax after the language has been bootstrapped.)

One of the talking-point features of any Lisp is “all of the language, all of the time.” All features of the language are always available in any context: macros give access to the entire runtime of the language at compile time, and eval gives access to the entire compiler at runtime. These are a powerful tools for developers, but they make both the compiler and the runtime more complicated. ClojureScript, on the other hand, does not have eval and its macros are written in a different language (Clojure). These choices were deliberate because ClojureScript was designed to be compiled into compressed JavaScript to run on resource-constrained platforms, but I think it also made the ClojureScript compiler easier to write.

Chas’s last point about code at the top-level comes up often in discussions around Clojure tooling. The fact that we can write arbitrary code anywhere in a Clojure source file — launch-missiles being the popular example — means we can never be sure that evaluating a source file is free of side effects. But not evaluating the source means we can never be sure that we have an accurate model of what the code does. Welcome back to the Halting Problem.

Furthermore, maintaining all the metadata associated with a dynamic runtime — namespaces, Vars, doc strings, etc. — has measurable costs in performance and (especially) memory. If Clojure is ever to be successful on resource-constrained devices like phones, it will need the ability to produce compiled code that omits most or all of that metadata. At the same time, developers accustomed to the “creature comforts” of modern IDEs will continue to clamor for even more metadata. Fortunately, this isn’t a zero-sum game. ClojureScript proves that Clojure-the-language can work without some of those features, and there have been discussions around making the ClojureScript analyzer more general and implementing targeted build profiles for Clojure. But I don’t think we’ll ever have a perfect resolution of the conflict between tooling and optimization. Programs are ideas, and source code is only one representation among many. To understand a program, one needs to build a mental representation of how it operates. And to do that we’re stuck with the oldest tool we have, our brains.

Versions: Almost everyone uses 1.6. A few are still using 1.5, and a few are trying out 1.7. Only a handful are still on 1.4. Fortunately, no one is on a version older than 1.4.

Reasons: These are more varied. The most common reason for not upgrading is lack of time, with “it just works” running a close second. A little less than half of respondents are limited by external forces: either operations/management or third-party dependencies.

Not much came out in the comments. Banks and other large institutions seem to be the most resistant to upgrades, especially if they’ve been bitten by past JDK changes.

Git

First and foremost, Git. We use Git on every single project, internal and external. Git is a great example of the Unix design philosophy: many small programs — 153 of them by my count — each of which does exactly one thing and does it well. But this is not “loose coupling.” The components of Git are tightly integrated: they all depend on the same repository structure and file formats.

One of the nice things about Git is how its internals are both exposed for the world to see and thoroughly documented. We can easily write scripts to automate common tasks or create different workflows. With a bit more effort, we could even write new tools that integrate with the Git suite. These tools can do things that Git’s authors never intended, as long as they follow the documented repository structure. Git isn’t so much a version control system as the means to construct one.

Still, Git is one project with many components, not many separate projects. All 153 executables in Git are governed by a single release cycle, tested and known to work together. We never have to worry about incompatible versions of, say, git-branch and git-merge on the same machine. Older versions of Git can read repositories created with newer versions even if they don’t provide all the same features.

Maven

In stark contrast to Git, we have tools from the Java world like Ant and Maven. The JVM cannot fork/exec, so the many-small-programs design is a non-starter. Instead, the Java tools usually favor some sort of plug-in architecture, which is a great idea in theory but hard to get right in practice.

I’ve tried writing a Maven plugin. Hacking up a one-off for a single project is not too difficult, but designing a general-purpose plugin that works everywhere is maddeningly complicated. Maven plugins are just Java code, so they can do whatever they want, but the APIs for interacting with the rest of the Maven system are woefully underdocumented. The contract of a Maven plugin, what it can and cannot do, is not well-defined. The internals of Maven itself are largely a black box.

The core Maven plug-ins have independent release cycles, so there is the possibility for unexpected incompatibilities, but I’ve never encountered such. On the whole, the Maven ecosystem is quite stable. The struggle comes once you venture outside the realm of what the standard plugins provide. Maven plugins are not designed to be composed, so adding new capabilities is rarely as simple as scripting plugins that already exist. You have to start from scratch every time.

Ruby / Rubygems / RVM / Bundler

Finally, we have tools from the Ruby world, the ever-changing cornucopia of Ruby implementations, libraries, and tools to manage it all. The problem with the Ruby tools is that they are both tightly-coupled and uncoordinated. Despite having separate tools for each task, each tool reaches into at least one of the others: Rubygems modifies the behavior of the Ruby interpreter, Bundler modifies the behavior of Rubygems, RVM modifies the behavior of the shell, and so on. Each one adds another layer of indirection, making debugging harder.

All of the Ruby development tools have independent release cycles, and they don’t seem to plan or coordinate with one another in advance of each release. Integration testing is left up to the users.

I admire the speed and eagerness with which the Ruby community produces new tools. But on almost every Ruby project I’ve worked on, we’ve spent hours or days sorting out incompatibilities among some combination of libraries, language implementations, and development tools. Our internal mailing list is littered with advice like “Don’t use Bundler version X with RVM version Y.” The speed of development comes with its own cost.

Thoughts

So what do I take from all this? Just a few principles to keep in mind when writing software tools:

1. Plan for integration
2. Rigorously specify the boundaries and extension points of your system
3. Do not depend on unspecified behavior

And a couple of ideas if you’re starting a new project from scratch:

1. The filesystem is the universal integration point
2. Fork/exec is the universal plugin architecture

Update 8/31: More comments at Hacker News.

Most programming languages, especially object-oriented languages, provide strong hints on how to structure your source files. Everything is a class, and (almost) every class is a file. In Clojure, everything (almost) is a function. Functions are much smaller units than classes. So how do we group them?

The important thing to remember about namespaces is that, from the compiler’s point of view, they don’t matter. Namespaces are a convenience for the programmer, to help you avoid name clashes without having to write longer names. There’s no reason why an entire application can’t be defined in a single namespace. Most of Clojure itself is defined in one namespace with over 500 symbols. (Common Lisp has 978 symbols in a single namespace.)

You can think of namespaces as a tool to express something about your application. Here are some ideas to get you started:

• Group functions into namespaces based on type of data they manipulate. For example, functions to manipulate customer data go in the “customer” namespace. This technique is familiar from object-oriented languages, but it has the same limitations: where do you put functions concerning relationships among two or more types? The OO answer would be to make a new name for the relationship. This style leads to a proliferation of small namespaces, which can become a burden.
• Divide a library into a public API namespace and a internal implementation namespace. Or define a high-level API for common cases and a low-level API for more advanced usage.
• Divide an application into namespaces representing architectural layers. You can examine “ns” declarations to prove that each layer calls functions only from the layer below it.
• Divide an application into namespaces representing functional modules, with well-defined contracts for communication between modules.
• Try to separate decision-making code from the code that carries out those decisions. That is, keep your business logic purely functional and free of side-effects, so it is easy to test. You don’t necessarily have to put side-effect code in a separate namespace, but doing so may help keep it cleanly separated.

With all these techniques, the point to remember is that namespaces are there to help you, not to get in your way. If you have a large namespace, you can still divide it up into multiple files.

The one hard-and-fast rule is that you cannot have a circular dependency between namespaces. That is, if namespace A needs to call functions defined in namespace B, then namespace B cannot call functions in namespace A. There is no workaround, it simply can’t be done. In practice, this is rarely a problem. If you encounter a situation where two namespaces are mutually dependent, it’s probably a sign that they should be merged into a single namespace.

In client projects at Relevance (home of Clojure/core) we often end up with one namespace for each aspect of an application — data access, UI, logging, and so on. Then there’s one “main” namespace that depends on all the others and ties it all together.

Update 9/5/2011: Chris Houser wrote a nice answer on StackOverflow about how to split a Clojure namespace over several files.

The event was a great success, with loads of excitement expressed by both the local New York crowd and the Internet at large.

Screenshot of IRC / Twitter during the ClojureScript announcement

Video was also recorded, which will be posted soon.

Thanks to Google New York for hosting.