Philosophy Made Manifest

De Morgan’s Laws are (Mostly) Constructive!

noreply@blogger.com (Marc Hamann) — Wed, 05 Sep 2012 22:27:00 +0000

Most programmers and anybody with a basic knowledge of logic will have heard of De Morgan’s Laws. They provide an easy way to switch back and forth between “or” and “and” in Boolean logic, which is very handy.

When I was first introduced to intuitionistic logic, I remember being somewhat scared off it by the statement that De Morgan’s Laws don’t hold in it. This will make any programmer nervous!

So, it came as a surprise to me to learn that in fact three quarters of the propositions that make up De Morgan’s Laws still hold for intuitionistic logic!

It is interesting to examine why. De Morgan’s Laws can be broken up into four implications:

1) (¬ P) ∧ (¬ Q) → ¬ (P ∨ Q)

2) ¬ (P ∨ Q) → (¬ P) ∧ (¬ Q)

3) (¬ P) ∨ (¬ Q) → ¬ (P ∧ Q)

4) ¬ (P ∧ Q) → (¬ P) ∨ (¬ Q)

Of these four implications, only the fourth one fails for intuitionistic logic!

Let’s check.

Case 1) If I have a proof that P is false, and a proof that Q is false than I can’t possibly find a proof that either P or Q is true, by the basic definition of ∨.

Case 2) If I have a proof that P ∨ Q can’t be true, then clearly there is no proof for either of P or Q, by the basic definition of ∨.

Case 3) If I have a proof that P is false, or a proof that Q is false, there can’t possibly be a proof that both P and Q are true, again by the basic definition of ∧.

So far so good. But here is where things go awry.

Case 4) If I have a proof that both P and Q can’t be true at the same time, that doesn’t tell me for sure that I have proof that either one in particular is false. They just can’t both be true. So case 4) isn't guaranteed to always be true, and therefore isn't a law after all.

However, if I find a definite proof that one of P or Q is true, I can still conclude that the other is false. This can be formalized as:

4') ¬ (P ∧ Q) → P → ¬ Q

So even this fourth case, with a small caveat, still contains a lot of useful information. Maybe we can say that De Morgan’s Laws are more than three quarters constructive!

The moral of the story is not to be scared away by constructive (i.e. intuitionistic) mathematics: more of the familiar rules still work than you might think, even if you have to be a bit more careful around the edges.

The Intuitive Minority

noreply@blogger.com (Marc Hamann) — Wed, 09 Nov 2011 17:20:00 +0000

Many people are skeptical about personality typing systems. Some complain about their over-generality. Others simply dislike being categorized. Personally, I have found them very useful for thinking about my own patterns of behaviour. They also help me to understand a person whose style is different from mine, and might otherwise be mystifying.

One of the most commonly used personality typing systems is the Myers-Briggs Type Indicator (MBTI) . It has four parameters with two settings for each: Extraversion vs Introversion (E vs I), Sensing vs Intuition (S vs N), Thinking vs Feeling (T vs F), and Judging vs Perceiving (J vs P). Each personality type is composed of four letters, one from each pair. For example, I am an ENTJ (Extraversion-Intuition-Thinking-Judging).

I want to focus in on one of these pairs, Sensing vs Intuition (S vs N). If you take a look at MBTI type statistics , you will notice that most of the pairings are pretty close to 50-50, with a little bit of skew for T vs F and J vs P, but that there is a very noticeable bias in the S vs N category: 75% of the population are Sensing (S).

With a majority that big on the other side of the divide, it is pretty much guaranteed that anyone in the N camp is going to be perceived as slightly odd relative to the personality norms of society. If you add N and T together, you get the NT temperament (sometimes known as “Rationals” ), which make up less than 10% of the population. It is not surprising that some members of this group (especially the introverts) have sometimes been stigmatized as “nerds” by the majority.

The summary explanation of N vs S characteristics is this: the N is more oriented toward abstract ideas, the big picture, the past, the future and general theories of things. If you need proof that I’m an N, this blog is proof enough: it is characteristic for an N to lump together superficially disparate areas such as, for example, philosophy, software development, and personality system, while presenting them as a coherent subject matter.

The S is a more concrete thinker, deeply rooted in their own direct experience, present circumstances, the nitty-gritty. They tend to characterize particular things by visible attributes. They tend not to like the kind of vague categorizations that Ns prefer. My experience is that S types don’t think that much of personality type systems, except as a kind of game or as handy stereotypes, since they don’t really think in terms of systems and tend to be annoyed by generalization that goes beyond their experience.

Steve Jobs, whose recent passing has caused such a vast cultural effect, was almost certainly an N. He saw beyond the current situation, persisted through some difficult times (such as the early days when Macs had a very small market share and his NeXT years) by focusing on what could be. Ironically, he also now appeals to Ss since his vision has become concrete and present in their lives in an obvious way: an S does not argue with clear and present success.

In some ways, I think it makes sense that most people are S. Paying attention to the here and now pays off more regularly, even though it has less potential to pay off big the way truly original and disruptive vision can. Intuition is a high risk/high reward proposition. The conditions that make it pay off are more remote and less certain, but the advantage of being ahead of the game when change comes can be enormous. A society without N would never progress, but a society with too much N might change too fast or fly off in too many directions.

As an N, I have found that understanding that most people don’t think like I do makes me a more effective communicator in practical situations. For example, when I detect the tell-tale signs of scepticism or confusion when I’m explaining an idea or plan to someone, I find that giving concrete examples with illustrative detail is often a good place to start. (This can help communicate with Ns as well, since another N might want to detect the patterns for themselves in your data.) When talking to an S, the trick is to choose examples that aren’t too different, to add in “irrelevant” detail for each, and then point out what the shared relevant details are that make the pattern.

To give an example, let’s say you want to make a point about luxury sports cars. This is a simple enough abstract category that it likely won’t cause much communicative difficulty, but it would nonetheless be more effective to give a specific example, such as mentioning a Porsche, or even a better, a specific model, such as Porsche 911. The extra detail, which can seem arbitrary to an N, helps to anchor the idea more concretely for an S. An N’s instinct is to leave the question open, since the particular model doesn’t matter to the point, but for the S choosing an appropriate example with the right specifics does matter.

The flip side of this technique can be used in rapport building or social conversations when you find that the person you are speaking with is giving a lot of specific detail. Instead of getting bored and tuning out, it can be helpful to pick out a particular detail that sounds interesting to you and ask for more context about it. To pick up on our previous example, suppose an S is discussing sports cars, most likely mentioning a specific make, the Porsche 911. You can ask the S what makes that particular make better/worse/special, depending on the context. To make this more meaningful, you could provide a different specific make as a comparison example. This helps to make the detail relevant to the N, while also engaging the S in a mode of conversation that is comfortable.

Careful observation of such interactions will lead to better results and more techniques. Armed with knowledge of this phenomenon, I hope all Ns will learn to better cope in a world in which they are a minority.

Curry-Howard for Non-Dummies – Part 3

noreply@blogger.com (Marc Hamann) — Mon, 25 Apr 2011 13:24:00 +0000

In Part 2, we considered the implication A → ⊥, saw that in the logic of the original formulation of Curry-Howard (known as Intuitionistic Logic for those who care) it was equivalent to asserting that A had to be an empty type. We also speculated that in some other interpretation, A → ⊥ might be the type of a program that never terminates. We’ll see if we can make this idea useful in a moment.

Double Negation

But first, let’s consider the type ¬¬A, which, using the formulation described in Part 2, expands out to (A → ⊥) → ⊥.

In the normal Boolean logic that most programmers will be familiar with, if you negate a false value, you get a true value. (This kind of logic is Classical Logic.) One way to define this property is to say that there is a program with this type: ¬¬A → A in whatever programming language corresponds to Classical Logic.

But in the logic we have been using so far this won’t quite work. As mentioned at the end of Part 2, we need to show that A comes from some place on the left side of the arrows. Since it isn’t made clear in this type, it won’t work for us. We can fix this by rearranging the expression so that A appears on the left side, which means that we can build a program of the type A → (A → ⊥) → ⊥.
To see that this is so, there are two cases to consider.
The first is if A is an empty type. Then we are back to a case like ⊥ → B;, where the program is never going to get any input and so it doesn’t really matter what is on the right side. If A is a type with members, then we already know that the type A → ⊥ is empty, so the second part of the program is just the same as the ⊥ → B; case. So either way, we can build this program using the rules we were already using.
Now let’s consider what we need to add in order to make the Classical Logic rule work.

Classical Logic

In order to get the behaviour we expect from Classical Logic, we need to be able to do something that we just showed is not possible for our previous programming rules. We need to be able to build a program that fits the description of the proposition type ¬¬A → A, which expands to ((A → ⊥) → ⊥) → A.
To be able to build such a program, we need to have a typing scheme that is a little bit less strict than the one we’ve been using – one that is willing to “wait and see how things turn out” before fully assigning type. It needs to be able to risk that the program might not terminate and, to really fulfill the proposition type, it needs to be able to make sure that we always end up with a definite value regardless of what actually happens.
In this sense, Classical Logic is the logic of perfect information, where you always have enough knowledge to see what is true and what is false, and you can always stop things when they go awry to make sure they turn out in the end.
There are two ways to ensure that you have this kind of knowledge and power: the first is to always deal with a world that is “small enough,” so that you can always find out the truth of things in a reasonable amount of time. For example, if we restrict our purple unicorn-finding program to looking for it in the room we happen to be in at the moment, then we can be sure that we will either find a purple unicorn or definitely not find a purple unicorn in a finite amount of time.
The second way to ensure the Classical Logic rule is to work in a “arbitrarily big world” with “magic powers” that allow you to find out the truth of things even when they are impossible. An example of this would be to have a “Halting Oracle”, a magic detector that will tell us whether our program will eventually terminate or not.
It turns out that there is a particular type of program that meets the description of the proposition type ¬¬A → A. It falls somewhere between the “small enough” and “magic power” solutions and is known as a “continuation”. This kind of program basically allows you to abort a program and return a value anyway. You can think of a “return” statement or an exception as a poor-man’s continuation. In principle, the program that you abort could have been non-terminating, but you get to see this and abort it to make sure it turns out anyway. The “seeing this” part is outside of the system itself, so it is sort of a magic power, but in practice we rely on the fact that we are working with a small enough world that the programmer can see what would happen and choose to abort the computation.

Searching for Purple Unicorns

Let’s return to our program that uses a spy satellite to search for purple unicorns.
Will it run forever or will it terminate? If we stick with our constructive method of building facts, then it will only terminate if in fact purple unicorns exists and it finds one. Unfortunately, there are ways that purple unicorns could in fact exist, but our program still won’t terminate: for example, if during the day purple unicorns hide in caves and at night when they come out it is too dark for our satellite to spot their particular shade of purple. Unfortunately, in this case our program doesn’t really fulfill its type and it doesn’t really prove anything at all, one way or the other.
If we avail ourselves of the Classical approach to things, we can make our program terminate even if there aren’t any unicorns by, say, allowing the satellite to do one orbit and then just assume that we were right to begin with that they don’t exist and abort the program with a negative result. This basically forces truth into a box we find convenient and doesn’t really prove anything either. It is just a fancy form of axiom.
Another option is to not worry about being able to reify our ideas and proofs as programs, assume that purple unicorns exist and develop elaborate theories about their habitat, physiology and behaviour. These might be interesting theories and some of them might be true and consistent with the real world because, by the way we have defined them, we know quite a lot about them. For example, we know that unicorns very horse-like, so many things that are true about horses can be comfortably said to also be true about purple unicorns. We can also assume that their characteristic purple is a known shade of purple. In this way, theories of purple unicorns might we’ll be perfectly compatible with existing zoological studies, or colour theory. Assuming the impossible does not necessarily produce inconsistency.
Medieval scholastics were able to do some good logic, rhetoric and philosophy by contemplating things that most of us think are impossible, such as angels dancing on the head of a pin.

Implications for Mathematics

Curry-Howard gives us a possible definition for truth in mathematics linked to computation. At this point in time, most mathematicians don’t seem all that interested in this possibility, since they are fond of studying purple unicorns, such as the kind known as “uncountable sets”. I don’t really want to dissuade them from this study, anyway. Purple unicorns are fun to think about, and can sometimes provide inspiration for useful things. (See continuations above.)
However, both mathematicians who work with computer science and computer scientists who have Classical mathematical training need to be very careful how they think about computational truth: it doesn’t work the way they are used to.

Implications for Logic

Curry-Howard gives us a reason to explore logics that are subtler or more complex than Classical logic. Computation requires us to incorporate a notion of uncertainty into our thinking, and logics that can handle this might offer new vistas of useful thought to explore. Lots of great work has been done this way.

Implications for Computer Science

Curry-Howard reminds computer scientists that we are not afforded the luxury of either believing in purple unicorns or being too certain that purple unicorns don’t exist. Infinity is a big place after all, and our programs can’t explore all of it.
Curry-Howard also gives us a tool to draw inspiration from what logicians and mathematicians are doing to better understand how to think about computation. We need all the tools we can get.

I hope that any brave soul who worked all the way through to the end of this has gleaned an understanding of Curry-Howard, and an appreciation for the rich, cross-pollination of disciplines that it enables.

Curry-Howard for Non-Dummies – Part 2

noreply@blogger.com (Marc Hamann) — Mon, 25 Apr 2011 13:24:00 +0000

In Part 1, I described the “Curry-Howard” principle in very simple terms. I culminated with the explanation that a program can be seen as a proof of the implication represented by A → B, where A is the type of its inputs and B is the type of its outputs. Moreover, A → B is itself both a type and a proposition in a simple implicational logic.
And that’s all well and good, but now it’s time to see some wrinkles in this idea in preparation for exploring the consequences of Curry-Howard upon mathematics, computer science and logic in Part 3.

What Happens When a Program Doesn’t Terminate?

In Part 1, you may have noticed that I made a bold assumption about my programs. The rationale for considering the program to be a proof of the implication was that I was able to run the program by consuming a thing that met the description of an A and produce a thing that met the description of a B. My program is a proof by construction.

But what happens if I made a mistake and wrote a program that doesn’t terminate for all inputs that meet the description of an A?

Let’s return to the example where the description of A is “a purple unicorn” and the description of B is “a horse-race winner”. Let’s say I decide that what I’ll do is create a new program whose input C has the description “a spy satellite in orbit” and I conceive a program for the implication C → A that says that I’ll have the satellite orbit the Earth, checking every square inch of it until it finds a purple unicorn, at which time it will let me know its location, and I’ll go capture it. Once I have this purple unicorn, I will immediately pass it to my previous program as input and will then be able to produce a horse-race winner. We can think of the composition of these two programs together as a new program that is a proof of the implication C → B. But are we sure this works?
It could be that there is such a thing as a purple unicorn after all, and that it just so happens that no one has ever seen one and that our spy satellite, which is very thorough, will do its job and find a purple unicorn that was previously missed. In this case, we would have a proof of C &arr; B. But maybe there really is no such thing as a purple unicorn (on the surface of the Earth, anyway), and our satellite will just circle the Earth forever.
In this latter case, I have a fully described program that would take a spy-satellite and produce a horse-race winner, but if I try to run it, it basically hangs up on the first step and never completes. So merely having a description of a program is not sufficient to have a proof. What I really need is the description of a program that is guaranteed to terminate for all its allowable inputs.

So does this mean that a potentially non-terminating program has no place in a Curry-Howard view of things? Is there not still some value to having some well-defined program that witnesses an implication, as opposed to having no proof at all of the implication, except perhaps “because I said so?”.
To tackle this, we must expand our repertoire of concepts a bit.

Negation

In Part 1, I covered how we can define the truth of the proposition represented by a type as the existence of a thing that meets its description. If the description of A is “purple unicorn” and I can present a purple unicorn, then A is true, and if there is no purple unicorn to be found, then A is false.
So any type that is false must have no possible members. If you think about this a moment, then you realize that any two types that are false must share the same list of members, i.e. none. This means that they share the same extensional definition. And by the same token, the members of both types will all meet the description “a thing that is impossible”. So they also share an intensional definition.
In this sense, all empty types are equivalent to a single basic empty type, which we will give as a name the symbol ⊥. This symbol can be pronounced a number of ways, but we can continue to think about it as “the empty type”.
Is there a way to use this type in an implication to produce interesting types of programs?
An obvious possibility is to put the empty type on the left side of an implication: ⊥ → A. As a program, this would be a program that takes members of the empty type as input. This is very much like our “purple unicorn produces a horse-race winner” program. It is a program that (if we are correct that there are no such thing as purple unicorns) will never get any input (since there are no instances of the input type to pass to it), and therefore it will never produce its output. This doesn’t mean that the type of its output is necessarily empty, just that this function can’t be used to prove anything definitely, or, looking at it from the other end, you could say that this kind of program could prove anything, since a program that isn’t going to run can promise anything you like, since it will never have to deliver. (Just like a politician who knows he won’t be in the next government…)
The next obvious possibility is to put the empty type on the right side of the implication: A → ⊥ . This one is a little bit trickier to make sense of. At first glance, this one might be a good match for our C → A implication, the “take a spy satellite and find a purple unicorn” program. And that has some interesting possibilities. But normally this form of implication is interpreted to mean ¬A, which is an assertion that A is empty and false. To explain how this works, we need to explain a little bit more about the original formulation of Curry-Howard.

The Original Curry-Howard

To understand the original formulation of Curry-Howard, you must understand some basic facts about a couple of things, which have been known to get quite complex but for our purposes are extremely simple. The first is the Lambda Calculus, which is really just the world’s simplest programming language. There are basically two “commands” in this programming language. One allows you to define a program that takes an input, and the other allows you to pass input into a program that has been defined by the first “command”. There are some technical details if you are really interested, but seriously, that is all there is to this.
The amazing thing about this simple programming language is that it is as powerful as any other programming language you could invent. The thing that makes this language so powerful is that you are allowed to pass a program to itself as input. Under certain circumstances, if the program gets applied as part of its own execution, the program might never finish. This is exactly like defining a function that contains only a call to itself: it will just go in circles forever.
Now, there is a limited form of the Lambda Calculus known as the Simply Typed Lambda Calculus, which basically works like the implication types we’ve been talking about. The important thing to know about the types in this programming language is that the description of the inputs and outputs of a program must be simpler than the description of the type of the implication itself. This ensures that you can’t pass a program to itself as input, and this means that all programs in this language always terminate.
To illustrate, consider a simple program that has the type A → A. Since its input is of type A, and since its own type is A → A, which is more complex than simply A, it can’t possibly get itself as input.
The amazing thing about this language is that, despite giving up the most important trick of the full Lambda Calculus (recursion), you can still do quite a lot with it, including a significant portion of arithmetic.
Another important attribute of this language is that it is strictly constructive, which is another way of saying that you can only output on the right side of the implication things that are built up from things supplied on the left side. This ensures that we don’t pull any conclusions out of thin air, and that we can always trace the origins of a particular conclusion from what we started with through the well-defined building steps along the way.
A consequence of this fact means that the implication A → ⊥ can only exist if A is in fact an empty type. This means that the only type allowed to have ⊥ on the right is ⊥ → ⊥. This explains why saying A → ⊥ is the same as saying ¬A., i.e. A is empty and false.

We now understand enough about Curry-Howard to think about some of its consequences, which we will explore in Part 3.

Curry-Howard for Non-Dummies – Part 1

noreply@blogger.com (Marc Hamann) — Sun, 17 Apr 2011 23:57:00 +0000

(Anyone who wants to learn is not a dummy)

At the risk of alienating the (small) audience who is used to me writing about not-too-technical topics, I would like to take a crack at an important but technical topic in theoretical computer science. This topic is often known as the Curry-Howard Isomorphism, though there are variations on its name. Despite the forbidding name, the principle it represents is useful to understand for anyone who is interested in logic, computation or programming languages, which would obviously include programmers of all skill levels. I hope to provide an introduction that anyone in this wider audience can understand.

Programs are Proofs

The very basic idea of Curry-Howard is that you can think of a program as a proof. A proof of what? A proof that if you provide certain inputs to that program you can produce the corresponding output.
Another way to think about this is to understand that both programs and proofs are series of rule-following steps that allow you to transform what you started with into a new result.
We often take this generating capacity of programs for granted, but the very nature of programming requires a deep understanding of the structure of inputs, the structure of outputs and the details of the relationships between them. This is exactly the same kind of activity as creating a proof in math or science.

Programs are Implications

In logic, the simplest way to state a proposition that captures this inputs-to-outputs behaviour of programs is as an implication: A → B, ( read as “A implies B”). This means that if I already know that A is true then I will also know that B is true.
To see the parallel to a program, I have to understand that “A is true” is equivalent to “I can find a thing which fits the description of A”.
This last step is perhaps not quite so obvious, so let me give an example. Let’s say that A stands for “a purple unicorn” and B is “a horse race winner”, then A → B could mean that “if I had a purple unicorn I would run it in a horse race, and since purple unicorns are magically fast, I would be sure to win”.
You can see that this is a kind of program: give me a purple unicorn as an input, and I can follow a series of steps to produce a horse race winner.
While this may be a good program, it can’t be run, since purple unicorns don’t exist. And since purple unicorns don’t exist A can’t be true.
If I change my program to match C → B, where C is “a well-bred horse”, I’m back in business: my program can now be run again, when I run it (or rather run the well-bred horse in the race) and produce a horse race winner, I will have proved the correctness of the implication.
To return to a slightly more programmatic example, let’s say that A is “a number” and B “a number greater than A”, and my program adds one to its input. I then feed it “1” as input, my program produces “2”, which is indeed greater than 1, and I’ve proven that, given the number “1” I can produce a higher number.

Types are Propositions

However, we aren’t quite done here. I just showed that for a particular thing that met the description of an A (“a number”) I could get a particular B (“a number greater than 1”), but what I really need to do is show that any thing which meets the description of an A can always be used to produce a B using my program. In order to do this, I need to think a bit more about what it means to be a “thing which meets the description of A or B”.
There are basically two ways to define “things that meet the description of A”: by listing all the individuals that meet the description (this is called an extensional definition) or by proposing a rule (typically an inductive rule) that allows me to produce or recognize an A at will (this is called an intensional definition).
It so happens that we already have a concept in programming languages that does this definitional work for us: types. Types can be defined by an exhaustive list. For example, the character type is typically defined by a list of ASCII or Unicode characters that is fully known already. Types can also be defined by a rule. For example, a number type might be defined as any sequence of digits. You wouldn’t have to list all the numbers (there are probably too many) – you just have to check the simple formation rule to determine membership in the type.
Based on this, we can see that any type can be construed as a logical proposition that can be verified by lookup or by rule: just produce an individual item that meets the description and we know that our proposition is “true”.

The Type of Programs

We now have enough machinery to show that our program works for all of its inputs. We showed that A and B are both types and propositions. We now have to see that the implication A → B is also both a type and a proposition. To meet the description of the proposition A → B we have to show that our program will take any A and produce a B. As with our simple propositions, we need either an exhaustive list or a rule that allows us to cover all the bases. The easiest way to do that is to start with our definition of A. When we look at the structure of our program, we will have to see that there is either an exhaustive list of the elements of A or some rule that follows the rule for A to cover all the possible values of A, and then for each path that results from this rule or lookup, show that a B will result.
This can be very complicated for complex programs but it is often possible to break a bigger program up into smaller bits, each of which can be checked in this manner. Generally programming languages and type systems are designed so that the compiler can verify the type of a program automatically.
When we have completed the process of verifying our program, we will know that our program fulfills the description of the proposition A → B, that it conforms to the corresponding type, and that it is also a proof of that proposition.

If you have followed this far, you have grasped the basic idea of Curry-Howard. It is, I hope, a surprisingly simple idea, but with profound consequences for logic, mathematics and computer science. In Part 2, I hope to expand a bit on some of the elementary consequences.

Truth, Sales and Leadership

noreply@blogger.com (Marc Hamann) — Sat, 09 Apr 2011 23:21:00 +0000

Salesmen, politicians and chief executives are all cut from the same mould. Their fundamental job is to impress and make others feel good about buying what they have to sell.
As someone who has spent his career in product delivery, and who is sceptical by nature, I’ve had a mixed relationship with the sales and presentation types. I have all too often ended up on the hook when one of them sold unicorns when what we had in the barn was donkeys.
But, unlike many technically oriented delivery specialists, I have a deep appreciation for the value and necessity of presentation and sales. I understand the symbiotic nature of our separate disciplines. In the end, things only work out well when both roles are filled competently, and both sides need to remember their dependence on the other if they want to succeed.

To Impress or to Be Impressive

If at this point you aren’t sure which side of this proposed divide is your natural home, let me propose a simple way to decide: which do you think is more important, to impress or to be impressive? If you don’t understand this distinction, let me rephrase: would you rather be famous and admired for something that you didn’t put a lot of effort or skill into, or be amazingly accomplished at something that no one knew about you?
Now some of you are I’m sure objecting that these are always or usually the same thing: accomplishment and skill, and recognition for that accomplishment and skill generally go hand in hand.
But the truth is that it is rarely the case that the two go together naturally. Are the best singers and actors the biggest stars, and vice versa? Are the most successful politicians most often the smartest and most qualified decision-makers? Is your average CEO the biggest expert in the business he is running?
This isn’t a complaint that the world isn’t fair. On the contrary, I think there is a good reason why things work this way. Impressing people requires that you present yourself in terms that they can understand, evaluate and appreciate. Being impressive requires honing your abilities and understanding beyond the experience and comprehension of most people, to make distinctions and to have perceptions beyond what is evident to a non-expert.

When Sales or Fulfillment Goes Bad

Each of these sides has its obvious pitfalls, and to understand these, all you have to do is think about the negative stereotypes of these two groups.
The bad salesman or politician is full of hot air, with plenty of promises that never get kept. The bad salesman is convinced he could “sell snow to the Eskimos”, the bad politician certain that he could finesse over the most egregious scandal and get away with breaking every promise.
The bad expert (let’s use software developer as an example) is disdainful of the buying public, mystified that they don’t see the deep technical skill that went into solving several difficult and subtle challenges but are instead impressed or put off by trivial details, such as the colour choices on the main screen. The bad software developer is convinced that good software will always sell itself, and sales and marketing are clueless bozos who soak up money and accolades that rightfully belong to the technical geniuses.
Both of these stereotypical bad guys are wrong: they desperately and inseparably need each other.

Commanding Officer and Executive Officer

To illustrate the symbiosis of these two callings, I will draw on an example that has literally been battle-tested over centuries. It is traditional in the military for there to be two command roles for every unit: Commanding Officer (CO) and Executive Officer (XO).
The CO is the external face of the unit. It is his job to represent the skills of his unit up the chain, to convince the command hierarchy to give his unit more resources, and to cheerlead and praise his men for their accomplishments. Ideally, he should be loved by his men, and they should feel proud and honoured to serve under him.
The XO is the internal manager and disciplinarian of the group. It is his job to push, prod, terrorize and manhandle his men to improve their skills and achieve their mission goals. He must be the first to observe and criticize any short-comings and ruthlessly punish any breach of discipline. He is the expert in building the best possible soldiers. Ideally, he should be respected and feared, at best grudgingly liked, and made fun of when the men are absolutely, positively certain that he can’t hear them.
With such radically different profiles, you would expect that these two officers should not work together very well, right?
But the truth is that, if both men actually understand their roles, they should be working in perfect collusion with each other. The XO should be feeding the CO with good things about the men that can be praised, and let him know when the men are achieving their best and can’t be expected to give anymore. The CO will let the XO know if any bad reports are coming in from outside the unit and share his fears of how the unit might fail in upcoming missions.
The two roles are distinct, and require different focuses, but they must be done in perfect concert to achieve maximum effectiveness in the situation. Such dual leadership roles can and often are performed by a single person in non-military leadership situations, but there is something to be said for the collaborative specialization of individuals dividing up the areas of responsibility. Even if you find yourself with both roles, you have to find some way to separate them in the minds of others so that they don’t contaminate and undermine each other.

Truth and Presentation

I think we would all agree that the ideal situation is the one where things both seem good and are good, but where the downside is also known and managed. Being aware that some aspects of the truth (possibly not the most important ones) need to be highlighted for the sake of presentation is important for experts to understand. Keeping in mind that finding and fixing negatives is essential for quality products and that it is in everyone’s interests to not diverge too far from the truth in presentation is vital for sales to understand.
A well-packaged truth can only come about when both perspectives work together and respect each other.

On Being an Amateur Academic — Part 3

noreply@blogger.com (Marc Hamann) — Mon, 04 Apr 2011 17:09:00 +0000

The Pros of Being an Amateur Academic

While I’ve already pointed out that there are some real problems and downsides to being an amateur academic, there are some distinct bonuses over being one of the professionals. I will discuss the three most important ones, in my opinion.

Free to Wander

There was a time when I was seriously considering going to grad school and becoming a professional academic. The biggest difficulty I had, other than how to support myself if I took that route, was choosing a focus for my studies.
My interests have always been wide-ranging, and in order to be taken seriously for graduate studies, I would have had to focus down to a tiny sliver of one conventionally defined discipline out of all the ones that seemed interesting.
To make this worse, in practice what you have to do is find an advisor who will tell you what to do, preferably quite closely aligned to their interests. (More on that later.)
Ignoring the fact that you can get caught in a dead-end area of focus that is on the verge of going out of fashion, from the point of view of sheer intellectual curiosity you need to accept that, for many years at least, you will have to stick to something pretty close to your chosen topic if you want to have any hope of finishing your degree and getting a job. (Two huge and separate hurdles.)
Once a professional academic has established their career, they can and do branch out, but you have to be able to stay devoted to one thing for longer than I can manage.
To give you an idea of the range of interests I’ve been free to explore, mentioning only fields that I have spent a non-trivial amount of time studying over the last ten years (the time it can take to finish a PhD and get an appointment), and including only topics I think I could have a sensible discussion with a pro about at, say, an intellectual dinner party, or in an online forum: computer science (programming language theory, type theory, finite model theory; mathematical logic), mathematics (foundations of mathematics, category theory, abstract algebra, model theory, proof theory), linguistics (syntax, phonology, sociolinguistics, historical, computational), ancient languages (Classical Greek, Classical Arabic, Classical Chinese), music (Middle Eastern music theory), as well as various topics in history, philosophy, political science, and the arts.
This is an impossibly diverse list by most academic standards, and to be honest I’m not sure how or if I’ve managed to grasp all these, except that I tend to go through periods of weeks or months focused on one or two them, and over time I revisit old topics to refresh my memory and learn the next increment.
How well do I know these areas? Some of them well enough that I think I could probably write a decent paper in the field, given the time. Many of them well enough to deliver a coherent introductory lecture, mostly from memory with some notes. None of them do I know as well as I’d like (or as well as true specialist), but again, that is one of the trade-offs I’ve made for breadth versus depth, which I’m only free to do as an amateur academic.

No Pressure or Competition

Professional academia can be a ruthlessly competitive arena. There are many more graduates than positions. Everyone in your field is trying to stake out their intellectual turf and achievements, and unless you come from money, most people starting out are strapped for cash.
To be able to succeed in this environment you need to be able to produce quality papers in some profusion and also work the circuit to develop contacts with and the favourable opinion of those in a position to advance your career.
To be honest, competitiveness is present in any professional environment, but the advantage to being, say, a starting programmer, is that you can make an acceptable living right of the gates, and, so long as you aren’t too obviously lousy, you can manage to build an okay career without being a star (except perhaps only in your own mind).
No matter how well you understand something or what great idea you have, writing a good paper that is not vulnerable to being shot down in flames within five minutes by a rival is a surprisingly difficult thing to do. Technical fields can be especially hard, since marshalling all the details so that they are both correct and convincing can be a monumental juggling task. People who can regularly pull it off are in my opinion under-lauded rock stars. (I won’t mention any names here, lest I start to gush and bore the last remaining reader to tears. ;-) )
This is a huge advantage of being an amateur academic: I only ever have to convince myself that I understand something (still rather challenging sometimes), rather than proving it to the world, and thus risk potentially making an ass of myself in the process (I still get to do that on the internet, so all is not lost. ;-)).

True Intellectual Freedom

The last benefit of being an amateur academic I want to discuss is also the only one that I think is a benefit to the world at large, not just for the amateurs themselves. It is the main reason that I would encourage someone to take this path for a reason that is other than just “to thine own self be true”.
As an amateur academic I’m not beholden to anyone for my learning or position in the academic world. I don’t know the players personally, nor have any tribal allegiance to one school of thought or another. If, after studying an area for some time, I decide that “the emperor has no clothes”, I don’t stand to lose any professional standing by abandoning the field. I risk nothing in breaking ranks from prevailing trends or fashions or taking on a heretical position that I sincerely hold.
As an amateur, I long ago realized that if I had any chance of making an original contribution to one of my fields of interest, it was going to be because I was freer to cross-pollinate disciplines that are normally considered distinct, or by calling BS on a field-wide group-think stemming from professional allegiances and lineages.
All disciplines (especially as they grow more specialized and esoteric) need more outsiders to take an interest in them, even if they dissent with their prevailing positions, and even if some of us who dissent are cranks.
For a professional academic, intellectual inquiry inevitably must be tinged with realpolitik, but an amateur academic is free to engage in a genuine dialectic of ideas, in the tradition of Socrates.
In any field of endeavour, to advance in skill and knowledge it is helpful to have good opponents, ideally people who share your values and interests, but with different methods and perspectives.
I think that the amateur academic is well suited to this role, and I hope that others will embrace it, and make a contribution to the intellectual health of our society, as I have tried to do in my own humble and questionably effective way.

On Being an Amateur Academic — Part 2

noreply@blogger.com (Marc Hamann) — Mon, 04 Apr 2011 15:06:00 +0000

The Cons of Being an Amateur Academic

There certainly are some drawbacks to being an amateur academic (as there are with any undertaking). I will discuss the three I think are the most important.

Takes Lots of Time with No Obvious Payoff

This is the most obvious problem with being an amateur academic. Keeping up with one field (let alone several as I do) requires a big time commitment that is hard to juggle with an unrelated (or distantly related) full-time job. To be fair, professional academics have this problem too, in the sense that they typically also have to teach courses, grade papers and assignments, serve in administrative positions and committees, write grant proposals, and a whole bunch of other tasks that don’t relate directly to their research interests in order to make their living. They probably still get a bit more time to do it during work hours and they also can get paid sabbaticals to be able to devote full time to a particular project.
Aside from the time and energy costs of full-time professional employment, the biggest challenge for the amateur in this regard may be explaining to friends and family why you spend so much time on something that doesn’t provide any remuneration or social standing. Most people will just think you are strange. (My family and friends are mostly pretty tolerant about this, but then again not all of them know how much time I spend on this stuff, and they have independent reasons to think I'm strange. ;-) )

Not Taken Seriously

In some of the fields I’m interested in (notably mathematics and linguistics) there are many honest-to-goodness cranks out there. You know, the kind that are trying to prove how some aspect of the field is related to alien visitations, or who have some pet racial or political theory they want to promote.
Especially in such fields, but also in most other academic fields, not having graduate and publication credentials in that field is a real handicap to being taken seriously, especially if you have modestly controversial opinions about some important topic.
On internet forums, it is easy to fall afoul of the academic-purity police and look like a crank. (I’ve noticed that this is particularly true if you post while sleep deprived ;-) ) By contrast, I could name some well-known academics with credentials left and right who I would argue are certifiable tinfoil-hats, but who routinely get a free pass from other posters because of their apparent standing.
If as an as an amateur academic you take it in your head to get published in respectable academic journals, good luck to you. Aside from the fact that you would be competing at a disadvantage with worthy graduate students and post-docs who desperately need to get published for professional advancement, and who have the more established sponsors to put in a good word for them, the easiest cut off for a time-strapped editor is to reject you based on lack of credentials and sponsors.
You can rail against the injustice of this, but many of the cranks take this approach too, so I wouldn’t recommend it.
Better to accept this limitation and, if you really want to make a contribution to a field, don’t worry about credit, but involve yourself in online communities of interest where you can explain your ideas. If someone steals them and advances their career with them, so much the better; you shouldn’t go into amateur academia for glory any more than for money.
I would also recommend being respectful of those with the credentials. They have given their blood, sweat and tears to their discipline, and have had to jump through hoops that an amateur academic hasn’t had to, so they are likely to know what they are talking about in some meaningful way, even if you disagree with them. This can be tricky, since nobody likes to be disagreed with by a nobody, and without credentials, that's how you look to them, at least initially. (You may want to try flattery as an opening gambit. ;-))

Lack of an Intellectual Community

Professional academics have colleagues, rivals, advisors, conferences, and other means to form communities of shared interest in the same field. The amateur academic must resort to regaling bored but tolerant spouses and friends with their latest fascination.
Personally, I think this is the greatest hardship of being an amateur academic, since it is a passion for learning that drives you to be one, and any passion wants to be shared.
The internet helps with this, since there are all kinds of blogs, mailing lists and other forums that can be used to connect the like-minded, but my own experience is mixed here. It is hard to find a community that is at exactly the right level for whatever state of mastery you have reached, and in many forums people with less knowledge resent the explanations they get of how they are off-base, no matter how gently presented, and people with more knowledge, or more credentials often resent hearing from the “peanut gallery” about their specialty (see the previous section).
The amateur academic can be a lonely fellow. (But hey, you wouldn’t spend so much time in solo study if you couldn’t handle being alone, now, would you?)

On Being an Amateur Academic — Part 1

noreply@blogger.com (Marc Hamann) — Mon, 04 Apr 2011 14:42:00 +0000

When I started this blog, I decided to add the description “amateur academic” after “professional software developer” in my "About Me". I thought it was a useful addition to explain not only the range of stuff that I intended to talk about, but also to explain my activities around the web for people who Google me and are trying to figure out why I’m posting somewhere about some pretty esoteric topics that are not obviously related to get-it-out-the-door software development.

What is an “amateur academic”?

Since the very concept of “amateur academic” will be unfamiliar to many, perhaps even oxymoronic, or maybe even quaint in a 19th-century English gentleman sort of way, I’ve always meant to explain what I mean by this description. More importantly, I also want to share my experience of the pros and cons of being an amateur academic with others of like persuasion. (I think there are more of us than you might think, especially among technically inclined people.)

The term itself is, I hope, a self-explanatory composition of the two individual words that make it up.

“Academic” means that I am interested in topics that most people consider esoteric, theoretical, and in the sole domain of professors. It means I spend a significant amount of my time reading books, papers and online materials related to these topics. Most of the latter are in fact the professional academic literature of those fields of study, and are produced by the professors and grad students we normally identify with the term “academic”.

“Amateur” is an even simpler term to explain in this context: it just means that I study these topics and materials only for fun rather than with the intention to also make a living at it.

Are you crazy?

Studying purely for fun is likely to make me seem off my rocker to lay people and professional academics alike, neither of whom may be able to imagine how such challenging material could be enjoyable to work through without the incentive of remuneration. (Graduate students, on the other hand, may relate to this more directly, though many of them may still be optimists who have hopes of cashing in on their knowledge some day.)

Whether I’m off my rocker or not, this is how I am, and I long ago accepted it and have been making life choices that make my proclivities possible. Aside from the existence of the internet, with its multitude of resources and public access to material and forums which used to be private and hidden, the most important ingredient to succeeding as an amateur academic is to be realistic about the pros and cons. For this reason, I will expand on these pros and cons for the next couple posts.

On Reviewing Old Code

noreply@blogger.com (Marc Hamann) — Mon, 13 Dec 2010 01:08:00 +0000

I recently committed an old programming project to Github.

The project is several years old – older than Java 5 at least, since I had to add some generics to the code to get rid of some compiler nags before committing it.

It was one of my early TDD exercises. It was the last iteration of a quest I was on to develop a realistic world map generator that could be used, for example, in games such as Civilization or for any other purpose that might call for fictional worlds. Over the years I had tried several different algorithmic approaches and used several different programming languages: C, C++, Visual Basic, and finally Java.

When I was about to take a look at the code after all these years, I felt some trepidation. I’ve downloaded many open-source projects and I’m used to the shock that often accompanies the first peek at unfamiliar code. Clean code is not a universal value, and sometimes beautiful code is in the eye of the beholder. This code for the map generator was old enough to have become somewhat unfamiliar, and I was prepared for the worst.

On the whole, I was relieved to find that the code was reasonably clean. My experience of reading clean code is that you feel a kind of disbelief that code that seems so simple could be producing such complex behaviour. Well-factored, clean code looks, paradoxically, unimpressive because it is so easy to follow. To reverse a questionable old adage, it was harder to write so it could be easier to read.

What I saw was certainly not perfect. There were many conventions regarding unit testing that I evidently had not yet established for myself at the time the program was written, and it could still use some even more aggressive refactoring. As I recall, I simply ran out of steam toward the end of the project, so it’s possible I knew about these problems at the time but didn’t get around to fixing them.

The biggest thing that struck me upon reviewing this project was how far away it seemed. I had obviously been obsessed with map generation for many years, to the point of learning some GIS techniques and using it as a guinea pig project for new languages and new techniques. Even though I’m pretty self-motivated, in the end I ran out of steam. In the absence of a user community for this project, it became impossible to rationalize the effort anymore.

No matter how independent-minded I am, no matter how internal the art of programming is, in the end it seems that software is all about fulfilling the needs of others.

The Usefulness of Philosophy

noreply@blogger.com (Marc Hamann) — Wed, 17 Nov 2010 18:36:00 +0000

I came across a comment on a blog recently where the author adjured the other participants to stop “philosophizing” so much and get down to the real matter at hand. This reminded me that, for many people, “philosophy” is a term of abuse. For them, philosophy is pointless blather among elitist twits with no practical consequence - the very opposite of anything practical and useful.

Given the name of this blog, you can guess that I don’t agree with this negative sentiment. I won’t deny there are some philosophers and some philosophies that I think are pointless blather, but to take them as our basic definition is to throw the baby out with the bath water.

I want to propose that philosophy is the study of mental models, and, as I said in my previous post, I think mental models are the basis of our competence. Since our competence determines how well we manage and how effective we are at realizing our goals, there is obvious practical importance in understanding how mental models work, getting used to taking them apart and building new ones.

As I explained in my very first post regarding my chosen name for the blog, I think software development is an eminently philosophical activity. It is all about constructing mental models of systems and manipulating those systems using the mental models. It doesn’t matter whether these systems are machines, protocols, teams, problem domains, programming languages, etc: how effectively you work with them depends on your ability to construct and manipulate good mental models of how they work.

Being unaware of your own mental models is a limit to your own effectiveness. We have all known people who thought they had found the perfect hammer and were busy nailing everything. Likewise, we have probably known someone who repeated the same dysfunctional pattern over and over again in spite of not getting the desired result.

Philosophy as the study of mental models can make you aware of the mental models underlying these behaviours and can give you the skills you need to improve them.

A word of warning though: as Socrates found out the hard way, people often get very upset when you question their cherished mental models, and this can happen even when we question our own. However, if you want to improve effectiveness and grow in competence, there is much to recommend the use of philosophy to take apart and rebuild our mental models.

The Importance of Mental Models

noreply@blogger.com (Marc Hamann) — Mon, 15 Nov 2010 00:49:00 +0000

Many years ago, my wife and I were in Paris, staying in a quaint Left Bank hotel that did not provide an iron and ironing board. My mental model of a hotel is that it should provide these, free of charge, and preferably one to each room: I like having wrinkle-free clothes, even on holiday. However, as we will see, I’ve learned that my mental models are not always adequate representations of reality, and that in order to solve my problems, I may have to construct a new mental model. We decided that the solution to our problem was to buy a compact travel iron to solve the problem once and for all.

To Find an Iron in Paris: How Hard Can It Be?

At home in Toronto, the obvious place to buy a small electrical appliance would be at a large department store, so we thought that the first place to look for our quarry would be at a well-known Parisian department store a healthy walk from our hotel. When we got there, we wandered around a bit, but couldn’t see an appliances department, so we asked a saleslady where we could find such a thing. (We are both fluent in French, so we had a leg up on most tourists in such a situation.)

The saleslady was polite and helpful, but bewildered that we would be looking for an iron in her store: in her mind this clearly wasn’t the kind of place you shopped for such things. It was as if I had gone into a sporting goods store and asked if they had a fresh produce section.

Another faulty mental model: in spite of being two adults with years of experience fending for ourselves, able to speak the local language, familiar with Paris from previous trips, it dawns on us that we are simply lacking the competence to perform the simple task of buying a travel iron in Paris.

What the Heck Is “Darty”?

Luckily, when we asked our friendly saleslady where we might purchase an iron nearby, she offered one word: “Darty.” We weren’t sure if this was the name of a street, a neighbourhood, a local shop-keeper or a store, but she pointed us vaguely up the street, and off we went. Along the way, we found a small shop whose sign indicated that they sold electrical supplies, and we thought this might be what we were looking for, but in fact this store only sold light bulbs of every shape and variety, all stored in rows of wooden drawers mounted like a library’s old card-catalog on the walls. My mental model of the world did not contain the possibility of such a store, so I was mystified and delighted: if I ever need a light bulb in Paris, I will now know where to go.

After wandering around in circles through the figure eight streets, repeatedly asking reservedly helpful passersby for directions, and being vaguely pointed, sometimes in contradictory directions, we finally found a fairly large store, set back from the street with a sign proclaiming it to be “Darty”. At last!

Making a Purchase: What Could Be Easier?

Once we entered the store, it was clear we had found the right place. There were rows and rows of various kinds of home appliances and electrical gadgets. Not all of the logical groupings were readily apparent to me: for example there might be electric fans and electric razors in the same shelving island. Each item had a single display model, out of box, on a shelf with a small card with a number next to it. After some wandering around, we did find a small row of irons, one of which was a compact travel iron.

Now for our next challenge: how to buy one? There were no boxed models to pick up and take to the sales counter, just the floor model and the little number. We stood their looking and feeling clueless for a while, until a saleslady spotted us and asked if we needed help. Saved! Now, we thought, she will get us our item, process our transaction and our quest will be over.

We indicated to her the travel iron we had selected. She took out a little paper form, filled out the number of our item on it, handed it to us, and cheerfully bid us good day. Another perfectly good mental model crushed by a cruel Gallic world! I sheepishly asked where I was supposed to take the form. She pointed vaguely across the store, saying there was a counter.

Sure enough, we crossed the store and found a counter with several clerks standing around. We handed one of them our form, they processed our payment, gave us a new stamped form, and bid us a somewhat perfunctory good day. I waited for a moment, expecting our iron to appear, in spite of the fact that our clerk seemed to have completely lost interest in us. After a few moments, I asked where my iron was. They pointed vaguely in a new direction across the store, and we traipsed off, ending up among rows of televisions sets.

The sales guy there took pity on us, despite my mild irritation that I had already paid for my iron, but did not yet have it in my hand (another mental model). He explained that we actually had to leave the store, and walk half-way down the hall of the indoor mall it was in and we would be able to get our iron there. I’m starting to feel like a sucker: they’ve taken my money, but they are now telling me to leave the store with just a little stamped form and someone down the way will give me my item? Riiiight! Nonetheless, we followed his instructions, finding a little kiosk down the way, unmarked and unattended. After a few moments standing there, dejected and simmering, an attendant appeared, took our stamped form, and handed us our boxed travel iron. At last!

The Joys of Travelling

Now you could take this story as a mockery of the French way of doing things, but the American tourists I’ve seen loudly berating French workers for their bad customer service have got that angle covered.
In fact, I love these kinds of experiences, though they can be distressing at the time, and they are exactly part of the reason I travel. I want to have my mental models challenged by a different culture.

There is a perfectly good system at work there that Parisians navigate every day, I just didn’t understand it, but now I do, and having done so, I’m now just a little bit more competent at how to get needful things done in Paris.

Working with Systems Is Working with Mental Models

Though you could just read this as an amusing anecdote about travel, my real purpose in telling it is to apply it to thinking about useful systems. Doing software development, or managing a team, or running a business are all about navigating, manipulating and improving systems. To work with a system effectively, you need to have a good mental model of that system.

If you want to lead a team, one of your biggest challenges is to communicate your mental model of the undertaking you want the team to pursue. Only if all the members of the team share a mental model which is adequate to the task at hand can they work together to produced the desired end.

In fact, I would go so far as to say that to call yourself competent at some skill is to say that you have acquired an adequate mental model of that skill.

Humility Is the Path to Competence

As the iron-buying story illustrates, it can be frustrating and humiliating to be confronted by a foreign mental model. We are comfortable considering ourselves to be competent adults, knowing how to do things in the world, and being confronted with our own incompetence in the face of an unknown mental model can be painful.

This often leads us to disparage the people that have that mental model. For example, I’ve seen tech teams and sales teams run each other down behind their backs, each side thinking that what they do is complex and valuable, and what the other does is simple or over-valued. The fact is that each has invested a lot into understanding a complex mental model that underlies their respective competence, and it is easier to run down the other’s mental model than to accept their own lack of competence in the other’s domain.

My experience is that if I’m not getting the results I want, or if I’m having trouble communicating with someone else, the challenge is to discover the right mental model to make me competent at that task. The necessary ingredient, sometimes hard to practice, is the humility to abandon the comfortable mental model I’ve already mastered to be able to absorb the new mental model at which I am just a clueless newbie. Only by accepting my incompetence can I find the road to competence.

Captain Kirk Was a Lousy Tech Manager

noreply@blogger.com (Marc Hamann) — Mon, 25 Oct 2010 23:52:00 +0000

Those of you who remember the original Star Trek series will remember the running trope of Captain Kirk asking Scotty how long something will take, and when Scotty responds something like “two days”, Kirk would respond with “You have two hours”, or some other ludicrously short period of time. Scotty would shake his head exasperatedly and go off to spin straw into gold (always making the deadline), while Kirk would get a smug, self-satisfied “There’s brilliant leadership at work” look on his face to let us know what a genius commander he was.

Years later on Star Trek: the Next Generation, Scotty made a guest appearance and confided to the engineer that he should never tell how long it would really take to do something: it turns out that the result of Kirk’s management style was to train Scotty to game the system.

Now sometimes good leadership requires pushing team members to pursue “stretch goals,” so that they continue to grow professionally and stay engaged with their jobs. But to make asking for the impossible into a routine part of every task assignment is just a bad idea. Scotty shows us why: it backfires, since the team member learns that honest estimations are punished.

The commanders in the next-generation Star Trek shows, Captains Picard, Sisko and Janeway, had much better leadership skills in this regard. They encouraged honest estimates and had open and respectful discussions with their team members about priorities and deadlines. It’s hard enough dealing with alien invasions and other calamities without introducing dysfunctional group dynamics into your own team through “heroic” Captain Kirk-style management.

The Diabolical Genius of C++

noreply@blogger.com (Marc Hamann) — Tue, 27 Jul 2010 02:37:00 +0000

I have been feeling a strange pull to revisit C++ lately. Partly this is because C++ continues to be the language of choice for certain domains such as games and graphics programming that I find interesting. But more importantly, I have been curious to see what I would make of C++ if I took a fresh look at it all these years later, with the benefit of all the practical and theoretical expertise in programming and programming languages that I have acquired in the meantime.

I remember when the first edition of Effective C++ by Scott Meyers came out, and I was very tempted to buy it, but in those days computer books were ridiculously expensive, and I bought Bjarne Stroustrup’s equally fresh-off-the-press The C++ Programming Language 2nd edition instead, on the logic that it was the official reference and so its utility would stand the test of time. (Something you couldn’t count on with most technical books of the time.)

So given that Meyers’ book is still in print (in its 3rd edition) almost 20 years later, I figured it was the best place to go to reacquaint myself with C++.

Overall, I found Effective C++ to be a good book with good advice (though I might quibble here or there) and an excellent reminder of what programming C++ is like. What I rediscovered was that C++ is both a paragon and abomination of programming language design. It is in a way a poster child for the title of my blog, “Philosophy Made Manifest”, in the sense that it is so exquisitely the logical outcome of its philosophical premises. More particularly, it is the synthesis of two, quite different philosophies of software construction, and the extent to which it is a paragon or abomination is a direct result of the relative compatibility and incompatibility of these two different paradigms , in the Thomas Kuhn, The Structure of Scientific Revolutions sense.

To give you a sense of what these two philosophies are like, I can use my own early development as a programmer as an example. Like many programmers of my generation, my first language was a flavour of BASIC. BASIC was a good language to get your feet wet with programming, but too much of it was “magic”, in the sense that it buffered you from the real workings of the machine. It was fine for relatively simple programs, but once you got to more complex applications on a machine with memory measured in kilobytes, it didn’t really give you enough awareness or control of your environment to manage your resources.

The next step up from there was often assembly code or even raw machine language, and that looked like alphanumeric gibberish rather than comprehensible language.

So when I discovered C, it was a revelation. Here was a language that had a comprehensible syntax like BASIC but that really allowed you to specify exactly how you were using your resources. By this time, I had 1MB of RAM and clock speeds in the MHz, which seemed like a lot at the time, but for some of the applications I was interested in, you still had to juggle your resources to make this work, and C let you do that. With C, I went from being a dabbler in programming to being a programmer.

Moreover, C helped to give me an entrée into the world of assembly. The C compiler I used allowed me to generate assembly code as output, and I became very familiar with how my C code was translated to the machine. Sometimes I even wrote super-optimized functions in assembly for maximum speed and efficiency and called them from C.

But this was the first sign of trouble in paradise. Two things became apparent to me around this time.

The first was that the “C with assembly” approach wasn’t very portable or maintainable. Different versions of the 80x86 architecture, of DOS (and soon Windows), and of the compiler and associated libraries, made work done this way very fragile.

The second was that the low-level approach of C was very awkward for higher-level tasks such as GUI design, where what you wanted was reusable components that interacted on an event model. And this is the locus of the paradigm shift: from programmer as juggler of resources on a machine to programmer as designer of solutions in the problem space.

I think this dichotomy is a major one in software development, and it is not the exclusive domain of the C/C++ world. I’ve seen it play out in the Java world too.

Many technical people are (reasonably enough) oriented towards the technical side of things. They’re focused on the solution space. They have staked their professional mastery on understanding the arcane details of various languages, platforms, libraries and tools. This leads to a natural tendency to believe that the role of the programmer is to redefine the problem until it fits the bounds of the available solutions. They seek to turn the problem at hand into the proverbial nail for the hammer they have.

This is not a wholly bad phenomenon. In fact, in the kind of resource poor environment my C-programming self was contending with, this kind of shoe-horning was a necessity for being able to accomplish anything of value. And since resources are never completely unlimited, some of this thinking always ends up being essential on a technical project of any scope.

But the world changed as more computing resources became widely available, and richer, more user-friendly GUIs and application came to be the norm. Customers for software products and the software teams that built them started to want a more problem-focused approach to software. Instead of reworking the problem to fit the technical solution, there was more value in squeezing the technical solution into the mould of a more natural, conceptual model of the problem space being addressed.

And this is where Object-Oriented Programming (OOP) came in. The classes and objects that were the ++ to C gave developers a new set of abstractions to capture such a model in the source code. In principle, OOP was supposed to remove the focus from the implementation details of the application and place it on the modelling of the entities and activities associated with what the user wanted the application to do. Again, in principle, the programmer of an OOP language was supposed to cede control over some of the low-level resource allocation issues to the language implementation, and focus on the world according to the user. Some OOP languages did go quite a way down this road.

C++, however, made a different choice. It decided to try to fulfill both paradigms. Want to micro-manage resource usage? No problem: C++ includes C. Want to work at the higher-level abstraction of OOP? No problem: C++ has all the OOP features you want.

In some ways, C++ has been wildly successful in its goals. It succeeds in having all the features of both so that the developer is free to choose his approach. But it is exactly this blend that makes it such a nightmare from a design perspective.

As I was reading Meyers’ book, I was struck by how often he says of some C++ feature “there is a very simple rule for this in C++, with a few specific exceptions”, and the exceptions turn out to be mind melting and highly unintuitive to someone trying to avail himself of the “high-level” approach to their application. The language is designed to “help” you by doing certain things automatically, but it often doesn’t do the thing that seems to be most reasonable, since it doesn’t want to conflict with the freedom of the programmer to operate at a lower-level of control.

I think this illustrates a general principle of design: simple solutions can only exist for focused design philosophies. When you try to be “all things to all people” you necessarily end up with complicated, hard-to-manage solutions.

Having identified the “original sin” of C++, I think we still need to give it its due. Decades later, it is still going strong in application domains such as bleeding-edge games and graphics and in device-embedded controllers – domains where the spirit of scarce resources is still alive and well, and where the need exists for both the large-scale organizing principles of OOP and the “down-to-the-bare metal” optimization of resources.

Given that the pendulum in popular programming languages has swung back to the spirit of BASIC (interpreted languages that are “fun” and where resource allocation is mostly “magic”), I wonder if the end of Moore’s Law spells a return to the spirit of C++. If so, any successor to C++ will have to start where it left off and learn from both the bad and the good of its diabolical genius.

Lessons from Confucius for Software Development

noreply@blogger.com (Marc Hamann) — Tue, 16 Feb 2010 18:56:00 +0000

In my previous post, I talked about lessons from Confucius that I think are still useful in the modern world, but there is a field of endeavour that I think can particularly benefit from understanding the Confucian worldview: software development.

At first glance, it seems highly unlikely that Confucianism might have some application to software development. The Confucian values of respect for the past, harmonious social relations, decorum and moral leadership don’t have an obvious affinity for an industry that is known for its focus on what is new and shiny, and which is stereotypically populated by raging individualists with a disregard for social standards.

But it is worth considering that the Confucians represent one of the earliest groups of knowledge workers in human history. They were trained in specialized knowledge for specialized tasks in a complex society, and they had a sense that their specialized knowledge made them an elite group in society.

An important concept in Confucian texts that bears on this is junzi. Etymologically, it means “ruler’s son, prince”, but already in Confucius’ time it is used more metaphorically, often translated into English as “superior man”, “gentleman”. The Yiddish term “mensch” has a similar ring. It represents what every Confucian was striving to be.

In spite of its etymology, junzi is one of the earliest-known notions of elite status that is not derived from the happenstance of one’s birth, such as being the literal son of a ruler, or a free man of Athens. The kind of “superior man” we are talking about here is defined entirely by his knowledge and skills, and his savoir faire in using them. Confucianism is a truly meritocratic system of thought, and a modern IT specialist can easily relate to such an ethic.

A second consideration is the fundamentally social nature of software development. If it ever truly existed, the age of the lone genius changing the world with his software is over. Any non-trivial software development these days involves a whole team of people with different specialties and knowledge, and building reliable and maintainable applications requires that these people work together as an effective and harmonious community.

The Confucian junzi has a sense of noblesse oblige, a sense that the status conferred on him by his knowledge and skills requires that he use them for the betterment of his society and to achieve collective goals. He is willing to lead and mentor new members of the fraternity of knowledge workers, and does this not by pontificating, but by providing an example through good practices.

The very fact that these kind of values are not what most of us think of when we consider software development suggests that there is still much benefit and advantage to be gained by nurturing them in software teams, and a team lead could do worse than to study the Analects of Confucius to prepare for the challenges they face.

After all, Confucius and his followers have several centuries of experience organizing and training knowledge workers to draw on.

Lessons from Confucius

noreply@blogger.com (Marc Hamann) — Fri, 15 Jan 2010 18:53:00 +0000

When I was in university studying East Asian studies, it was the height of the political correctness era. Of the three major streams of Chinese “religious” thought — Confucianism, Taoism and Buddhism — Confucianism was considered to be the “bad guy”, representing everything that is authoritarian, sexist and hierarchical in Chinese culture.

By contrast, everyone loved the individualistic enthusiasm of Taoism, which extolled the virtues of the feminine, or the egalitarian austerity of Buddhism. Confucians, though, were the “dead white men” of Chinese studies.

So, while I got an early grounding in Taoist and Buddhist thought, it wasn’t until I had been in the work-world for some years that I came back to Confucius, starting by reading the Analects in translation, and later, in the original.

The Confucius I found was quite different from my pre-conceived understanding of him, and I was surprised to find that his outlook and advice were unexpectedly relevant to modern life. I also found an attitude quite different from the stern authoritarian stereotype I had previously accepted.

In fact, my surprise started with the first line of the Analects (translations are my own):

To learn something, and to review it now and again, isn’t it pleasurable?

I had, of course, learned as a student about the traditional Chinese respect for learning, but I had been left with the impression that the kind of learning that was meant was rote memorization in strict conformance to orthodox interpretation. But this primary source of Confucius’ personal thought starts off with a child-like enthusiasm for the simple pleasure of learning, a feeling I knew well. This attitude struck me as even more relevant for our modern world, where there is so much to learn and where knowledge and skills are the currency of our society.

Another traditional value of the Confucians I had learned about in my student days was often translated as “ritual”, but it seems to encompass politeness, observance of correct social forms and social hierarchy, as well as what we would think of as actual rituals. We used to roll our eyes at this, since we tended to think that these things are intended to suppress our sincere, individual feelings – that they are empty formalities intended to ensure conformity. But in another line from the Analects, I got another surprise:

Lin Fang asked about the basics of ritual. Confucius said: A big question! In ritual, prefer modesty to extravagance; in mourning, prefer sincere sadness to formality.

Confucius is genuinely concerned with the observance of social forms (and the social forms of his time and place sometimes seem very foreign to us), but he doesn’t recommend them as a replacement for individual feeling, but rather as a vehicle to allow the free expression of individual feeling in harmony with the functioning of the community.

This sense of communal harmony is underlined by the cardinal virtue of Confucian thought, often translated as “benevolence”. It is related to the Chinese word for “person”, and I can’t help but feel that the best modern equivalent for it is the Yiddish word “mensch”. Etymologically, “mensch” means “person”, but its full meaning is someone who has a strong sense of community, someone who can be relied upon to help others and who has the courage to stand up for what is right. This is a pretty close match with the Confucian principle.

Confucius is also very concerned with good leadership, and a recurring preoccupation of Confucians (and ancient Chinese thinkers in general) is how to be an effective ruler. Here is another representative passage from the Analects:

Ji Kang asked: How can the people be made to respect the ruler, to be loyal and take his advice? Confucius said: Ruling them with solemnity will result in respect; showing respect for elders and being kind will result in loyalty; promoting good and instructing the unskilled will result in persuasion.

Contrary to my early stereotype, Confucius has no time for the “because I said so” school of leadership. This is all the more remarkable when you consider that rulers in ancient China did have absolute power. His prescription for leadership is firmly in the “lead by example” camp. Those millennia ago, Confucius had already recognized that people can tell the difference between a cynical, self-serving leader and one who genuinely has the collective good at heart. He knew that the difference between genuine commitment and mere grudging compliance rests on this distinction.

So, my re-examination of Confucian thought not only changed my mind about its essence but actually showed me that there were values and lessons to be learned that I could apply to my life, personally and professionally.

A genuine love of learning, accepting social forms as ways of collectively expressing individual feeling, cultivating sincere feelings of communality with others, and leading by moral example: all of these ideas are still relevant in the 21^st century and, if practiced, can make a real difference in one’s life as a leader and as a human being.

Creativity is a Balance of Freedom and Control

noreply@blogger.com (Marc Hamann) — Wed, 30 Sep 2009 21:48:00 +0000

About a year ago, I took up playing a middle-eastern instrument called the oud.

One of the characteristic forms of music for the oud is called a taqsim , which is a short improvisation. I’ve always loved improvisational musical styles, and this was part of the draw of the oud for me.

Improvisational music can be a very pure form of spontaneous creativity, and when it is going well, it gives a powerful feeling of freedom and self-expression.

Many people who don’t play an instrument or who play an instrument only by reading sheet music often ask me how one can possibly make up music on the fly, and I always tell them it is easier than it sounds.

A common misconception is that to be creative means that you throw all rules and restraints out the window, pulling great stuff out of thin air, as if by magic. Many people think that creativity is somehow inherently chaotic and random. But a simple example can show that this is false.

Have you ever listened to a small child with no musical training hammer away at a piano? Based on the “creativity is chaos” misconception, this should be the height of creative expression, since the child is following no structures or rules aside from the physical limits of the piano. But most people would agree that the result is not music, but rather irritating noise.

The first thing that many piano methods do with our budding creative genius is to teach the scales. This is because these are the basic structures that underpin western music. To be either a competent listener or a competent player of western-style music begins by internalizing the sound landscape that is defined by the diatonic (or pentatonic) scales and the melodic and harmonic palette they imply.

After learning even one of the scales, our enriched child could hammer away the same as before, but restrict the hammering to the notes of that scale. Because there is some musical structure, most people would judge the result as more musical, and in that sense morecreative.

This is the trick with improvisation: you aren’t really making up something from scratch. All common forms of musical improvisation I’ve ever studied (Indian classical raga , the blues , Middle Eastern maqamat and Jazz , for example) actually have a lot of structure to begin with: there are specific scales, rhythms, sequences and patterns that have to be learned and internalized or else the result won’t sound like music, or at least not the kind of music you are trying to play.

Once the player (or the listener) has internalized all the rules and structure, then freedom from rules comes back into play. If you have too many rules followed too rigidly, the results become boring and predictable and the music loses its vitality. Great improvisers know where their freedom lies and know where and how to break the rules to produce something fresh and unexpected, and it is this quality that can make improvised music so vital and exciting for players and listeners. So-so improvisers sometimes fall flat on their face by going too far away from the basic structure and have to scurry back to it. But even this freedom to fail is part of the process.

What is true for music seems to be true for other domains with complex structure that require creative solutions: software development, management, politics and economics, for example.

In any of these domains, too few rules results in inharmonious chaos, and too many stifles inventive and effective approaches.

Any approach to these problem areas that looks only at adding regulation or only at removing regulation is probably missing the boat.

The real question always is: “How do you plan to balance freedom and control to maximize harmonious creativity?” Any philosophy of these domains that can’t answer this question is probably not worth considering.

Why proving that P ≠ NP is hard

noreply@blogger.com (Marc Hamann) — Tue, 07 Jul 2009 12:43:00 +0000

A while ago I spent the time to “audit” online a Cambridge University course given by Tim Gowers on computational complexity.

In this course, Gowers takes the students through a proof by Razborov that there is no “natural proof”, for a particular form of the problem, that P ≠ NP, which I found very interesting.

This got me thinking about why this is a hard problem to solve, and I thought it might be useful to explain the intuitive reasons behind it.

So let’s think about this by looking at the Traveling Salesman Problem, since it is a fairly intuitive example of one of the hardest problems in the NP complexity class, known as NP-complete.

In this problem, you have to find the shortest path through a number of cities, visiting each city once and only once. This is a typical NP problem, where the naïve approach to solving the problem would be to just work through all the combinations possible, check the value of that combination (in this case the length), and pick the shortest one.

The problem with that is that the number of combinations gets very large very fast, so it really isn’t practical to do this.

Let me show you what I mean. Let’s take the case where there are 5 cities: A, B, C, D, E. Any path through these cities can be described by a sequence of these letters. For example, CDEAB would be the sequence starting at city C, then going on to city D, etc. To figure out how many combinations there are, I have to use each of the letters once and only once, so there are 5 possible choices for the first slot, four for the second, and so on. This gives me 5x4x3x2x1=5!=120.

In general, this brute force approach to solving the problem, which represents the worst case scenario among solutions, will take n! steps, where n is the number of cities we are considering. This gets very big, very fast. For example 10! is equal to 3628800, so you can see that as you add cities it becomes vastly harder to solve the problem this way.

Now it is very important to realize that this is the worst-case scenario only, or as it is described in mathematics, an “upper bound” on how hard the problem is. For example, if all the cities are in a straight line, you can solve the problem in many fewer steps: just start on one end and move to the next in line and so on
This works because we know that on a straight line, the shortest path will be on that line, so that if the cities are aligned ABCDE, we know right away that we aren’t going to have to consider any of the combinations that don’t follow the sequence on the line. In other words, there is a pattern to the data that allows us to ignore a number of combinations (in this case, all but the two, which are of equal length: ABCDE, and EDCBA).

But to say that some problem is in the class NP (and not in P) is essentially to say that there will always be some collection of cities you could choose that would require you to go through the worst case scenario to find the shortest one. (Technically, the actual worst case scenario for this problem is only exponential in complexity, but that is still pretty bad and doesn’t affect my argument here.)

That is, unless someone can prove that P=NP. This would mean that there was always a shortcut that allowed you to find the solution without having to work through the majority of the possible combinations. And this shortcut would almost certainly involve being able to consistently find some pattern in the data, just as we did with the straight line example, that simplifies the data down to a smaller number of cases.

So in a sense, it should be much easier to prove P=NP than that P ≠ NP. (That no one has proven the former yet is a pretty good reason to suspect that the latter is true, even though no one has proved it either.)

The reason is that to prove P=NP, you just need to show that there is always a pattern in the data that will simplify it, whereas to prove P ≠ NP, you would have to show that there can’t be such a pattern for some scenarios in the problem set.

Why is it easier to show that there is a pattern than that there can’t be a pattern? Well, if you have a pattern, you have a relatively small number of steps needed to describe that pattern, and it is much easer to work through those steps and convince yourself that you have covered all the bases. You can also try your pattern on all the hairiest known instances of the problem and show that they work.

In a sense, having a pattern simplifies the search space required for the proof in exactly the same way that having the pattern simplifies the search space to find the answer to the problem.

But how do you show that there is no possible pattern for some set of data?

Let’s say that I give you a list of digits and tell you that one of two scenarios generated these digits: either I am generating them randomly for each digit as you ask for them, or I am using some relatively simple-to-genarate pattern to generate them. Here is a sample list:

15926535897932384626433832795…

Can you see a pattern? You could do a statistical test on the digits to see if they show the properties of randomness, and in this case, I think you would find that they do show those properties.

So, can you then conclude that they are random?

No, unfortunately not, since there is a pattern here: these are the digits of pi after the well-known first three 3.14…
So the challenge is that you can never really be sure that there is no pattern to some data set; it could just be a really, really unobvious pattern that you haven’t thought of yet.

So you can see how very hard it would be to prove that there is always some configuration of cities that can’t be simplified by some pattern, which is only part of what you would have to do to prove that P ≠ NP.

This is why it is unlikely that anyone will find such a proof unless they can develop some hard-to-imagine technique that allows us to detect when there aren’t any patterns in a set of data.

What problem are you trying to solve?

noreply@blogger.com (Marc Hamann) — Sun, 14 Jun 2009 17:04:00 +0000

Anyone who has worked in IT for any period of time is going to be familiar with the dilemma of choosing one particular solution over another. Should I learn language X or language Y? Should I use framework A or framework B for my next project? Should I use the Foo Architecture or the Bar Architecture? Is the Widget application better than the Grommet application?

Since sometimes it seems that the choice you make can have big costs in terms of your project’s success (and your future employability), these dilemmas can result in a lot of stress. It doesn’t help that for each solution you have some proponent telling you that if you don’t use their approach you are stupid and ugly (see Linus Torvalds talking about his source control tool ).

So what’s the smart way to decide?

I think the first step is to realize that you have the wrong question. Technology people by definition tend to be interested in technology (tools and solutions) and we tend, as a result, to give short shrift to the problems themselves, especially when they fall outside our domain. As a result, solutions pile up faster than you can evaluate them, and problems often get shoehorned in a pet solution. So what is the right question?

Do you remember those word problems we used to have to do in school? Trains leaving the station at certain times and speeds, Mary and Bill trying to divide up a pie, and all those hypothetical problems?

The secret of those problems was often just figuring out what the problem was in the simplest terms. Once you rephrased the question, you often knew exactly which technique from that week’s lesson to apply to solve it. You could just whack the problem with all the techniques you had learned, hoping to hit the right one, but you would probably take too much time or might even get an “answer” that was totally off base.

To make this brute force approach even less likely to succeed, the teachers and textbook sometimes threw in irrelevant details that sounded important but had nothing to do with the real solution, in the same way that mystery writers throw in red herrings to make it harder to guess whodunit.

So the only reliable way to solve these problems correctly was to learn to filter out the extraneous blather and find the pithiest answer to the question “What problem am I really trying to solve?”

So how can we use this to solve our solutions dilemma? Let’s start by recognizing that the real problem is not what solution to choose but rather resolving the problem the solution is for. Trying to choose a tool without explicitly defining the real problem is exactly like trying to solve those word problems by throwing all the techniques you know at them. You may get lucky, but more likely you will make a mess.

If you realize, having thought about it, that the problem you are trying to solve is to avoid having others think you are ugly and stupid, then there is an easy solution to the problem: give up on technology altogether. In technology there is always someone telling you are ugly and stupid because you didn’t choose the same solution as them, so you might as well get used to it. If you want to go one step further, reject all forms of ideological thinking on principle, and craft your own rational set of criteria by which to judge things.

If you discover instead that there is some substantive problem you want to solve, spend some time trying to understand the problem. Ask yourself what the essential characteristics of the problem are. Identify some things that might normally be of interest but don’t really apply in this case or have less importance than normal. The truth is that no solution is perfect, all solutions require trade offs, and the secret of success is focusing on the must-haves while sacrificing the merely nice-to-haves or, more often, even the not-quite-so-must-haves.

Sometimes I find it very useful to start sketching out my own solution from scratch. At some point you realize either that you have a nice simple solution and don’t need to go any further, or that, now that you have a better feel for the problem and its challenges, that you have clear criteria upon which to base the choice of an external solution.

Anyone who wants to get good at software architecture has to be willing to throw out conventional wisdom and make choices that fit the problem at hand, not the problem that advocates of particular solutions tell them they should have. Sometimes just spending some quality time with your specific problem clarifies what you really need and melts the solutions dilemma away.

Functional Programming is Not All or Nothing

noreply@blogger.com (Marc Hamann) — Thu, 28 May 2009 12:47:00 +0000

I’ve noticed a flurry of discussion around the blogosphere lately about what is and is not a functional programming language and what is or is not stateful. After being interested in functional programming (FP) for many years, I’m glad that it is finally getting some mainstream attention. I think FP has some valuable lessons for all programmers, and that it provides a better default mindset for programming than traditional imperative programming.

However, a lot of the discussion seems to make the mistake of assuming that functional programming and statefulness are all-or-nothing properties, that either you have a total purity of statelessness or you have opened Pandora’s Box and let all the evils of state into your program or programming language. But the truth is that functional programming is best viewed as a style of programming with different default assumptions about state. And statefulness is best viewed as a continuum that varies along at least two dimensions — locality and monotonicity — both of which I will explain shortly.

What is state?

Before I do that, I had best describe in simple terms what state is. The easiest way to understand state is to imagine that I have given you a distinctive-looking box. If each time you open the box you find exactly the same contents, then the box can be said to be stateless. (This is also known as referential transparency). If each time you open the box the contents vary — the box could be empty sometimes, but full later, it could have different items in it or the same items with some new ones added — then it is stateful.

As you can see from this description, most boxes that we use in the real world are stateful, since it is very unusual for their contents to remain the same throughout their lifetime. However, if you are doing something complicated, say keeping financial records from past years in banker’s boxes, you might impose a discipline on your boxes, such as designating a particular box to contain only those financial records from April 2002, so that you can reliably find the records for a given month when you are looking for them.

Two styles of programming

Programming is like this as well. “Normal” imperative programming uses variables that can have changing values throughout the life of the program, which matches our understanding of boxes in the real world. FP takes the banker’s view of boxes. Since programs are very complex, and you want to be assured of finding what you expect, then the default assumption is that the contents of boxes will be the same throughout the life of the program.

So the key difference between a functional and an imperative style of programming is not whether there is any state at all in the program but where you start from. With imperative programming you start by assuming total changeability of all values you work with and have to add disciplines to get that state under control, whereas with FP you start from a place of statelessness and strategically loosen the reins in small amounts when you need to. The great strength of the FP approach is that it is much easier to maintain control by letting the genie out of the bottle slowly than by starting with the genie out of the bottle and trying to jam it back in.

Locality of State

The first dimension of state I want to discuss, locality, should be the easiest for programmers to understand but, for some reason, often causes confusion. Locality for state is just like scope for names in that it ranges from totally private to global, depending on how widely in your program it can be seen. A well-known property that makes state more local is encapsulation, when it is used to limit the visibility of a particular bit of state to a small part of the program. For encapsulation to accomplish this, though, it must actually hide the effects of the state from the parts of the program outside its scope.

To explain this, it is helpful to consider a common objection to functional programming languages by people new to the subject: how can a stateless program be built on a computer, which is an inherently stateful machine? The answer is that so long as the state is completely localized “under the hood”, it isn’t state at the next level up.

If we go back to the box analogy, let’s say that you have a very high-tech box that has the same contents every time you open it, but that, instead of the contents just being there when the box is closed, it actually disassembles the item when you close it and re-assembles it when you open the box again. The box is stateless because its contents never change from your point of view, even though its contents change when you can’t see inside.

So you can reduce the state in your program by ensuring that its various components look to each other as if they are stateless, even if internally they are implemented with large amounts of state, say for efficiency.

Monotonicity of State

Monotonicity of state is a little bit more subtle and probably less consciously familiar to programmers. Monotonic is a fancy way to say that once you add something, you can’t take it away. Using our boxes, the simplest example of monotonic state would be if, when you first open a box, you might find either nothing or something, but, as soon as you open the box and find something there, you must find the same something in the box every time you look in it thereafter. Upon finding the box empty, you could stop what you are doing and wait for something to show up in the box, after which you could be sure that it was not going to change, or you could take the opportunity to fill the box with something you want to fill it with, confident that someone else won’t be able to change it later. This is how dataflow variables work.

A more complex example of monotonic state would be a stream.
Once you have asked for the first value from the stream, it is as if you have filled the first box in the sequence, even if the box for the next item in the stream might still be empty.

With non-monotonic state, items can be added and taken away from the box at will, so that the box might have one thing in it the first time you open it, nothing in it the second time, and something totally new the third time. This is the kind of state that we normally think about when we talk about default imperative statefulness.

You can see that, though monotonic state is still state, it is much more predictable and manageable then full non-monotonic state, since you can always tell how much has changed from the last time you checked and you can be assured that what you found before will still be there. Furthermore, there is an explicit order to the changes in state that allows you to understand the history of operations, and thereby to make better automated decisions about what to do.

This is why the kind of message-passing concurrency, such as is found in Erlang is less stateful and relatively safer and more scalable than full shared-state concurrency, since an asynchronous message queue can be thought of as a monotonically-stateful stream.

Conclusion

While it would be much easier to talk about functional programming if statefulness were a nice black and white property that could be assigned to a particular language or program, I hope it is clear now that things are not quite so simple, and that degrees of state and “functionalness” could be present in any language or program. Obviously some languages and programs are going to make reduced state the default, and thus easier to implement, but the functional perspective can be applied to any language or program.

And making use of this perspective will help tame the complexity of software, especially in the face of concurrency.

Metaprogramming still needs programming discipline

noreply@blogger.com (Marc Hamann) — Thu, 14 May 2009 03:00:00 +0000

This post will be a little bit more technical than previous posts, but I’ll try to keep it comprehensible to those who don’t already know what I’m talking about.

I want to talk about meta-programming. Meta-programming, in the most general sense, means making programs that produce programs or that change what existing programs do by altering the environment in which they operate. Writing an interpreter or a compiler for a programming language can be considered an example of meta-programming. Macros, monkey-patching and code generation are also examples. I would argue that the XML “configuration” files that haunt many Java frameworks, such as Spring, Hibernate, Struts, etc., are a form of meta-programming.

Now meta-programming is a very important idea, and I would go so far as to say that everyone who is at all serious about programming should learn about and understand meta-programming, even if only at a basic level. Meta-programming is often considered an advanced topic, and there certainly are advanced forms of it and advanced ideas that fall under its domain, but I think that anyone who is smart enough to program at all is smart enough to understand and perform basic meta-programming.

Now, since meta-programming is very important, has deep things to say about computation, and is intellectually stimulating, many people find it very exciting and even beautiful.

I will confess: I am one of those people. I have spent a non-trivial chunk of my leisure time throughout my adult life studying the semantics of programming languages, and all the supporting theories and math. And though knowing this stuff has been directly and indirectly useful in my software development career, I really did it because I enjoy it, and because I think it is beautiful and exciting.

Having said all these great things about meta-programming, whenever I see someone using meta-programming in a project, I get queasy. And that is because people often forget that just because meta-programming seems to let you go beyond the rules of “mere” programming, it still requires all the same discipline you would apply to any other kind of programming. For example, you still need to remember source code discipline and the enforcement of locality principles, such as modularity, the single-responsibility principle, encapsulation, don’t-repeat-yourself (DRY), and many others.

Take DRY for example. I have seen many programmers who would never tolerate cut-and-paste boilerplate in the source code blithely create megabytes worth of XML “configuration” files, or use a source code generator to do the same thing, even if there might be more acceptable alternatives with a bit of creativity.

Moreover, I think there are two kinds of locality that meta-programming should have that wouldn’t apply to single-level programming. First, meta-programming level code should be modularized away from the programming level code, and second, any domain-specific-languages (DSLs) or language variants created by the meta-programming should be clearly demarcated from the “normal” programming language (in their own source files if possible).

The reason for this is simple. Imagine if I started writing this post by alternating between English and some other language, say French. Some sentences or phrases in one language, some in the other. First of all, any member of the audience who is not fluent in both will probably be lost immediately. For those who are bilingual, there are many words that are spelled the same or very similarly in both languages, but may not have the same meaning, either subtly or not so subtly. So even if you are fluent in both, aside from the difficulty I’m adding by making you code switch, I may also cause you to misinterpret what I’m saying with similar or ambiguous words.

Just as the user of a programming library only wants to have to think about the interface to that library and not have to understand the gory details of how it actually works, the consumers of meta-programming “enhancements” need to be insulated from having to understand the details of how it works under the hood. This is where non-local meta-programming, such as reckless global monkey-patching, can really mess things up.

So anyone who wants to keep meta-programming beautiful should use the same judgment, taste and discipline that an experienced programmer would apply to keep any other type of programming beautiful. Otherwise, it can morph into something very, very ugly.

Good to Great

noreply@blogger.com (Marc Hamann) — Sun, 03 May 2009 21:49:00 +0000

I finally got around to reading the book Good to Great. I first heard about it in early 2002 in the introductory speech of the incoming president at an organization I was in the process of leaving. I was quite impressed with what he had to say about leadership and organizational greatness and made a note to read the book as soon as it came out in paperback. (Hardcovers use up too much shelf space, which is at a premium in my home, so I tend to avoid buying them.) Since it is still in hardcover all these years later, I gave up and borrowed a copy from my brother.

There is a lot of criticism of the book online (here, here and here), but I think most of it misses the point. It doesn’t matter if you are happy with the experimental model the book was based on, or if the companies profiled didn’t stay good stock picks forever after the book came out, or if the advice Collins gives boils down to “obvious” suggestions.

The best business books tell you in an organized and compelling way what you already know to be true from your own experience. No business book can predict the future or give you a sure fire recipe for success, and Good to Great doesn’t pretend to anyway, so why do the critics expect it to?

What I did find in this book was a well-written and thoughtful explanation of the only reliable strategy I know of to accomplish any undertaking: for a disciplined group of people to pursue a focused goal in a determined manner, while being willing to see failure.

Now one can say “Hey, that’s obvious!”. But unfortunately, the obvious is more often observed in the breech than in the practice. The path of least resistance in many organizations is to let ineffective or obstructive people stick around way too long, to let the day-to-day crises and tempests-in-a-teapot derail their long-term goals, and to rationalize or ignore failures.

If Good to Great manages to inspire people by reminding them to struggle against these tendencies and to set a higher standard toward greatness, I think it deserves its place on the best-seller list.

Dancing Monkeys

noreply@blogger.com (Marc Hamann) — Sat, 02 May 2009 19:26:00 +0000

Many years ago, before I started my software development career, I had a job call-center representative for a national automobile association. This included a twelve-hour shift by myself on Sundays.

The Sunday shift was a bit of a grind. It could be a long, silent, boring wait for calls that never came, or if the weather was inclement somewhere on the other side of the country, it could be very busy handling a spate of calls all alone.

During the normal workweek, when there were a bunch of us working, we would pass the slow times doing paperwork. By Sunday, when I was often the only person in the building, the paperwork was usually all done, so to pass the time I took to listening to the radio. As soon as the phone rang I would shut the radio off and pick up. There was no doubt in my mind that I was doing my job fully and faithfully, in spite of the discomfort and difficulty of the job.

One Sunday, the Vice President of the company decided to come to the office to catch up on some work. At some point, he wandered back to the call-center where it was a slow day, and I was sitting there by myself listening to the radio. He may have greeted me, or asked me how busy I was, but he didn’t stay long and he didn’t say anything significant.

However, on Monday I heard from my boss that he had complained that I was listening to the radio instead of working.

This was my first significant experience of a phenomenon that I call “Dancing Monkeys”: the tendency of arms-length leadership to prefer situations where everyone “looks busy” or is “hopping to”, even when the expected effort would have no additional effect or might even be counterproductive.

There are many reasons for this phenomenon some pernicious and some benign. The pernicious ones that we will all have seen at some time or another is pure ego-tripping: “I’m the top monkey here, so you lesser monkeys start dancing!”

A common, more benign reason is lack of understanding of the work domain under observation. Software development is particularly prone to this one.
For example, more than once I’ve heard some executive complain that he stopped by the development team area and no one was typing, with the implication being that no work was getting done, since as every non-technical person knows programming is all about lines per minute of code typed.

Another common scenario is that inevitable day when, under some unforeseen deadline pressure, some executive asks the development manager when he is going to institute overtime to help meet the deadline. Because of course, as any non-programmer knows, there is no degradation to the quality of software development when the team is exhausted, since typing lines of code is a purely mechanical process.

To come back to my call-center job, effective execution of my duties required that when I got a run of long, stressful calls, I had the energy and focus needed to solve the clients’ problems efficiently and effectively. Anyone who has ever made a service call to a droning zombie service rep knows what happens when someone doing that job lets their energy reserve run down.

Listening to the radio helped me recharge between calls, kept my morale up on quiet days and improved my performance. Doing mindless, unnecessary busy-work would have sapped that energy and morale. Asking my boss to find work for me to do that was meaningful but would not interrupt my real work would have just increased her workload without increasing the efficiency or effectiveness of our department.

So that Vice President had to ask himself: was making himself feel better by getting a dancing monkey really in the best interest of his company?

Ideology vs Principles

noreply@blogger.com (Marc Hamann) — Thu, 02 Apr 2009 13:43:00 +0000

Ideologies are strangely seductive.

Ideologies promise a sure-fire recipe to solve all problems. They often give you a single value to maximize, and equally often tell you who to blame.

Got a problem? Easy. Just trust the free market. Or increase government spending. Or blame the wealthy. Or blame the poor.

Whichever solution an ideology offers, that is supposed to be the first solution to apply whenever a problem arises, and if that doesn't work you're supposed to do apply it again until it does work.

Given the complexity of some of the problems that face us in our daily lives as individuals and even more so in our collective life as a society, it is easy to see the allure of straightforward solutions you don't have to think about too much.

But history has shown us over and over again that ideologies are fairly poor at solving problems in the short run, and in the long run usually create worse problems of their own.

If you want effective problem solving, you're better off with principles rather than ideology.

A set of principles is like an ideology in that it expresses values upon which to base decisions, but principles can't be applied unconditionally. Principles compete with each other, forcing you to make trade-off decisions. The exact trade-offs have to be worked out differently for each instance of a problem.

Ideology is like the imaginary fifties family where everyone listens to Dad and the kids never compete for attention. Principles are like a real family, where you love everybody, but you can't always make everybody happy.

The amount of work required to implement a decision is about the same for both ideological and principled approaches is. The real difference is in the amount of certainty you have, both before you choose a solution, and after you've made a choice.

In a principled approach, you always have to wonder if you could have gotten the balance of competing values better, and you may have to constantly rejig the balance as new situations develop. With an ideology, you always have it right, even if you aren't getting the results you want.

As nice a feeling as it is to be certain, if you really care about outcomes, uncertainty has the benefit that it keeps you more closely in touch with the realities of your problem. It makes you listen more attentively to the feedback from your solution. You are much more likely to succeed in the end, even if paradoxically you are less sure at any one point in time that you have found the right solution.

Abandoning certainty leads to greater confidence of real success.

This idea has so many applications to software development, project management, government, and other governance and systems thinking domains, I'll leave elaboration to future posts.

Explaining through Skillful Means

noreply@blogger.com (Marc Hamann) — Mon, 30 Mar 2009 12:53:00 +0000

In Buddhist philosophy, there is a concept whose Sanskrit name is upaya. This is often translated as "skillful means", though, as with many other ancient philosophical terms, it is hard to translate exactly and has many interpretations.

To illustrate one useful interpretation, I'll start by explaining a little bit about Buddhism. If you boil Buddhism down to a one-sentence summary, it is about reducing your suffering (and that of the world) by living a moderate life and practicing meditation.

As simple as this sounds, there is a lot of theory behind it, covering — for example — what it means to live a moderate life, why that helps reduce suffering, what good meditation is and how it works, and a host of other questions.

The theory and practice of Buddhism is in fact so complex that it is usually taken for granted that it takes years of study and practice to really understand it. To make matters worse, there are many different opinions about what kind of study and practice, and how much.

But most of the concepts can be understood at first in a simplified way, and since all students of any complex skill have to start somewhere, people often start with these simplified forms that aren’t quite right to a more advanced student.

For example, there is an uncomplicated form of Buddhism called Pure Land, that more or less believes that if you faithfully chant a particular mantra, you will be reborn in a special paradise. You can still see in this belief some of the elements in my one-sentence summary above: chanting a mantra is a form of meditation, being reborn in a paradise is a reduction of suffering, and faithfully practicing a good habit is a rudimentary form of living a disciplined life.

So even though a “more advanced” student of Buddhism might roll their eyes at the simplicity of this approach, they would have to admit that practicing it was still an advancement in the direction of the aims of Buddhism, and therefore better than no understanding of Buddhist principles.

This, in a nutshell, is what upaya means: it is OK to let someone have a “wrong” idea about something you are explaining to them, so long as it takes them one step closer to understanding.

What does this have to do with software development?

Software development, boiled down, is about solving practical problems by producing reliable applications with maintainable code bases. (All of these things, incidentally, also reduce suffering. ;-) )

There is a complex skill set and complex methodologies that must be learned to accomplish this task. It is taken for granted (at least by some people) that mastering them takes years of study and practice, but there are many different opinions about what kind of study and practice, and how much.

But many concepts have a simplified version, and since all learners have to start somewhere, you often see versions of good practices that a more advanced practitioner would see as not quite right.

For example, a developer with less experience — or perhaps just different experience — might chant a mantra such as “always document”, “never document”, “always design before you program”, “never design before your program”, or “paradigm/language/library/framework X is always better than Y”. These mantras might strike you as over-simplified or incorrect based on your experience (especially when you favour an opposite version of the chant. ;-) ).

You can still see that they share the core concern of solving the practical problem reliably and maintainably. Maybe it is OK to let them be “wrong” for now, so long as they are engaged with the ultimate concerns of software development and are using the mantra to get one step closer to understanding.

When mentoring developers as a team lead or when introducing new ideas, techniques and methodologies to your peers or to an on-line community, it can be hard to determine when to hold a hard line, to nitpick or to qualify and when not to. Often it is best to introduce a simplified form of the idea and let them run with it for a while, trusting that if they share the same goals they will come to a deeper understanding on their own and that they will ask for more input when they need it.

Sometimes the best policy is to let others be "wrong" for the time being.