Programming Seeking Understanding

SEEK Bytes Podcast - Journey in GraphQL

2023-09-22T00:00:00+10:00

For anyone interested in GraphQL generally, and a little about our use of it at SEEK on the SEEK candidate website, you can listen to a few of us discussing it on the SEEK bytes podcast, Journey into GraphQL.

Wardrobe Kata

2020-11-11T00:00:00+11:00

Having recently done the wardrobe kata, I was asked about the development process that led to my implementation, particularly in relation to TDD.

The development process that was followed was roughly:

Define the problem in mathematical terms.
Think up a solution, in mathematical terms.
Implement the solution in Typescript.
Determine if the solution dealt with various additional cases for which tests were added requiring a few minor coding changes.

Two points are particularly relevant to TDD. First, typically I would write tests for functions starting with the degenerate and trivial cases. For the wardrobe kata this could include:

distance	wardrobe sizes	expected result
0	[]	[]
10	[]	[]
1	[1]	[1]
1	[2]	[]

Often such tests pass with very little effort. While I would normally follow this process, in this case I did not and suffered the consequences later with step 4 when I finally turned to such cases.

Second, in the context of other problems it has been observed that at the point of jumping from degenerate/trivial cases to a single more complex case an entire solution can be forced out that works for all other cases (for instance, see Mark Seemann’s discussion of the diamond kata, notably the section on ‘Example-Driven Development’).

For instance, all the degenerate/trivial cases could be solved as follows:


export function combinationsEqualling(n: number, amounts: number[]): number[][] {
  if (amounts.length === 1 && amounts[0] === number) {
    return [[amounts[0]]];
  }
  return [];
}

It would be possible, albeit annoying, to code for all combinations of amounts with up to two elements, but by the time one gets to three elements you get that unnerving sense that it is necessary to solve for the general case.

For this implementation of the kata, solving this single more complex case, which was essentially the question posed in the kata, was solved in steps 1-3 without any modifications to the single test defined in the kata description.

1. Define the problem in mathematical terms

In thinking about the problem, the first step was to express it as a mathematical model. At this point, a concrete problem about wardrobes and available space becomes an abstract problem, expressed in mathematical terms.

Given a list of numbers, in how many unique ways can they be combined to add up to a particular number without regard for ordering. Essentially, we are looking for the complete collection of Set<Bag<Integer>> generated from a Set<Integer> where each Bag<Integer> adds up to a certain amount.
- Set: collection of unordered numbers with no duplicates.
- Bag: collection of unordered numbers allowing duplicates.

There is not necessarily a single abstract way of expressing the problem. this can be a case of oscillating back and forth between the concrete and abstract until an appropriate abstraction is found. This may occur when an abstract expression of the problem yields no solutions, or even as it becomes more clear precisely what is being modelled and the model needs to change. However, in this case the statement of an abstract problem led to a potential solution right from the start.

2. Mathematical Solution

Given the abstract expression of the problem, a solution is:

Create a tree of options. Each node branches out covering all the elements of the Set<Integer>. A node’s value is the Bag of values consisting of the node above it, with the new branch value added in.
Keep going down the tree - when a nodes value equals the desired amount add it to a Set of all the solutions, If the sum of amounts is greater than the total, end that branch at the point.

Admittedly, it was probably a particular mathematical solution I had in mind that suggested this approach:

Recursively calculate the Cartesian product of the amounts in the list with the values previously calculated.

3. Implement a solution in Typescript.

Having started from the hint of a Cartesian product as a way to keep track of the currently in progress leaves of a tree together with a list of numbers, the problem then became implementing it in Typescript. At this point the solution was largely in terms of a few algorithms and data structures:

the end result I was after was a Set<Bag<Integer>>. That meant order of values didn’t matter (a bag), and no repeats were allowed (Set). Performing a cartesian product was going to mean taking the following:
- Set<Bag<Integer>>
- List<Integer>
- perform Cartesian product, resulting in List<(Integer, Bag<Integer>)>, which I would reduce to List<Bag<Integer>>.

Given that I was working in typescript, and was aiming not to use any additional libraries, I decide to use Lists, and worry about reducing a number[][] with repeats to an number[][] without any repeats and ignoring order as a second step.

Hence, the end result in Typescript is two steps:

Keep recursively applying the Cartesian product of array amounts to the current Integer[][], pulling out arrays in the number[][] that sum to the desired total, throwing away results over the total, and recursively calling again with the new number[][] that remain summing to less than the total.
The resulting number[][] has repeated combinations. Treating a number[][] as a Set<Bag<Integer>> was implemented as follows:
- order each of the arrays of numbers.
- Add each of the arrays to a Map<String, number[]>. Assuming the hash key for each number[] is unique, this removes all duplicates. In typescript, as it turns out a plain old object can serve as a Map<String, *>. The hash, is the the list .toString().

Continuous Delivery For A Nuget Package Using Appveyor

2017-10-12T00:00:00+11:00

This post outlines some of the details involved in setting up a continuous delivery pipeline for a nuget package. The primary goal was to be able to deploy a new version of a package with a command from a development machine. It is based on experiences in setting up Hdq.Lib. The primary tools used are Appveyor for building and testing and Nuget.org as the package repository.

Pipeline

The besic pipeline from source code to nuget package is:

Solution containing C# project and corresponding test project
Local git repository
Github repository
Appveyor (build, run tests, create nuget package, deploy nuget package)
nuget.org (package repository)

Requirements

The basic requirements were:

Release of a new version of the library to nuget.org is by an operation from the developer’s machine.
The mechanism to implement this is is tagging a local git repository commit and pushing to GitHub.

git tag -a 1.2.3 -m "New release of amazing things"
git push --follow-tags

Deployment builds (where a release is deployed to nuget.org) use version numbers for the package including the tag and the build number.
For instance, if the tag is “1.2.3” and the appveyor build number (a continuously incrementing number) is 999, then the version number of the package would be “1.2.3.999”
Non deployed builds including build/test/packaging should occur for commits that are not tagged. These builds should be versioned in a way that is clearly distinguished from the release builds.
For example, when code is pushed to GitHub (origin), a build should occur in appveyor. If the appveyor build number was 888, and the git commit hash was ab1cd23, the build version number would be 0.0.0.888-ab1cd23

Constraints

The following are constraints or behaviours of the systems being used:

Nuget package numbers must be of the form: n.n.n.n[-ssss]
Appveyor allows for any sort of build version number, but if you want to patch dll’s with the version, it must be of the form n.n.n.[-ssss].
However, while the build will be named after the full version number, the dll’s will be patches with n.n.n.n, and the package pushed to nuget.org will also have the form n.n.n.n (i.e. the string from ‘-‘ onwards is dropped).

Setup

Most of the setup from Visual Studio solution through to Github repository is not difficult, simply being a standard GitHub repository setup. Appveyor and Nuget are where most of the interesting work lies.

Nuget

There are plenty of helpful guides to getting started with nuget. The two I found most helpful were:

Create and publish a package
A very simple getting started guide.
Creating NuGet packages
Somewhat more detailed description of getting started.

A high level summary of the process (for more detail see Create and publish a package) is:

download the nuget executable and put it in a location in the PATH.
go to the directory containing the csproj file and create a spec file there

nuget spec

Edit the newly created spec file. Mine ended up looking like Hdq.Lib.nuspec The most notable things in the nuspec file are:
- <id> is going to be the name of the project in nuget.org.
- <version> must be ‘$version’ so that it be externally set (ultimately by appveyor) when the package is built.
Create the nuget file. Ensure that the project is built first, or it won’t work.

nuget pack projectname.csproj

This can be tested from the dev machine. If you do happen to try this take note that once a version number for a project is used, while it is possible to delete that version of the package from nuget.org, it is not possible to ever use that version number again. For testing purposes, the version number can be temporarily changed from ‘$version’ to something like ‘0.0.0.1’.

Upload the file package to nuget.org.

nuget push packageName.versionNumber.nupkg api-key-here -Source https://www.nuget.org/api/v2/package

Appveyor

Creating an Appveyor account and connecting it to your GitHub repository is pretty much as simple as following their Getting Started. If you get this far, commits to the GitHub master branch will cause Appveyor to build the project and run the tests. Most of the subsequent setup I performed using the project “SETTINGS” user interface online. The settings allows for an appveyor.yml file to be exported. This file can then be add to the project and becomes the project definition, replacing whatever is defined by the settings UI.

The main alterations to the default settings involve getting the versioning to work and deployment to nuget. Following is my appveyor.yml, with comments liberally added to explain what is happening.

# By default the version is set to the same thing as non-deployed builds.
# However, this is always overwritten using Update-AppveyorBuild below.
version: 0.0.0.{build}
image: Visual Studio 2017
# Package should use the Release, not Debug configuration.
configuration: Release
# Following is a powershell script that sets the version.
# If the build is triggered by a repo tag, then the version number 
# gets set to something appropriate for a deployed build (e.g. 1.2.3.456)
# Otherwise, the verison number gets set to something like 0.0.0.456-er43f3
init:
- ps: >-
    if ($env:APPVEYOR_REPO_TAG -eq "true")

    {
      Update-AppveyorBuild -Version "$($env:APPVEYOR_REPO_TAG_NAME).$($env:APPVEYOR_BUILD_NUMBER)"
    }

    else

    {
      Update-AppveyorBuild -Version "0.0.0.$($env:APPVEYOR_BUILD_NUMBER)-$($env:APPVEYOR_REPO_COMMIT.substring(0,7))"
    }
assembly_info:
# Patch true ensures that the assemblies version number 
# gets set to the build version number
  patch: true
  file: '**\AssemblyInfo.*'
  assembly_version: '{version}'
  assembly_file_version: '{version}'
  assembly_informational_version: '{version}'
# The nuget package always get published to the project feed. 
# This allows it to be downloaded and tested.
nuget:
  project_feed: true
before_build:
# Before building, restore any nuget libraries required.
- cmd: nuget restore Hdq.Lib.sln
# Build specifies that the nuget package must be published.
build:
  publish_nuget: true
  verbosity: normal
# The following is the details allowing for appveyor to push the 
# new version of the package to nuget.org.
deploy:
- provider: NuGet
  api_key:
    secure: j5g0Htw7zXZ/iAMHi68yb+9AZIO2bno642h/Djfe62Ux4ZiHIVNg0SpiIgI/BtqG
  artifact: /.*\.nupkg/
# Only deploy to nuget.org when the build was triggered by a tag.
  on:
    APPVEYOR_REPO_TAG: true

Getting started with Postgres, Docker and C#

2017-05-06T00:00:00+10:00

Postgres proto documents the basic steps for getting Postgres up-and-running on a windows machine. It is intended as a very simple dev machine setup for experimentation with postgres.

It includes the following:

Postgres running in Docker.
Test C# unit test to connect to a database, insert and retrieve a row using Npgsql and Dapper.

Elasticsearch and Docker

2017-05-05T00:00:00+10:00

Installing and running Elasticsearch on a dev machine is a relatively simple task. Yet once you throw in Kibana, different Java versions on your dev machine, and you want to be able to start it all easily and stop it all easily on demand, while still in the realm of simple it starts to require some management. On my machine I have various versions of Elasticsearch that I can run up, and along with that a script that tends to look something like:

cd c:\Code\proto\elasticsearch5
set JAVA_HOME=C:\Program Files\Java\jre1.8.0_91

cd elasticsearch-5.2.2
start cmd /K .\bin\elasticsearch.bat

cd ..\kibana-5.2.2-windows-x86
start cmd /K .\bin\kibana.bat

However, even easier than all that is to simply use Docker, and the official Elasticsearch provided docker images. With that in mind I created a docker-compose file to put it all together to further simplify that task. See elasticsearch-docker.

Introduction to Elasticsearch Presentation

2016-11-05T00:00:00+11:00

Having found myself working on a search project using the amazing Elasticsearch, I recently gave a beginner’s introduction to Elasticsearch presentation at work. A very brief overview was followed by a live demo.

Data Source

As a data source I used ‘Airplane Crashes since 1908’ – primarily chosen because it had the types of data I wanted to demo, but morbidly interesting in its own right!

Source data was obtained from Airplane Crashes Since 1908 on Kaggle

Csv converted to json using nodejs csvtojson

Json reformatted and bulk posted to elasticsearch, using jq

cat AirplaneCrashes.json | jq -c '.[] | {"index": {"_index": "airplanecrash", "_type": "type1"}}, .' | curl -XPOST localhost:9200/_bulk --data-binary @-

Writing a Jekyll Blog on Github with Docker For Windows

2016-11-05T00:00:00+11:00

Having moved development computers a few times in the last year I was keen to make blog maintenance as simple as possible. Generally finding myself on a Windows machine as my primary work development environment I wanted that to be the primary blogging environment as well.

While a number of changes to jekyll and github’s usage of jekyll have make it a lot easier to maintain a jekyll blog on windows, docker seems a better long term proposition.

The end result is that the following gets blogging up and run it in jekyll watch mode.

git clone git@github.com:hombredequeso/hombredequeso.github.com.git  
cd hombredequeso.github.com  
docker run -v %cd%:/usr/src/app -p "4000:4000" starefossen/github-pages

The devil, of course, is in the detail.

The first detail to have caused problems was getting a correctly formatted docker run, specifically mounting a volume from the current directory in a cmd shell. %cd% is the end result to achieve that.

The second detail that seems to cause problems for people is ensuring that docker itself is able to share folders. To do that, go to Docker > Settings > Shared Drives > and make sure the drive with the blog on it is selected as Shared.

Frameworks and Libraries for an F# Project

2016-02-02T00:00:00+11:00

Finding ourselves pushed in the direction of a more functional programming paradigm, our team chose a new, stand-alone project on which to try out F#. At first sight, it was not the sort of project that would show off the benefits of a functional language. But it did have the advantage of being relatively simple (or at least, as simple as anything looks before you actually try to do it). Historically, that suited our team’s approach to new technologies: try it first on something fairly simple, something that if things went wrong wouldn’t ruin the company; then if it works out, broaden its scope. And so we set out on our new project.

Commencing a project in a new language poses a number of distinct challenges. Such challenges can be even greater when a change in programming paradigm is thrown into the mix, as is case in moving from C# to F#. Some of the main challenges we faced included:

determining appropriate frameworks and libraries to use
structuring an F# solution
effectively using the F# language

Frameworks

As a .NET language, F# has the capability of working with a very large number of existing .NET libraries, primarily written in C#. In addition, there is also a small, but rapidly growing number of F# libraries. Our project required a website, with distributed background services, which ended up hosted in Azure. The final choice of key frameworks/libraries was:

Nancy: For the website. All out code was written in F#, using the C# Nancy library.
DBML file type provider An F# library used for SQL Azure database access.
NServiceBus: a framework for distributed communication between back-end services written in C#. All our code was written in F#, however the messaging assemblies were in C# to simplify communication with additional nservicebus endpoints written in C#.
BrockAllen.MembershipReboot For authentication/authorization, another C# library.
FsUnit : a unit testing assembly providing functional testing syntax. We used it on top of NUnit as the base testing framework..
FsCheck: an F# library for property based testing.

Frameworks: The Trade-Offs

This choice of libraries and frameworks came with a number of advantages and disadvantages: One of the most significant advantages, which most heavily swayed our choice, was the benefit of using familiar tools. A second advantage is that the more mature libraries and frameworks have a host of support options, the most obvious being the sheer volume of support from our friend google. This is not to suggest that the support from the F# community for newer F# specific solutions is anything less than great, but simply a question of volume of support, particuarly around the odds that someone somewhere has hit the same problem as you and written about it.

Probably the biggest disadvantage in using existing C# based libraries is that they are written, rather predictably, for usage in an object-oriented context. It is undoubtedly true that F# is a hybrid language, capable of being used in a functional, or object-oriented way. However, its full benefits, and arguably most elegent code, is functional. Interacting with C# libraries forces one into an object-oriented mode of programming. We found this posed a few challenges. It meant that rather than setting out to write specifically F# code, at these points in the program we found ourselves effectively writing C# code translated into F#. Such code mostly swerved between passably adequate, through to pretty ugly. Undoubtedly, some of that was due to inexperience. However, even with the benefit of rework given more experience, such code rarely seems to rise above the level of adequate. Overall, the use of existing C# based libraries is something of a two edged sword. It provides familiarity, avoids having to learn yet another library/framework, and when chosen carefully offers a good deal of support. On the other hand, it pushes one into a ‘translated from C#’ mode of programming which shields one from the extent of the jump between programming paradigms, and at worst retards one’s progress in the language as the ingrained habits (good and bad) of one paradigm prevent one from understanding the new paradigm.

Conclusion

So in hindsight would we do the same again? On balance, I find myself thinking we probably made the right decision. Yes, there was a week or so of weeping and gnashing of teeth while we put it all together. Sometimes I pine for the purity that using a more functional library would have given (In particular I’m looking at you Suave). But overall, using libraries and frameworks we were familiar with, notably Nancy and NServiceBus, enabled us to get something going without having to also learn a new framework, with all its features and, well, idosyncracies. Now in production, the choice of nservicebus in particular, means that we continue to get the benefit of using a familiar distributed messaging system, with all the functionality and tooling it has build up over years.

I suspect that the worst part of this trade-off is that the very early stages of getting a project going involve heavily working with the frameworks and libraries. For the developer this means that having seen all the wonderful things that functional F# offers, the first production-like experience one might have of F# is little more than struggling to mentally translate C# code into F#. Not only can that be a frustrating experience, it doesn’t shine a light on the best things F# has to offer. Yet, all being said and done, we remain content with our choice of technologies, and in the longer term, it has left us working with mature libraries we are familiar with (which has helped in sorting out production issues). More importantly, as time goes by the glue-code between the libraries and our code requires little attention, leaving us to enjoy F# the way nature intended it.

Why So Functional?

2016-01-29T00:00:00+11:00

In a recent project at work the decision was made to use F# for parts of the system, specifically a website and back-end processing service. A number of reasons contributed to this decision. A common thread was that changes in our style of using an object-oriented language, C#, was naturally leading us towards a more functional style of coding. In this post I outline some of these issues.

Immutability

Overtime, more situations appeared in which problems were solved primarily using immutable objects. This stood in contrast to the more object-oriented approach of encapsulating state within a object, mutated by methods on the object. Initially this tended to appear around the periphery of the primary business domains, but over time it crept into various domains as well. Typically, in approaches such as Domain Driven Design, immutable objects are associated with value objects, as opposed to entities. Elsewhere, immutable objects are prevalent at boundaries between systems within the application, say as DTO’s (Data Transfer Objects). But when immutable objects start to form central parts of the business functionality, it is a something of a different matter.

Value Equality

Increased use of value equality was also experienced within the application. This stood in contrast to reference equality, or entity equality determined by an Id property. To a certain extent, this goes along with the increased use of immutable objects. In more and more places beyond the traditional value objects it made sense to work with value equality. In C# this means implementing the IEquatable interface on a class, with all the extra code that goes along with that.

Static Methods

An increase in static methods partially occurred along-side the increased use of immutable objects. Elsewhere, however, other reasons contributed. For instance, message handling classes started to get more static methods on them. Sometimes this was driven by the realization that a good number of such methods simply could be static, and so should be. In other cases it occurred to make testing easier. For instance, the DoSomething method here:

class MyHandler: IHandleMessages<MyCommand>
{
    private readonly ISession _session;
    public MyHandler(ISession session)
    {
        _session = session;
    }
    public void Handle(MyCommand cmd)
    {
        ...
        DoSomething(x, y, z)
    }
    public void DoSomething(int x, int y, int z)
    {
        ...
        // use _session
        ...
    }
}

can be rewritten as:

        
class MyHandler: IHandlerMessages<MyCommand>
{
    ...
    public static void DoSomething(ISession session, int x, int y, int z)
    {
        ...
    }
}

An immediate result of this change is that the DoSomething method is freed from dependency on the MyHandler class, and it’s dependency on ISession is clearly highlighted. Beyond this its new found freedom from the MyHandler class can be reimplemented as:

static class ThingsToDo
{
    public static void DoSomething(ISession session, int x, int y, int z)
}

While the benefits for a contrived example such as this are somewhat limited, it might be a different matter if MyHandler looked more like:

class MyHandler: IHandleMessages<MyCommand>
{
    private readonly ISession _session;
    private readonly ISomeFactory _factory;
    private readonly ISomeService _service;
    public MyHandler(ISession session, ISomeFactory factory, ISomeService service)
    {
        _session = session;
        _factory = factory;
        _service = service;
    }
    public void Handle(MyCommand cmd)
    {
        //... handle using _factory and _service, then:
        DoSomething(x, y, z)
    }
    public void DoSomething(int x, int y, int z)
    {
        ...
        // use _session
        ...
    }
}

If DoSomething is not dependent on ISomeFactory and ISomeService, there is no need to bundle it together with them in MyHandler. Practically, this has the immediate benefit of making testing cleaner and easier. This benefit occurs because the actual dependencies are more accurately represented in the code. A further development for DoSomething may well be to refactor it to return something:

static class Things
{
    public static T DoSomething(ISession session, int x, int y, int z)
}

This can give it the characteristic that it becomes easier to reuse in other contexts.

It is also worth pausing for a moment to consider the nature of the relationship between MyHandler and ISession. Considering the idea of an object within the object-oriented programming paradigm as state encapsulated within an object, with methods available to the object which manipulate that state, it can be seen that a message handler like MyHandler is not quite the same thing as say, a customer object in which the customer’s state is encapsulated within the object and manipulated by methods on that object. MyHandler’s use of constructor injection is a convenient way to use the object-oriented paradigm, typically together with an IOC library to compose the outer dependencies of a system.

With that it mind, requirements of an IOC aside, there is little difference between the first version of MyHandler above and the following, at which point there is nothing particularly object-oriented about its implementation, aside from the linguistic requirement of having a static class:

static class MyHandler
{
    public static void Handle(ISession session, MyCommand cmd)
    {
        ...
        DoSomething(session, x, y, z)
    }
    public static void DoSomething(ISession session, int x, int y, int z)
    {
        ...
        // use _session
        ...
    }
}

Recurrently Appearing Patterns

Certain recurrent patterns were also gaining use which pressed further in a similar direction, notably the use of event-store like designs. Event-store like designs do not necessary result in a system of immutable objects. However, the events themselves are immutable. If the events are the means of persistence then that begins to change the characteristics of the system’s persistence, as nothing is edited, only new entries appended. Of course, this is not purely functional as adding new entries to something is itself mutating the something. But it does lead to thinking about the program in new ways. Beyond the events on their own, the reconstitution of aggregates within a domain, or any other entity that could be constituted from a series of events (we used it to constitute JSON entities on a REST API) presents further food for thought.

Within an object-oriented paradigm, events may be used to reconstitute aggregates via a method such a constructor injection:

public class MyAggregate
{
    public MyAggregate(IEnumerable<IEvent> myAggregateEvents)
    {
        // replay myAggregateEvents here to reformulate AggregateState
    }
    T AggregateState;
}

Whether by design or accident, it may not be long before an equivalent variant on the object-oriented paradigm arises:

public static class MyAggregate
{
    public static T Apply(T initialState, IEvent e)
    {
        // apply e to initialState, and return the new state.
    }
    public static T Apply(IEnumerable<IEvent> eventStream)
    {
        T emptyState = default(T);
        T currentState = emptyState;
        foreach (var e in eventStream)
            currentState = Apply(currentState, e)
    }
}

The problem is not so much that these patterns need to be learned. Given enough time and enough situations, I suspect most developers are likely to ‘discover’ such patterns, simply because there are circumstances in which they are advantageous, or simply a more natural implementation, over the more object-oriented approach.

SOLID principles

Beyond the somewhat eclectic points listed above, there are more design oriented reasons for why object-oriented programming may naturally lead towards a more functional approach. In a post that reflects our own experience, Mark Seemann has noted in SOLID: the next step is functional, that consistent and thorough-going application of the SOLID principles within an object-oriented environment has the effect of turning one’s programming into a more functional paradigm.

Composition Over Inheritance

It is hardly a new observation that composition is generally to be preferred over inheritance. Inheritance is one of the strongest forms of dependency. And once one has a bit of experience with how the wonder of inheritance is not all apples and oranges, or cats and dogs, it does not take long to appreciate the wisdom of this advice. Of course, once the emphasis in code shifts towards composition, the primary object-oriented mechanisms for achieving polymorphic behaviour takes a back seat to one of the basic building blocks of functional programming.

Conclusion

The features discussed above primarily emerged over a period of time as a change in programming style within an object-oriented programming mindset. We were aware that these features had simpler implementations, or were more naturally implemented and enforced in a functional paradigm, a fact that led us to consider using a functional programming language.

Webforms and Gulp

2015-04-21T00:00:00+10:00

Recently at work we have been moving our front end development over to newer technologies. This was motivated by a number of factors.

The existing javascript packaging, using the Telerik ScriptManager was not necessarily difficult to use, but tended to be fiddly to work with.
The existing setup required msbuild to package the javascript, slowing development.
There was a desire to be able to use newer front-end development technologies, but most would prove very difficult to put into the existing development pipeline.
Javascript unit testing was being done with jstestdriver, not a bad choice a few years ago, but now a largely defunct project.

We were looking for a solution that

Would allow us to easily take advantage of more recent front-end development tools and technologies.
Disconnect front end development from the rest of the application as much as possible.
Would not require compilation using Visual Studio when doing front end development. One of the important factors here was to speed up front-end development, reducing the feedback cycle time when editing javascript or html.
Kept the development environment as close as possible to the production environment.

It was decided the way forward was to use Gulp for processing the javascript development pipeline, and Karma for running javascript tests. Putting all this into an existing ASP.NET Webforms application presented a few little hurdles. A sample application with the major features of how it all hangs together can be found here.

Adding karma testing was relatively easy. Using Gulp for the javascript development pipeline posed a few more issues. The majority of these revolve around the fact that there are essentially two independent compilation processes, msbuild for the C# server side, and gulp for the javascript. Where previously these two sides were intertwined through the Telerik ScriptManager, now they are completely independent. Their independence makes front-end development easier and quicker, but of course, at some point they must come together. The critical point for this is where a page requests a javascript file. Our particular solution to this can be seen in the way MyCacheBustedpage.aspx uses the CacheBuster class to request the current merged, minified, cache-busted javascript file, which was produced by the gulp pipeline.

The end result of this work has largely been positive. Front-end development is generally easier, faster, with quicker feedback cycles whether working with javascript unit tests or viewing pages. Javascript development with Karma tests does not necessarily require Visual Studio at all. When developing and viewing pages Visual Studio may be running to serve up the pages with useful data, but the development environment itself can be anything (I frequently use vim), and gulp compilation on save is only a second or two long.

Installing yo ko

2015-03-17T00:00:00+11:00

If you have watched Steve Sanderson - Architecting large Single Page Applications with Knockout.js, you may be inspired to try out the Yeoman Knockout single-page application generator. However, if all the bits and pieces are not installed up on your machine, it may take sometime to sort through making it work.

The main challenges likely to be faced are having everything installed so that it ‘just works’. In sum, it is necessary to have: nodejs installed, git installed and in the path, Yeoman installed, the Yeoman Knockout generator installed, bower installed, and an http server available. If you have all that sorted out, it just might work!

Here is something of the procedure required on a machine I used to try it out.

Install nodesjs.
Ensure that git is somewhere in the path.

> git --version
git version 1.9.5.github.0

If git is not in the path, if it is already installed on the machine, add its location to the path env var. Note that if using github for windows, this may be of assistance in figuring out where it is. If git is not even installed on the machine (how could that be?), you could do a lot worse than GitHub for Windows.

Install Yeoman.

> npm install -g yo

Install the Yeoman Knockout generator.

> npm install -g generator-ko

Install bower.

> npm install -g bower

Create a single-page application website using Yeoman.

> yo ko

A simple web server is required, to simply server up the files. The nodejs http-server is a good alternative.

> npm install http-server -g

The dev version of the website should now be able to be run.

> http-server src

Compile the application.

> gulp

Now the release version of the website should run.

> http-server dist

Run the tests.

> karma start

Git and Firewalls

Firewalls are wonderful things, but if an error something like the following appears, it is likely you are behind a (corporate) firewall that does not have the default github port open.

bower ECMDERR       Failed to execute "git ls-remote --tags --heads git://github.com/millermedeiros/crossroads.js.git", exit code of #128

Additional error details:
fatal: unable to connect to github.com:
github.com[0: 192.30.252.131]: errno=No error

In that case, instruct git to use https instead of its default port.

> git config --global url."https://".insteadOf git://

Flyway Installation on Windows

2015-03-11T00:00:00+11:00

As the existing homebrew database migration tool started to show it age and certain limitations, our team recently went looking for a simple, existing solution to the sql database migration requirement. Directed to Flyway based on the Thoughtworks Tech Radar ‘Adopt’ recommendation, we set out to replace the existing tool.

Following outlines the steps required to get Flyway working on a windows machine, using windows authentication.

Prerequisites

Ensure java is installed. Test in a command line: java -version

Download Command Line Flyway

Download command line flyway and unzip it.

This outline is based on using: flyway-commandline-3.1.zip

Create a Test Database

Create a test sql server database in the default sql instance on your local machine. In this instance, the database was called: flywaytest

Configure Flyway

Go to the file: flyway-3.1/conf/flyway.properties And set the flyway url line like this: flyway.url=jdbc:jtds:sqlserver://localhost:1433/flywaytest

Test whether flyway is correctly working. The simplest way to do this is to run flyway (in fact, the flyway.cmd script): > flyway info

It is highly likely this will not work, resulting in something like the following error:

> flyway info
Flyway (Command-line Tool) v.3.1

ERROR: Unable to obtain Jdbc connection from DataSource (jdbc:jtds:sqlserver://localhost:1433/flywaytest ) for user
'': I/O Error: SSO Failed: Native SSPI library not loaded. Check the java.library.path system property.

Install ntlmauth.dll (if required)

If so, download the jtds driver distribution of the same version as the jtds driver found in flyway-3.1/drivers (in this instance, that was: jtds-1.3.1.jar).

See jTDS - SQL Server and Sybase JDBC driver

In this instance, the appropriate version was here

Unzip it, and find the ntlmauth.dll appropriate to the java being used on your machine. In this case, 32-bit java was used, so the dll was: jtds-1.3.1-dist\x86\SSO\ntlmauth.dll Place this file somewhere where it can be loaded by java, namely somewhere specified in the path environment variable.

Watch it run

Try running flyway, and this time it should work.

> flyway info
Flyway (Command-line Tool) v.3.1

Database: jdbc:jtds:sqlserver://localhost:1433/flywaytest  (Microsoft SQL Server 10.0)

+---------+-------------+---------------------+---------+
| Version | Description | Installed on        | State   |
+---------+-------------+---------------------+---------+
| No migrations found                                   |

If there are problems with the sql driver, note that there are (at least) two different sql server drivers for java. The one used here is the jtds driver. It is different from the Microsoft driver These drivers have different, but very similar, connection strings and other settings, so be sure to check carefully which driver you are reading about on stackoverflow etc.

Use it

At this point, flyway itself is installed and working. For details on actual usage of flyway, see the documentation.

FsUnit Data Driven Tests

2014-06-20T00:00:00+10:00

First things first, it was with no small joy that I discovered that that FsUnit plays nicely with one of my favorite testing tools, nCrunch.

Happily on my way developing F# with tests continuously running, it was not long before I was wanting to do data driven tests, and the syntax proved a small challenge. Unit Testing Sequences of Events using TestCaseSource, FsUnit, and Xamarin was a helpful resource. Based on that, the following example shows a relatively simple way to perform data driven tests, passing parameters of various types, including a list:

static member PlaceValuesTestData =
    [|
        [|10,9,[9]|];
        [|10,10,[1;0]|];
        [|10,11,[1;1]|];
        [|10,12,[1;2]|];
        [|10,123,[1;2;3]|];
        [|3,4,[1;1]|]
    |]

[<TestCaseSource("PlaceValuesTestData")>]
member x.PlaceValueTests (testData:(int*int*(int list))) = 
    let baseValue, number, expected = testData
    let result = placeValues baseValue number []
    result |> should equal expected

A couple of points worth noting:

The TestCaseSource (PlaceValuesTestData) is an array of arrays. Each inner array element is passed to the test function arguments.
Each of the inner arrays of PlaceValuesTestData is actually an array of tuples. This is not necessarily immediately obvious, because of the implied (missing) parenthesis.

The following from F# Interactive should make this clearer:

> (1,2);;
val it : int * int = (1, 2)
> 1,2;;
val it : int * int = (1, 2)  

So in the example above,

[|
    [|10,9,[9]|];
    ...
|] is the same as

[|
    [|(10,9,[9])|];
    ...
|]

That tuple, the first and only element of the array of parameters, is what is passed to the test function, which has one parameter, which is decomposed by pattern on the first line of the test function.

Denormalized Or Not?

2013-12-05T00:00:00+11:00

It is worth considering futher the nature of the following code, discussed briefly (as option 2) in the previous post:

class CustomerValueObject
{
	public string Id {get; set;}
	public string Name {get; set;}
}
class Invoice
{
	public string Id {get; set;}
	public CustomerValueObject Customer {get; set;}
	public List<InvoiceLine> Lines {get; set}
	public decimal Total {get; set;}
}

Furthermore, let us assume the existence of another entity, the Customer:

class Customer
{
	public string Id {get; set;}
	public string Name {get; set;}
	...
}

Invoice.Customer.Name is set to the name of the customer at the time the invoice is created. If the customer’s name is changed for any reason (correction of clerical entry error, name change by deed poll, etc), then Invoice.Customer.Name will be different to the Customer.Name. The question at this point is what the nature of consistency should be between these two fields. Are they consistent within themselves, permissibly inconsistent, or should they be eventually consistent?

Technically, the simplest solution is that Invoice.Customer.Name should never change its value. It does not represent the current name of customer.Id. Rather, it represents the name of customer.Id at the time the invoice was generated. It is the name that appears printed on the piece of paper/pdf/email. As such, it should never change. It is consistent within its own context, namely that of the Invoice, and it is quite acceptable for it to be different to TheCustomerAggregate.Name.

Alternatively, if the Invoice.Customer.Name represent the current name of the customer in the system, then it is clearly inconsistent. Maybe it should be eventually consistent, in which case it becomes necessary to write code to update the denormalized references to TheCustomerAggregate.Name. This is the sort of updating of denormalized data that is quite correctly associated with increasing the probability of defects.

The various issues raised regarding the Invoice.Customer.Name field are easily muddied when the answers to two different questions become merged together:

What does this field represent?
What is this field going to be used for?

Too often, it seems there is a desire to update Invoice.Customer.Name so that the Invoice aggregate/document can be used as part of a view model. This runs two concerns together, domain operations (which can mutate an aggregate, the command side) and read operations (the query side). In RavenDB we have found it easiest to delegate to read operations to an index, and not compel the aggregate to serve dual Command and Query purposes. Sometimes, it turns out that the name on Invoice.Customer.Name is the one required to be used in a view model (or data export, or whatever), and sometimes Customer.Name is the one required. Either way, it is best not to project the needs of what is likely to be many view models onto the data stored as part of the aggregate.

In the vast majority of cases, by carefully defining what a field means (usually as “the value of something at the time that an operation took place”), along with the the fact that very many entities in a system rarely if ever change (for example, currency codes, port codes) our solution is entirely devoid of the need to update denormalized fields (going by memory here, there might be a case somewshere by I can’t recall it). While it is not necessarily going to be the case for all domains, much of the time in business type applications, data which can be viewed as denormalized is more appropriately understood as values at the time of some operation. Consequently, the problems sometimes envisaged with having to update denormalized data simply do not arise, because the data is not, in fact, denormalized.

Aggregates and Documents

2013-12-04T00:00:00+11:00

The most effective principle we have found in determining what should be contained in a RavenDB document is to follow the principles around DDD aggregates. An aggregate is a collection of objects, with one of the those objects as the root, which represents the smallest collection of objects that must be kept consistent at a time. Typically, an aggregate corresponds to the extent of data modified in a single transaction.

Take the following example:

class Invoice
{
	public string Id {get; set;}
	public List<InvoiceLine> Lines {get; set}
	public decimal Total {get; set;}
}
class InvoiceLine
{
	public string ItemDescription {get; set;}
	public decimal Amount {get; set;}
}

It is likely that Invoice.Total should always be consistent with Invoice.Lines.Sum(l => l.Amount). This should not be eventually consistent, it should be immediately consistent. Other parts of the system may be making business decisions based on Invoice.Total, and it will not do for them to be making decisions based on an incorrect total.

Consequently, Invoice will be an aggregate, and a single invoice will end up in a single document.

Invoice i = new Invoice()
{
	...
	Lines = ...
}
ravenSession.Store(i);

Things get more interesting if we consider what happens if we consider that the Invoice belongs to a customer. There are at least three possibilities for how the invoice is related to the customer.

Use a Customer key

class Invoice
{
	public string Id {get; set;}
	public string CustomerId {get; set;}
	public List<InvoiceLine> Lines {get; set}
	public decimal Total {get; set;}
}

In this case, CustomerId is the document key for whatever the Customer entity is. This obviously corresponds quite closely to how the situation may be envisage if using a relational database.

Use a Customer Value object

class CustomerValueObject
{
	public string Id {get; set;}
	public string Name {get; set;}
}
class Invoice
{
	public string Id {get; set;}
	public CustomerValueObject Customer {get; set;}
	public List<InvoiceLine> Lines {get; set}
	public decimal Total {get; set;}
}

In this case, the CustomerValueObject is a snapshot of the Id and Name of the Customer at the time the Invoice is created. If the customer’s name changes, it would need to be determined whether it is necessary to update any of the corresponding denormalized CustomerValueObject.Name properties. While I have suggested that the focus of a document should be on issues of mutation and consistency, one advantage of this design is that if you have a collection of Invoices and want to use them for a view model, it is not necessary to hit the database to get the customer’s name (assuming for the sake of the example you want to show a list of invoices and the name of each invoice’s customer).

Note that exactly what CustomerValueObject is, is not the important point here. The main point is that it is not an instance of Customer, but rather in most cases a subset of the Customer object. On our project, we have settled for using something like this for virtually all references:

class EntityReference
{
	public string Id {get; set;}
	public string Name {get; set;}
}

Make Customer the aggregate, including the invoices

class Customer
{
	public string Id {get; set;}
	public string Name {get; set;}
	public List<Invoice> Invoices {get; set;}
}

This is the sort of design that I would suggest may be problematic. It’s main apparent advantage is that it is possible to hit the database once, get a single document, and display every invoice that a customer has ever had. Furthermore, if the customer’s name changes, it avoids the (potential) problem of solution #2, that a denormalized customer name becomes inconsistent. The disadvantages are:

If there is no consistency requirements between the Customer and the customer’s invoices, then this does not represent a valid aggregate. One of the reasons this may cause problems is not immediately clear in this highly-contrived example. However, if you imagine a system where there are numerous additions and edits of invoices happening very quickly, the extent of this aggregate is going to cause problems (if we assume that only one process gets to update an aggregate at a time). Everytime any invoice for a specific customer is edited, or a new one added, the entire Customer object for that customer is effectively locked.
If there are a lot of invoices for a customer, then the document may end up becoming unmanageably large, even a performance problem.
If this is being done primarily because it is influence by a particular view model, then the document runs the risk of moving in the my-entity-database-behaves-more-like-a-cache direction. A document that is essentially a view model is likely to be problematic when today’s view model changes tomorrow, requiring slightly different data.

An easy question to ask one-self is this: am I deciding on the document (aggregate) boundaries on the basis of consistency requirements, or on the basis of a view of the data (for example, a view model required for a page). If the former, you are likely on the right path. On the other hand, if the extent of the document is determined by the fact that there exists some page in your application that shows some amount of data and it is possible to put all that data into a document, and hence, achieve 1 page = 1 db hit, then you may want to carefully consider the disadvantages.

A Note on Aggregates and Domain Driven Design

Aggregate is a technical term in Domain Driven Design (DDD). The definitive explanation is found in Eric Evans, Domain-Driven Design . An excellent worked example is found in Vaughn Vernon, Implementing Domain-Driven Design, (chapter 10 on Aggregates is one of the best in the book). For a taste of Vernon’s book see his articles, Effective Aggregate Design, which were subsequently revised and turned into the Aggregates chapter in his book.

It's In The Way You (Ab)Use It

2013-11-28T00:00:00+11:00

Databases can be used and abused. It’s nice to use them as intended, but can be quite fun to abuse them too, if you know what you are doing. For example, if I create a table in a relational database that looks like this:

CREATE TABLE [dbo].[KeyValue](
	[Id] [int] NOT NULL,
	[SerializedData] [nvarchar](255) NOT NULL
) ON [PRIMARY]

And write code like this:

class Invoice
{
	public int Id {get; set;}
	public int CustomerId {get; set;}
	public int ItemId {get; set;}
	public DateTime Date {get; set;} 
	public string Description {get; set;}
}

{
	...
	Invoice invoice = new Invoice() {
		CustomerId = customer.Id,
		ItemId = item.Id,	
		Date = Datetime.Now,
		Description = "Test Invoice"
	};
	SaveSerializedInvoiceInKeyValueTable(invoice);
}

Aside from having some very average code I am not going to be able to take advantage of the natural features in a relational database. Searching will be, well, problematic, and joining between tables — its just not going to happen. Nothing stops me using the database in this way, but really, if I want a key-value database I may well be better off using a key-value database like Riak or Redis.

The moral of this little illustration is, of course, that whatever the type of database being used, if you build a key-value store, you will end up with something that looks and behaves like a key-value store. If you build a cache, you will get a cache.

What is less clear is that while this may happen by deliberate design, it may also happen unintentionally, simply because the database is not being used as its design intended.

To help understand how document databases are typically intended to be used, a comparison between key-value databases and document databases may be useful. Both types are databases can be broadly classes together as aggregate based databases (Here, I am drawing on terminology from Sadalage and Fowler’s book NoSQL Distilled. This means that both types of databases use a key to lookup some aggregate value, potentially stored as JSON as is often the case. It is in the differences between the two types that an indication as to how a document database should be used becomes more apparent, and here I quote:

“The two models differ in that in a key-value database, the aggregate is opaque to the database – just some big blob of mostly meaningless bits. In contrast, a document database is able to see a structure in the aggregate. … A document database imposes limits on what we can place in it, defining allowable structures and types. In return, however, we get more flexibility in access.” NoSQL Distilled, p. 20

Practially, the means by which the structure of data in a document database is taken advantage of is by some means of grouping together like documents, which in RavenDB (and MongoDB) is called collections. Documents in a collection have the same, or a similar enough structure that indexing of some sort can be performed, and they can be combined with other documents, potentially from other collections, in predictable ways, allowing for putting documents together (the closest idea to a relational database’s join you are likely to find) and querying.

So, if you are using RavenDB and the documents in a collection start to look like a blob of meaningless data, you might find that you are accidently implementing a key-value database in a document database. It may, in fact, look like little more than a cache. In practice, if you are using the C# client to interact with RavenDB then you are unlikely to fall into this trap. However, it is also possible that if your aggregate (persisted document) is too big that you may find the document database starts to feel more like a key-value store.

On our project, in order to get the best results out of RavenDB, we have found it best to keep to the principle that a document typically corresponds to an aggregate (the mininum amount of data that must be kept in sync at any given time – essentially the Domain Driven Design definition, see Evans and Vernon). This is not necessarily the principle that everyone using RavenDB is following, nor is it necessarily a rule/law we always follow (I used the word ‘principle’ for a reason). However, for our program, a freight forwarding application following well defined business processes, it has served us well.

Why You Should Consider Using RavenDb

2013-11-27T00:00:00+11:00

Why You Should Never Use MongoDB is a detailed post describing the experiences of the open source Diaspora project with MongoDb. I do not think it would be a spoiler for anyone if I tell you that it didn’t turn out too well for them. For myself, the outcome seemed nearly inevitable somewhere during the discussion of what constitues a document, with the large scope of data that a document held.

What was a little more difficult to see coming was the various conclusions the article came to. Most notably, I would highlight these:

The only thing it’s (MongoDb) good at is storing arbitrary pieces of JSON. “Arbitrary,” in this context, means that you don’t care at all what’s inside that JSON.
I’ve heard many people talk about dropping MongoDB in to their web application as a replacement for MySQL or PostgreSQL. There are no circumstances under which that is a good idea. Schema flexibility sounds like a great idea, but the only time it’s actually useful is when the structure of your data has no value. If you have an implicit schema — meaning, if there are things you are expecting in that JSON — then MongoDB is the wrong choice.

I do not have any production experience with MongoDB, but I do have experience with RavenDB. It is a document database and as such, has a number of fundamental similarities to MongoDB. Certainly, at a high level it is possible to read the author’s critique and conclusions as applying equally to RavenDB.

It would be fair to say that neither the author’s description of how MongoDB was used on this project and problems they ran into, nor the conclusions, rang true with my experience of RavenDB. By the time I was reading the conclusions, it was almost like reading about another kind of database (and maybe I was - it sounded a lot like a key-value store to me).

However, it is not my purpose primarily to discuss the article in detail. One of the strengths of the article was it’s form, that of a case study. As such, I think it appropriate to offer up a counter case study of a project using a document database that did not end up a train-wreck.

Over the past year or two, I have worked on a freight-forwarding application that used RavenDB. The database stores data related to costing including buy rates, sell rates, and standard tariffs. This data is used for such purposes as determining the cost and sell amounts of a shipment, and marginal variations in quotes.

It would be fair to say that for the developers on the team, most would tend to prefer using RavenDB over a relational database. That said, lest one peg us as RavenDB zealots, for our most recent greenfields project we chose to use a relational database.

Furthermore, when I look at our database I simply do not recognise the (mis)characterization that each document is nothing more than an arbitrary piece of JSON. Somewhere between the outcome of the Diaspora project and the conclusions drawn from it something went wrong.

There is one thing that I suspect we would all agree on, that choosing a database for a project is a matter of choosing one appropriate to the problem at hand, be that a relational database, a document database, graph database or something else. Where we would disagree is over appropriate uses of document databases. I certainly am not claiming RavenDB as an appropriate choice for the Diaspora project. I do, however, find the conclusions of the article problematic, as they contradict my own happy experience using RavenDB, which I will present in the upcoming series of blog posts.

Domain-Driven Design, What is it?

2013-03-19T00:00:00+11:00

For many people, Domain-Driven Design is closely associated with entities, value objects, aggregates, factories, repositories - “The Building Blocks of a Model-Driven Design” (Evans, Pt 2). However, these building blocks arise within the context of what may be considered the guiding principle of DDD, a Ubiquitous Language. Vernon’s Implementing Domain-Driven Design effectively reverses Evans’ order of dealing with DDD concepts, in dealing with the strategic patterns first (bounded contexts, core domain, sub-domains, …), then the tactical patterns (entities, value objects, aggregates, …). But like Evans, Vernon closely ties his discussion with the concept of a Uniquitous Language.

As Vernon notes, DDD “is essentially a large pattern language” (Vernon, p. xxxv). The Gang of Four popularised the design patterns idea with their seminal Design Patterns book, in which a large number of relatively low level design patterns are presented. Fowler’s Patterns of Enterprise Application Architecture presented a number of higher level design patterns. In that tradition, Evans’ DDD may be thought of as design patterns for complex software.

Individually, most of the patterns in DDD have been described before. What makes Evans’s treatment of them significant is that they are drawn together within a broader context that offers them a direction and precision they lack individually.

When attempting to understand DDD, it is helpful to hold this broader context in mind. DDD is a collection of patterns appropriate for building complex software. The principle integrating these patterns and guiding their usage is that of a Ubiquitous Language.

Review, Implementing Domain-Driven Design, Chapter 1

2013-02-19T00:00:00+11:00

In the beginning, we had the Big Blue Book (Eric Evans, Domain-Driven Design). Now, we have the Big Red Book (Vaughn Vernon, Implementing Domain-Driven Design). Vernon’s goal is to provide a somewhat more practical guide to using Domain-Driven Design. Where Evans’ book is quite a heavy read, Vernon’s book is likely to be more accessible, both in its approach to the content and in style.

Structurally, the Big Red Book takes a major departure from Evans book, in presenting the strategic patterns discussed in the final part of Evans book first, before moving on to the tactical patterns that get discussed earlier (primarily in part 2) of Evans book. For many people, this means that many of the tactical patterns that they primarily identify with domain driven design (aggregate, value object, entity, repository, factory) are discussed after the lesser known strategic patterns (bounded context, context map). I expect that this will have the effect of elevating the strategic patterns to the more prominent place they deserve in people’s thinking about DDD.

Chapter 1 addresses some of the high level questions around domain-driven design: why and when you should use domain-driven design, a basic introduction to ubiquitous language with and its connection to the code, and the business value of domain-driven design.

One part that especially stood out to me was on when to use domain-driven design. Vernon gives a scoring table against which you measure your project, and if you score greater than 7 suggests that you consider using DDD. For instance, having 30-40 user stories/use cases, will score you 2 points. Variations on your system growing over the years, in various/unexpected ways score 3,4 or 5 points. Suffice to say, I think every project I have ever worked on would score more than 7 points. Perhaps that should not be a surprise, but depending on what wisdom you read elsewhere, a good number of people seem to suggest that DDD is only applicable to a small number of the largest and most complex projects.

The material through the book is closely tied to a specific example which is used to illustrate the various aspects of DDD, namely a corporate collaboration suite together with a scrum management product. I am especially interested to see how this approach works out. One of the recurring issues found in discussions about DDD on the internet is the request for examples and/or samples by people starting out trying to learn the DDD approach. More often than not this seems to be met with the response that anything requiring DDD is so complex that a sample will not illustrate what is required, or the provision of a any sample is roundly bagged for either not being a complex enough domain, or adding needless infrastructural complexity. I know that I experienced similar frustrations myself when starting out with DDD, and there is only so far you can get with questions on forums without some sort of more overarching guidance. With any luck, Vernon’s use of an example throughout a book of this length (611 pgs) will steer the right course.

Design Patterns in Testing, the Builder

2012-11-01T00:00:00+11:00

Following on from Ayende’s recent post on the builder pattern in his design patterns series, I thought it would be interesting to discuss one of most useful purposes I have recently experienced the builder pattern used for.

Integration tests typically require putting the system into a certain state to perform the test. Baring certain minimal data that may be assumed to be pre-existing in a database, the setup can still amount to a considerable amount of work. Having a collection of builders to produce sample aggregates within a system can assist greatly in this task.

For any given aggregate, it should aways be possible to do something like this:

var quote = new QuoteBuilder().Build();
// Perform test for which a quote is required.

Doing so would result in a valid quote being created. The state of that quote is completely irrelevant. The sole guarantee of the builder is that by default, it will build a completely valid aggregate without having to fuss over any details at all.

Builders also allow for the setting of state when that is important to the test. For instance, if the customer with specific properties was required, the following might be done:

// Arrange
var customer = new CustomerBuilder().SetOverCreditLimit().Build();
var quote = new QuoteBuilder().SetCustomer(customer).Build();
// Act
// perform test on the quote where the customer is of significance to the test.

Although this can result in builder setter functions which do little more than setting a value within the aggregate, the benefit is that only properties that matter to the test appear in the test. Consequently, the nature of what matters to the test is evident, simply by reading the test.

Sometimes, more complicated relationships exist between aggregates. For example, to setup a quote which could be used for a particular shipment it might be necessary to do something like:

// Arrange
var shipment = new ShipmentBuilder().Build();
var quoteMatchingShipment = new QuoteBuilder()
    .SetCustomer(shipment.Customer)
    .SetValidFrom(shipment.Eta.AddDays(-1)
    .SetValidTo(shipment.Eta.AddDays(1)
    .AddTradelane(shipment.Origin, shipment.Destination);

Consequently, it becomes useful to establish relationships between aggregates more simply via methods that are scenario based. For instance:

// Arrange
var shipment = new ShipmentBuilder().Build();
var quoteMatchingShipment = new QuoteBuilder().Match(Shipment).Build();
// Act
// perform test where it is required for the quote to apply to the shipment.

The stateful nature of the builders can make for some interesting patterns. For example, if one wanted a quote which matched a shipment in all ways except for the customer, the following is possible:

// Arrange
var shipment = new ShipmentBuilder().Build();
var quoteNearlyMatchingShipment = new QuoteBuilder()
    .Match(shipment)
    .SetCustomer(new CustomerBuilder().Build())
    .Build();
// perform test that makes sure that a quote matching in all regards 
// except customer is not applied to the shipment.

This relies on the fact that setting the customer after matching the shipment clobbers the customer match that was set-up as part of .Match(shipment).

Some of the benefits in having aggregate builders for integration tests include:

it simplifies putting the system into a suitable state for the test.
tests are setup in consistently.
test readability is improved, because only things that matter to the test appear in the test.

However, the biggest benefits arise from the way it compels developers to think about their testing. Assuming the aggregates are correctly identified with respect to the business, the test setup using builders is easily envisaged as closely corresponding to what actually happens in the business. This stands in contrast to setup which appears to have little correspondance to the business, or at least, where that correspondance is hidden behind a wall of property setting code that obscures what is the essence and what is incidental to the test.

Conceptual Models

2012-08-30T00:00:00+10:00

‘Conceptual models’ is one of the core ideas discussed in Donald Norman’s book, The Design of Everyday Things. A conceptual model is the mental model a person forms that relates their interaction with an interface to the underlying thing the interface controls. It is the explanatory framework a person has to explain how a control operates something.

One example Norman gives is that of thermostat controls. A thermostat interface presents the user with a range of temperatures over which to set the control. However, the interface itself does not explain how the thermostat controls the heating device.

The problem is illustrated well by a common misconception about thermostats. If a room is very cold and I want to heat it quickly, does setting the temperature very high, much higher than I want it to be, cause the heating device to work harder and heat the room faster or not? For most thermostats, almost certainly not. While it might be a popular enough conceptual model, most thermostats are simply a on-off switch. When the temperature gets too low, the thermostat turns the heater fully on. When the temperature gets too high, the thermostat turns the heater fully off. For anyone with a bit of engineering knowledge, the conceptual model may well extend to include an understanding of how an electrical control system uses an hysteresis loop to keep the temperature revolving within a certain range.

As this very simple example shows, an interface is not the same thing as that which it controls. More troubling, even a very simple interface, which one may initially consider to be virtually self-evident in what it does, is open to interpretation - and misinterpretation.

What ever the outcome of a person’s interaction with a control, there is one thing in common between the person who correctly interprets and the person who incorrectly interprets what the control does: both are seeking an explanation for their interaction with the device. The device, however simple is not self-evident and requires interpretation. Human beings, by their very nature, seek explanations that order the world around them. People will form explanations, rightly or wrongly, given any amount of information provided to them. Whether that information is sufficient or insufficient, helpful or misleading, people will create conceptual models in their interactions with a device.

If this is so for the humble thermostat, one can imagine how much more complicated things get when dealing with even a relatively simple computer program’s user interface, let alone a complex one.

As software developers, it is important to recognise that people will develop conceptual models of what is happening in the software, whether we like it or not. The only question is, does the software operate in such a way as to guide people’s conceptual models correctly. Does it reinforce their correct conceptions, and effectively bring into question their incorrect conceptions? How to go about that forms the bulk of the content in Norman’s book. To appreciate the significance of that content, however, it is important to recognise the existence and significance of things like the conceptual model. It is, after all, incredibly easy to fail to recognise the interpretive effort going on in the mind of a user, even for the very simplest of user interfaces.

Review of Donald Norman, Design of Everyday Things

2012-08-15T00:00:00+10:00

Donald Norman, The Design of Everyday Things.

This is a book about opening doors. It is a book about setting the temperature of your fridge. It is about using your company phone. It is about turning lights on, and it is about turning taps on. Sounds like a real page turner, doesn’t it?

Have you ever been annoyed when pushing on a door to discover it doesn’t work, only to pull on it to discover it still does not work, before repeating the process on the other side of the door? Ever had trouble figuring out the temperature controls in a combined fridge/freezer? Do you find it hard to remember the various obscure key combinations to operate the company phone? Ever wasted time trying to figure out which of the six light switches in a row is the one you want to turn on? Ever had trouble with some obscure kind of tap, either trying to turn it on, or determine which way is hot or cold? Ever struggled to figure out how to interact with some piece of software? If so, then The Design of Everyday Things is likely to offer you at least some consolation, with the message that the problem is not (usually) you, but most likely the lack of thought given to the design.

Donald Norman’s The Design of Everyday Things (formally published as The Psychology of Everyday Things) is something of a classic, dealing with human interactions with everyday items in life. It includes a wide range of examples, from the humble door and tap, through to somewhat more complex devices such as corporate telephones.

When it comes to actual examples, the book certainly shows its age (originally published in 1988), with the computer coming in for quite a bad rap. But of course, much has changed since then and the good news is that at least some of today’s user interfaces begin to approach Norman’s glorious vision for a future computer (p. 185f, ‘The Invisible Computer of the Future’).

In an odd sort of way, the various examples given in the book are potentially its biggest, if not exactly weakness, temptation perhaps? For the examples are so entertaining in their own right that they risk drawing all attention away from the governing design principles Norman is seeking to present. It is the importance of these principles that means the dated nature of a few of the more technologically oriented examples are of little consequence to the ongoing significance of this book. Key ideas such as afforance, conceptual model, visibility, mapping, feedback, constraints, human memory, are the underlying principles drawn out through the examples, good and bad, to advocate what Norman calls ‘user centered design’.

Ultimately, Design of Everyday Things is worth reading just for the examples. It is far more when one thinks through the examples to the principles espoused in user centered design. While not directly about software development, it is well and truly a worth-while read for anyone designing user interfaces. Moreover, the principles apply just as well to anyone programming an interface to be consumed by another person, be it a user interface or a programming interface. And these days, that is practically every programmer.