Los Techies

Happy Hack-o-ween: Electronics and a microcontroller spice up the haunt

Sharon J. Cichelli — Sun, 08 Nov 2009 16:27:00 GMT

Ah, Halloween, when a young woman's fancy turns to love. And zombies.

I had two personal requirements for the costume I would build this year:

It shall be spooky.
It shall blink.

I'll tell you about the final result, the electronics and the software that went into it, plus the techniques I used to achieve wearable electronics. I'll introduce you to the Arduino, an open-source microcontroller prototyping platform, which is an exhilarating tool/toy for making your software skills manifest in the physical world.

My husband and I have been teaching ourselves electronics. A few months ago, Dad taught me to solder. I recently read Syuzi Pakhchyan's excellent primer on wearable electronics and smart materials, Fashioning Technology. And I've been making things with my Arduino. All these ideas were swirling and combining in my head to inspire this year's Halloween project. Er, costume. Same difference.

What was I? I was a nightmare... the thing under your bed... the reason for your well developed sense of paranoia...

I sported a crop of writhing eyeballs erupting from my head. Each eyeball has an LED inside it, and they blink randomly and independently, until I trigger a hidden switch, which causes the blinky ones to go dark and two red eyes to pulse menacingly. In the Flickr photoset, you can see the construction process.

The Arduino Sketch

The term "Arduino" is overloaded to mean:

a particular chip and circuit board which you can buy or build;
the IDE in which you write programs for the chip;
the language, which is C-flavored;
fun.

Arduino programs are called sketches. Every sketch must contain two functions: setup (runs once) and loop (runs continuously). Here's my sketch, with extra explanatory comments, that blinks the six regular eyeballs and responds to the switch by pulsing the red eyeballs.

    1 #define SWITCH 8

    2 int ledPins[] = {2, 3, 4, 5, 6, 7};

    3 const int ledPinsCount = 6;

    4 int redEyePins[] = {10, 11};

    5 const int redEyePinsCount = 2;

    6 long durations[ledPinsCount];

    7 int ledStates[ledPinsCount];

    8 long previousTimes[ledPinsCount];

    9 int i;

   11 void setup()

   12 {

   13   pinMode(SWITCH, INPUT); //Specify the switch pin as an input.

   15   for (i = 0; i < redEyePinsCount; i++)

   16   {

   17     pinMode(redEyePins, OUTPUT); //Specify each red-eye LED pin as an output.

   18   }

   20   for(i = 0; i < ledPinsCount; i++)

   21   {

   22     pinMode(ledPins, OUTPUT); //Specify each regular LED pin as an output.

   23     ledStates = random(1); //Randomly set the LEDs to on or off (1 or 0).

   24     durations = GetRandomDuration(); //Define a random duration for each LED to stay in that state.

   25     previousTimes = 0; //At time of setup, the "last time we changed" is at 0 milliseconds, the start of time.

   26   }

   27 }

   29 void loop()

   30 {

   31   if (digitalRead(SWITCH) == HIGH)

   32   {

   33     TurnOffLeds();

   34     PulseRedEyes();

   35   }

   36   else

   37   {

   38     for(i = 0; i < redEyePinsCount; i++)

   39     {

   40       digitalWrite(redEyePins, LOW); //Turn the red eyes all the way off.

   41     }

   43     for(i = 0; i < ledPinsCount; i++) //For each LED:

   44     {

   45       if (millis() - previousTimes > durations)

   46       {

   47         ChangeLed(i); //If this one's duration is up, then flip it.

   48       }

   49     }

   50   }

   51 }

   53 void TurnOffLeds()

   54 {

   55   for(i = 0; i < ledPinsCount; i++)

   56   {

   57     digitalWrite(ledPins, LOW);

   58   }

   59 }

   61 void PulseRedEyes()

   62 {

   63   //Fade on, then off.

   64   int j;

   65   for(j = 0; j < 255; j+=5)

   66   {

   67     for(i = 0; i < redEyePinsCount; i++)

   68     {

   69       analogWrite(redEyePins, j);

   70       delay(10);

   71     }

   72   }

   73   for(j = 255; j > 0; j-=5)

   74   {

   75     for(i = 0; i < redEyePinsCount; i++)

   76     {

   77       analogWrite(redEyePins, j);

   78       delay(10);

   79     }

   80   }

   81 }

   83 void ChangeLed(int ledPin)

   84 {

   85   previousTimes[ledPin] = millis(); //Update the "last time we changed" to now.

   86   durations[ledPin] = GetRandomDuration(); //Give it a new random duration.

   87   ledStates[ledPin] = 1 - ledStates[ledPin]; //Flip the state between on and off.

   88   digitalWrite(ledPins[ledPin], ledStates[ledPin]); //Set the LED to that state.

   89 }

   91 long GetRandomDuration()

   92 {

   93   //Random number between 1 and 10, then multiplied by 400 to give it a detectable duration.

   94   return random(1, 10) * 400;

   95 }

I like the way the eyes blink independently. If they all flashed in unison, they would look like Christmas lights, and you would notice that two were "special" because they weren't flashing. Instead, the duration that any given eyeball is lit or dark constantly changes.

The blinking is managed by a collection of arrays. One array represents each of my LED pins, so that I can address them in a for loop. The three other arrays hold: the state (on/off) of each LED; the duration each LED should stay in that state; the reading from the millisecond counter when the LED last flipped its state. Each time the loop function executes, if the switch is not connected, then I look at each LED; if the difference between the current time and the time when it previously changed is greater than its duration, flip its state (from off to on, or from on to off), randomly assign it a new duration, and record "now" as the new "previously changed" time. If the switch is connected, then I make the red eyes fade on and fade off.

Fading with PWM

PWM (Pulse-Width Modulation) is a technique for making a digital component (one that turns on or off) simulate analog behavior (be a little bit on, and then a little bit more on). If you turn an LED off and on really quickly, you won't perceive the flickering, but it will look half as bright, because it is actually off for half the time. If you let it spend a little more time off than on, it will appear even dimmer. So by varying the width of the pulses, you can control how bright the LED looks.

The Arduino comes with built-in PWM functions; some pins are already set up to be PWM pins. If you plug an LED into one of the PWM pins, then you can write to it as if it were an analog component. That's why, in my sketch above, I set the brightness of the red eyes using analogWrite(), instead of digitalWrite(). My for loop increments the counter j from 0 to 255, and sets the brightness of both red eyes to the value of j. The Arduino takes care of (imperceptibly) flickering the LEDs with the right ratio of on-time and off-time to achieve a j amount of brightness. So the eyes get gradually brighter, then gradually dimmer. (Then control returns to the main loop function, but if my switch is still connected, the red eyes will throb again.)

Snaps: Wearable Plugs

A metal sewable snap is like a plug for your clothing, an interface between the world of textiles and the world of wires. This is handy when you need the electronics to be separate while you are getting into the clothing, or if you want to wash the clothing. My Arduino hung out at the base of my neck, to be near the LEDs on my head but hidden underneath my wig, but my control switch was near my hip. I could have run a wire down to the switch, but conductive thread was more subtle and more comfortable.

To complete the connection, I soldered a short wire to one side of the snap. That wire plugged into a pin on the Arduino. The conductive thread ran from the switch at my hip up to the back of my dress near the Arduino, and I sewed that conductive thread to the other half of the snap. When the two halves are snapped together, the wire and the thread make a complete connection, as if they were one continuous wire.

I had two threads (going out to the switch and back), so I encased them each in a bias tape tube, to prevent them from touching each other and shorting out.

I've been saying "switch," but actually, I simplified at the 11th hour. I tied each thread around a safety pin, and stuck the pins to my dress. When the safety pins touched each other, they completed the circuit, which the Arduino sketch interpreted as triggering the switch—cue red-eye glare.

What's Next

Soldering and sewing are both liberating skills to possess—they free up your creativity to make wilder and more integrated stuff. If you are currently proficient with only one, ask around and see if you can find a buddy who's good at the other, and teach each other.

The Arduino comes with a great community of hackers and makers, lots of people to learn from and collaborate with. Definitely check it out. There is lots of fun to be had, and blinking LEDs is the barest beginning of what it can do.

Using MSDeploy to automate your Enterprise Application remote deployments.

erichexter — Fri, 06 Nov 2009 17:00:00 GMT

MsDeploy is a newish technology that is a bit schizophrenic. What I mean is that it is a tool that is useful to both Developers and Administrators but it is not clear from the documentation how to best use the technology and how to approach it. I believe it stated as a Server Administrator tool and the team was able to work in an integration with Visual Studio which made it into a more robust framework, but at the same time left is command line interface with so many options that it is challenging to get a grasp on. To top that off the Web Platform Installer uses the MsDeploy packages as a way to distribute packages through that tool. After working through some various ways to use the technology I have settled into some commands that work well for our projects.

Let me set the context of how we are using the tool and what we already had in place so that you can better understand if my use of MsDeploy will work for you. At Headspring we use Continuous Integration on all of our projects and as a result, once our software is built by our build server the next logical step is to deploy our software to a server and than run User Interface tests against the application. Most of our projects are web applications running on ASP.Net MVC using Sql Server as the back end. We install our software using a lightweight zipfile that contains the web application files, database migration scripts and a deployment script which can poke config files and execute our database migration tool. We already have all the pieces in place to deploy our application on a local machine.

Up until know we have used two methods to deploy instances to remote machines. The first methods was to install Cruise Control.Net on the server and have it monitor our source control repository for a new installation package being committed to the repository. Once it sees a new package CCNet will pull down the package, install it locally and go on its merry way. The second method, is to kick off the deployment from the build server and connect over the network to the target database to run upgrades and then xcopy the files to a unc share. Both of these methods require setting up some configuration on our target servers. My goal with our deployments is to reduce the amount of per server configuration we do on each server and use some conventions to make each server look similar to one from a different project.

Using MSDeploy allows us to do the following.

1. Remove the need to install cruise control on the target server and update the configuration for cruise control on each target server. We do have to install MSDeploy on each server but we do not have to mess with any configuration after that.

2. We do not have to mess with setting up unc shares and deal with the mess. It sounds like a silly thing but by getting away from the unc share we can also test our deployments from any machine and msdeploy is actually firewall friendly. xcopy to a unc share is not firewall friendly which means that we cannot use it for all of our clients which means variations between our projects.

3. We use msdeploy as a mechanism to distribute our deployment packages and then remotely execute the packages. This means I can have a single instance of our Continuous Integration server which reduces the number of places to maintain configuration. That is a big win. This also means the log files for all of the deployments can be tracked in a single place.

4. Another benefit of using msdeploy to push our deployments means that I can easily setup new instances of a test configuration and push it to multiple servers without having to log into each machine.. This is good for efficiency.

Our use of MSDeploy now boils down to two steps. Distribute and Execute. We have some of our scripts in NAnt and we are in the process of migrating to PowerShell now that version 2.0 is available from the older operating systems. Below is a sample of executing msdeploy from a NAnt script.

Calling MsDeploy from Nant

The dirPath command tells MSDeploy to synchronize a directory from the source computer a target computer. This is a pretty easy command to understand.

The second command is the runCommand this command was added between the RC and 1.0 release of MSDeploy and I am so happy they added it. The run command is told to execute a command on the remote machine. Since we are running installation scripts, they do not execute instantly and as a result the waitInterval and waitAttempts need to be specified so that the command does not timeout before it has completed running. other than that the output of the console application is piped back to the source computer. The one caveat about the run command is that when it starts it runs from the C:\windows\system32 directory. In order to work around this issue I have found that passing the directory of the command into the batch file that I run allows my batch files to first cd to that directory as its first step. This is a pretty harmless thing to do and works pretty well.

This is what a sample deployment batch files looks like.

cd %1
rd ..\codeToDeploy\ /s /q
applicationNamePackage.exe -o..\CodeToDeploy\ -y
cd ..\CodeToDeploy\
cmd /c %systemroot%\system32\inetsrv\appcmd stop site applicationName_dev
iisreset
call dev.bat
cmd /c %systemroot%\system32\inetsrv\appcmd start site applicationName_dev

This is pretty basic and does some IIS commands as well.

I am sure I left some information out, but I wanted to get a brain dump of our use of MSDeploy.

Long term I would like to see the use of PowerShell driving msdeploy and adding some configuration around each Server Role in an application and tie it to the servers needed for each environment. I have started a project to put this together called psTrami but I have not put any of the code together yet, just some small spikes to prove it out.

More to come sometime soon…..

Hang in there baby birds, I'll feed you

Scott C Reynolds — Thu, 05 Nov 2009 20:19:29 GMT

Quick housekeeping note - How We Do Things series is not cancelled, I haven't given up on it or run out of steam, it will still press forward.

Trying to get a lot wrapped up both at home and at work in preparation for my move to New York, so I lapsed a bit, but I will be back on it soon.

Automapper Auto Profile Registration.

erichexter — Thu, 05 Nov 2009 03:24:15 GMT

On some of our projects we have been experimenting with smaller AutoMapper profiles. The idea is that it is easier to digest a smaller profile. We have gone so far as creating a profile for each Domain object and handle all of the mappings to and from the domain object. We are also trying out a Profile per scenario. While these smaller profiles are easier to dig in and understand, the registration of them are a little painful. So I put together a quick way to auto register all the profiles for automapper. Below is the code to discover all the profiles in an assembly than register them with AutoMapper. There is nothing fancy here and I could certainly spend more time making it better performing, but realistically this is startup code that runs once at application start up. That being said I would rather focus performance optimization efforts on places that actually make a difference to the End User Experience.

Here is the sample.

It is important to know that we have full code coverage over our application including integration tests, so that if something were to break as a result of loading this in a non deterministic order, we would know before we commit our changes to source control.

Here is the code for the ForEach extension method, since I breezed over it.

This is a smaller post than I normally put together.. is this small of a post useful?

A Time For Change

derick.bailey — Wed, 04 Nov 2009 15:36:03 GMT

Change comes in groups of three, right? Well, I have a 4 week old baby girl (I can hardly believe she’s 4 weeks old already). That would be change #1. For change #2, I turned in my two week notice to my current employer on Monday.

Starting on November 16th, I’ll be joining up with fellow LosTechie, Joey Beninghove, over at TrackAbout.com. I’m excited about the opportunity. There’s a great bunch of developers on that team and it should be a lot of fun and a significant challenge for me, at the same time.

Now I just have to wait and see what change #3 is going to be…

ASP.Net MVC Portable Area – Part 4 IoC framework support.

erichexter — Wed, 04 Nov 2009 05:00:00 GMT

Part 1 – Introduction
Part 2 – Sample Portable Area
Part 3 – Using of a Portable Area
Part 4 – Using an Inversion of Control Framework. (this post)

Using an Inversion of Control Container is a common scenario that would be needed by a Message Handler. The Bus has an extensibility point to create the handlers. The model for this should feel similar to the ControllerFactory extension point in the ASP.Net MVC framework. The sample below shows how to create a Message Handler factory for the StructureMap framework.

The SetMessageHandlerFactory method should be called once at application startup. Other than that … this is a pretty simple way to add a Dependency Injection Framework support to the Portable Area infrastructure.

This makes it pretty simple for your message handlers to use constructor injection to manage their dependencies.

ASP.Net MVC Portable Areas – Part 3

erichexter — Tue, 03 Nov 2009 17:00:00 GMT

Using a Portable Area

This is the third part in a series about using a Portable Area (PA) using MvcContrib. This sample walks through the Host Application side of consuming the Login Portable area. This example demonstrates how a portable area such as a login can send messages and recieve responses from the host application. This allows the host application to control the core of the application and lets the Portable Area solve the User Interface portion of the Login. While this is a pretty simple example it is important to look at the concept of what a Portable Area could solve. A portable Area could just handling wiring up multiple user interface screens and deal with simple form validation while leaving a lot of from for the host application to control the parts that are important to the application. Or a Portable Area could provide an entire self contained piece of functionality like a Blog engine, or mulit-instance forum. There is a large spectrum for how much the Portable Area could provide. This example focuses on an example where the Portable Area provides the UI and lets the host application own the domain logic.

Part 1 – Introduction
Part 2 – Sample Portable Area
Part 3 – Using of a Portable Area (this post)
Part 4 – Using an Inversion of Control Framework.

In order to use a Portable Area (PA), the following references need to be added to your MVC project: MvcContrib & the assembly of the Portable Area.

Wiring up the Portable Area to the application is done at startup. Portable Areas use the message bus as a way to communicate between the PA and the application. The main method for integrating with a portable area is to register a Message Handler. The sample below demonstrates wiring up a LogAllMessagesObserver, LoginHandler, and a ForgotPassword Handler for the Login Portable Area.

The call to the InputBuilder.Bootstrap() is required to initialize the Embedded Resource view engine used by this Portable Area infrastructure.

Here is an example of an Observer message handler. This handler simply logs the message to the debugger. This could be used to collect metrics about the system, or log messages to a loggin framework. The idea of an observer is that it is looking at the base message and not modifying the state of the message.

The Login Message Handler receives the login request and than sets the result object which is part of the contract that is specified by the LoginPortableArea from Part 2.

In this example the main logic for login is still in a authentication service. The handler is just used to wire up the message and response of the portable area to the host applications domain services.

Whats Next?

The constructor of the LoginHandler takes an IAuthenticationService as a dependency and my next post will walk through how an Inversion of Control container can be connected to the Bus to allow the framework of your choice to put your pieces together.

ASP.Net MVC Portable Areas – Part 2

erichexter — Mon, 02 Nov 2009 17:00:00 GMT

Sample Portable Area

This is the second part in a series about creating a Portable Area (PA) using MvcContrib

Part 1 – Introduction
Part 2 – Sample Portable Area (this post)
Part 3 – Usage of a Portable Area
Part 4 - IoC Framework Support

To create a Portable Area, the first step is to create a separate class library project for the Portable Area. The output assembly is the single file needed to add this functionality to applications which wish to consumer it.

The minimal references required for the Portable Area are shown below:

While DataAnnotations is not required to build a PA, using DataAnnotations for Model validation makes distribution of the PA very easy, since the MVC2 DefaultModelBinder supports the DataAnnotations validation out of the box. If you choose a different validation framework than you would need to distribute those assemblies or possible IL Merge them into your PA.

The sample PA in the MvcContrib project is a Login Portable Area, this is a pretty simple example, but demonstrates how a PA and application can integrate. This is also a piece of functionality that most applications need.

Below is the code listing and structure for the project. It looks identical to a mulit-project ASP.Net MVC Area.

The first difference between creating a Portable Area and a standard area from the files perspective is how the Views are registered with the project. The views Build Action should be changed from content to an Embedded Resource. This allows the views to be embedded into the output assembly so that they view folders and files to not have to be manually copied into each application that uses it. Instead a special view engine will pull these views out of the assembly at run time and render them the same way that a physical view would be rendered.

The second item that is needed to create a Portable Area is a specialized registration class for the that inherits from PortableAreaRegistration. This class provides an extended RegisterArea method that sends in the MVC AreaRegistrationContext and a IApplicationBus object to the method. The IApplicationBus is the mechanism that a Portable Area uses to send messages to the hosting application during. The use of the bus during registration is not required, but it is available in case and specific start up logic is needed to correctly configure the Portable Area.

The last line of the method show how the Portable Area registers the embedded views with the view engine, using the call to the RegisterTheViewsInTheEmmeddedViewEngine method.

An important caveat around the setup of the Registration class is to make sure that the Registration class is sitting in the same Namespace as the controllers and views. If the Namespaces do not match the views will not be able to be located. Since the namespace for the Embedded Resource views cannot be specified declaratively they are created using the Assemblies Default Namespace and the folder structure that they are stored in. So keeping all of these properties consistent will make everything work. If you change the default namespace of the project, you will need to update the namespace of your registration and controller classes.

The first call in this registration method sends a message to the hosting application. This is a contrived example but it demonstrates sending a tracing message that can be used by an observer message handler to log the registration activity. This tracing can be useful since the Portable Areas use the built in MVC registration mechanism for auto discovery and there is not a lot of visibility into the built in registration process. This example shows how a message can be sent between the PA and the hosting application.

The LoginController demonstrates the functionality needed to handle displaying the login form and then handling the form post. There are two Actions in the controller, one to render a Login View and a second Action to process the form post and redirect the user on success or , in the case of a login failure, display the login error messages to the user.

The message bus is the main mechanism for communicating between a portable area and the hosting application. This is a very simplistic mechanism that allows the Portable Areas to communicate with the hosting application in a synchronous manner.

For some scenarios, like when the area should not implement its own database, the host application can own how the domain model is persisted in data storage and the Portable Area can worry about handling the multi-page user interface and other user interface concerns like validating simple data types.

The messages that are passed to the Bus are shown below.

The LoginInput represents the data that is collected from the Login screen. It is using the built in DataAnnotations attributes to handle the simple validation rules.

The result object is used to determine if the credentials were valid for the host application and are then used to either display the error message or redirect to a protected page. Since the LoginInputMessage is passed to the bus with a reference, the handlers in the host application can change the Result object's state and the portable area can than access the Result and determine the correct logic to perform given that result. The implementation for the handlers will be in the next post in this series.

The Index view for the login uses the MvcContrib Input Builders to render a standard form to the page. The use of the Input Builder allows an host application to override any of the html mark up using the Input Builder partials. This means that the Portable Area and Input Builders play very nicely with each other.

This is what the view looks like in a browser.

This is the validation errors that are handled by the Portable Area, using the MVC 2 default model binder.

This shows an error message that is passed back from the resulting application. Other rules could be applied as well.

This is the approach for creating a Portable Area (PA) using the MvcContrib Portable Area feature. Let me know what you think about this approach, we tried to stay with the simplest approach that could possible work, and focused on making it easy to both develop and consume the Portable Areas.

Please leave your comments on what you think of the approach.

ASP.Net MVC Portable Areas via MvcContrib

erichexter — Sun, 01 Nov 2009 18:00:00 GMT

This is a multi post series on ASP.Net MVC Portable Areas

What is a Portable Area?

A Portable Area is a set of reusable multi page functionality can be dropped into an application to provide rich functionality without having to custom build functionality that is literally the same in every application. This could be considered a plug-in or add-in type of functionality. The portable portion of this approach is that the area can be distributed as a single assembly rather than an assembly and a host of other files, like views or other html assets that need to be managed and maintained over time. By making a portable area totally self contained in a single assembly, this should allow for easier reuse and upgrades to the area. The challenge for doing something like this has been how do you allow enough control over the User Interface by the application yet still allow the actual views to be packaged with the logic.

Why Now?

There has been some discussion in the past on the MvcContrib mailing list about creating an plug-in framework and plugins but I do not think we had enough of the pieces in place to do this properly. I believe that with MVC 2 we have those missing pieces figured out. The first enabler is the inclusion of Areas into the MVC 2 feature set. This includes having the Area and Controller namespace become part of the route data which is used both for Controller/Action selection but this also flows down to the selection of a view. The second enabler is some of the work that came out of MvcContrib and the Input Builders. While implementing that feature we came up with a way to pull views from an assembly as an embedded resource. This with the ability to override the default view engine in a way that allows an application developer to place their own version of a view in a folder so that they have the option to change the view to their needs was huge. The last enabler really comes from what we have learned from all of the SOA greats and see how frameworks like nServiceBus and MassTransit have demonstrated that a messaging approach for integration can keep our concerns separated.

The other why now is that my company, Headspring, has found that in order to make our practice more successful, we need the ability to drop in some of the essentials for an application, we would prefer to do this in a binary form that is easy to upgrade and does not leave us with copy and pasted code between our various projects. We would like to see that if we learn something from one project that we have the potential to apply those learning's to projects that are still in flight. We all prefer that rather than waiting for the next project to start so that we can apply what we have learned to the new project. This approach will be much better for us as developers and our client will benefit as well. We could take the approach of working on this in a bubble but by putting this out for the community we can learn from every else and potentially help others in the process and raise the collective bar for the industry, in our own little way.

Logical View of a Portable Area

Below is a logic view of a Portable Area. It shows how the Green block is an application. Inside the application the blocks in dark blue are framework components in ASP.Net MVC 2 and MvcContrib. These blocks provide some minimal framework support for registration view resolution and communication between the application and the portable area. The light blue blocks represent code the developer create. The code in the Portable Area is created by the Portable Area developer. The code in the application block is coded by… you guessed it. The application developer.

Thanks for the Pictures but where is the code?

The code is currently available in the MvcContrib MVC 2 Branch. You can get the latest binary from our (TeamCity/CodeBetter) build server here: http://teamcity.codebetter.com/guestAuth/repository/download/bt83/.lastSuccessful/MVCContrib.release.zip

There is a sample application you can download here: http://teamcity.codebetter.com/guestAuth/repository/download/bt83/.lastSuccessful/MVCContrib.Extras.release.zip

or from the code repository on GitHub: Download the source as a zip. or Fork it on GitHub.

Building a Service Gateway Using MassTransit, Part 3

Chris Patterson — Sun, 01 Nov 2009 02:58:25 GMT

This post is the third in a series on building a highly available service gateway. The implementation will be built in C# using MassTransit, StructureMap, ASP.NET MVC, and NHibernate.

Did somebody say code?

The past two posts began to explain how to build a service gateway using MassTransit. In this post, I'm going to share some of the initial code that makes up the gateway service. The gateway itself consists of two components. The first implements the communication to the external service with a set of messages that are only used internally by the service. The second is the saga that provides the interface to the service gateway.

Service Contract

The interface exposed to the application consists of two messages, the first for the command and the second for the response. The message contracts are defined using interfaces, allowing the class for the message to be an internal implementation detail.

The message contract representing a request for order details includes the CustomerId and the OrderId.

public interface RetrieveOrderDetails
{
	string OrderId { get; }
	string CustomerId { get; }
}

When the order details are received, the following message is published.

public interface OrderDetailsReceived
{
	string OrderId { get; }
	string CustomerId { get; }
	DateTime Created { get; }
	OrderStatus Status { get; }
}

public enum OrderStatus
{
	Unknown = 0,
	Submitted = 1,
	Accepted = 2,
	InProcess = 3,
	Complete = 4,
}

The CustomerId and OrderId are the same as the values passed in the request. Created is when the order was created, and Status is an enum representing the status of the order. Notice that no internal values are included -- no primary key from the order table and no primary key from the customer table. The request and response are correlated on identifiers that make sense in the application domain. While SQL purists will point out that numeric primary keys are quicker for retrieving rows in a database, they make for a very fragile interface with other components in the system. Reliance on a primary key outside of the context of the system storing the order details is a path to friction or outright failure.

Time To Make the Saga

At this point in the design of our service, the need for a saga to manage the request state is not entirely obvious. While the TDD purists might want to call YAGNI at this point, let me assure you that "it will all be... revealed!" So for now, let us take a look at the first pass of our saga definition.

public class OrderDetailsRequestSaga :
	SagaStateMachine<OrderDetailsRequestSaga>,
	ISaga
{
	static OrderDetailsRequestSaga()
	{
		Define(Saga);
	}

Our saga state machine uses a static initializer to define the states, events, and transitions of a saga. The previous code merely defines our class as a saga and calls our saga initialization method (shown below).

	private static void Saga()
	{
		Correlate(RequestReceived)
			.By((saga, message) => saga.CustomerId == message.CustomerId &&
			                       saga.OrderId == message.OrderId &&
			                       saga.CurrentState == WaitingForResponse);

		Correlate(ResponseReceived)
			.By((saga, message) => saga.CustomerId == message.CustomerId &&
			                       saga.OrderId == message.OrderId &&
			                       saga.CurrentState == WaitingForResponse);

Since our request criteria include our customer id and our order id, we use those to correlate the message to the saga. We also include the state of the saga to ensure that we do not match to a request that has already completed. We will look at some other ways we can enhance the performance of the service later on by using some additional states.

	public static State Initial { get; set; }
	public static State WaitingForResponse { get; set; }
	public static State Completed { get; set; }

The three states we have defined, including an initial state when a new saga instance is created, a waiting for response state one our request has been sent to the service, and a completed state once the response has been received and published.

	public static Event<RetrieveOrderDetails> RequestReceived { get; set; }
	public static Event<OrderDetailsResponse> ResponseReceived { get; set; }
	public static Event<OrderDetailsRequestFailed> RequestFailed { get; set; }

The three events that we have defined, including the message contract that maps to the event. The subscription logic for the saga will automatically map message handlers for these events that will invoke the actions depending upon the current state of the saga.

		Initially(
			When(RequestReceived)
				.Then((saga, request) =>
					{
						saga.OrderId = request.OrderId;
						saga.CustomerId = request.CustomerId;
					})
				.Publish((saga, request) => new SendOrderDetailsRequest
					{
						RequestId = saga.CorrelationId,
						CustomerId = saga.CustomerId,
						OrderId = saga.OrderId,
					})
				.TransitionTo(WaitingForResponse));

The first event handler, RequestReceived, is invoked when the saga is created in response to the RetrieveOrderDetails message. The handler copies the properties of the request, and then publishes the request message to the proxy that will call the external web service. After the message is published, the state of the saga transitions to the waiting for response state. When using transactional queues, the receipt of the message, creation of the saga in the database, sending of the command message to the proxy, and saving the saga are all part of a single distributed transaction. This ensures that everything completes as a single operation to ensure no requests are lost.

		During(WaitingForResponse,
			When(ResponseReceived)
				.Then((saga, response) =>
					{
						saga.OrderCreated = response.Created;
						saga.OrderStatus = response.Status;
					})
				.Publish((saga, request) => new OrderDetails
					{
						CustomerId = saga.CustomerId,
						OrderId = saga.OrderId,
						Created = saga.OrderCreated.Value,
						Status = saga.OrderStatus,
					})
				.TransitionTo(Completed));
	}

The second event handler, ResponseReceived, is invoked when the OrderDetailsResponse message is received. The results of the request are stored in the saga and a message is published containing the details of the order back to the original requestor. Another approach would be to capture the requestor address (via the ResponseAddress header from the message) and then resolve that address using the endpoint factory to send the response directly to the requestor. I don't really encourage this approach without having a truly unique identifier for each request.

	public OrderDetailsRequestSaga(Guid correlationId)
	{
		CorrelationId = correlationId;
	}

	protected OrderDetailsRequestSaga()
	{
	}

	public virtual string CustomerId { get; set; }
	public virtual string OrderId { get; set; }
	public virtual OrderStatus OrderStatus { get; set; }
	public virtual DateTime? OrderCreated { get; set; }
	
	public virtual Guid CorrelationId { get; set; }
	public virtual IServiceBus Bus { get; set; }
}

The rest of the saga class is shown above for completeness. The properties are part of the saga and get saved when the saga is persisted (using the NHibernate saga persister, or in the case of the sample the in-memory implementation). The constructor with the Guid is used to initialize the saga when a new one is created, the protected one is there for NHibernate to be able to persist the saga.

The Service Proxy

The saga uses the external service proxy to perform the actual work, which is shown in the proxy class below.

public class OrderDetailsWebServiceProxy :
	Consumes<SendOrderDetailsRequest>.All
{
	public void Consume(SendOrderDetailsRequest request)
	{
		// make the call to the service to get the order details here

		var details = new OrderDetailsResponse
			{
				OrderId = request.OrderId,
				CustomerId = request.CustomerId,
				Created = (-1).Days().FromUtcNow(),
				Status = OrderStatus.InProcess,
			};

		CurrentMessage.Respond(details, x => x.ExpiresAt(5.Minutes().FromNow()));
	}
}

The message handler uses the criteria from the SendOrderDetailsRequest message (which was published by the saga) to call the external service and retrieve the order details. The details are then returned to the saga in the form of an OrderDetailsResponse message which is internal to the service (and therefore not part of the interface assembly that is provided to applications that want to use the order details service).

Test That Thang

Now that our saga has been developed, we need to be able to test it. A unit test will be created that creates a testing instance of the service bus (see the sample for the implementation details) and verifies that the saga responds properly to the request. To request order details, a very simple client would subscribe to the response message and then publish the request.

const string orderId = "ABC123";
const string customerId = "12345";

LocalBus.Subscribe&lt; OrderDetailsReceived >(message =>
	{
		response.Set(message);
	},
	x => x.OrderId == orderId && x.CustomerId == customerId);

RetrieveOrderDetails request = new RetrieveOrderDetailsRequest(customerId, orderId);
LocalBus.Publish(request, x => x.SendResponseTo(LocalBus.Endpoint));

The subscribe method used above specifies that when a message of type OrderDetailsReceived is received, if the contents of the message match the predicate specified (which in this case, is checking the OrderId and CustomerId contained in the message) then the statement specified should be called. Our example is from the integration test (built using NUnit) that verifies the service performs from end-to-end.

The syntax above is functional, and it will work, but it does not represent the most scalable approach. In the next post, I'll start to explain how to build a much more scalable method of handling thousands of concurrent requests on a single machine using IIS.

In the meantime, the sample code is available in the MassTransit trunk as a standalone solution. You can find it in the trunk\src\Samples\ServiceGatewaySample folder. There are unit tests that verify the calling syntax shown above and a service that hosts the services (both the saga, and the proxy service).

“Productization”

chadmyers — Fri, 30 Oct 2009 16:37:08 GMT

I need some ‘wisdom of the crowd’ here.

I’m guessing just about ever single one of you (at least the software developer types) reading this have deployed software to a production environment at least once in your career. One of the products we’re working on is getting near to a major milestone and will start seeing wider deployments than the beta and preview deployments we’ve seen so far. I’m in charge of developing a list of different things we’ll need to do to the product to start wrapping it in a bow for wider distribution and deployment.

What sorts of things, in your opinion and experience, should I be most concerned about? Some of things I have on my list are: Good logging and error reporting, well-defined and practiced strategy for handling updates and patches going forward (including incremental DB schema migration), etc.

I’m also looking for any “Whatever you do, DO NOT do *this*” where *this* is some thing you tried and ended up being a terrible idea or blew up in your face.

If you comment, please keep your advice short and to the point. Though I would encourage you, if you have a funny and/or insightful story to share, to blog and link it back to this post for me and others to read!

What I have been up to.

erichexter — Fri, 30 Oct 2009 03:36:02 GMT

I have been pretty busy over the last few weeks. I just wanted to document what I have been doing.

I co-presented a free Advanced ASP.Net MVC half day with Jeffrey Palermo in Austin. We had a good group of people show up to the Austin Microsoft office. As I was preparing and presenting the session, it has become very obvious how much

I co-presented with Jackson Harper from the Mono team at Monospace on Mono and the Web ASP .NET MVC, WebForms, Tools, and Servers. In preparation for this session I took the code camp server, codebase and got it running with Mono on openSuse… yeah that Linux, That means that ASP.Net MVC, StructureMap, NHibernate, Automapper, Castle Validators, and MvcContrib all work on mono and are binary compatible! I was able to compile the application in Visual Studio and using the MonoVS plugin deploy a vm. We also demonstrated ftping the bits to a Linode and Jackson showed how to get is all setup. He even made a Virtual Appliance that when you start it up in vmware it will be running the application on the web server.. You can get the slides, link to the mono version of the source code, download the virtual appliance from the mini site that was setup for the talk. http://MonoOnTheWeb.info . I thought it was pretty cool seeing all of this stuff work. It was a great way to prove out that the application stack that Headspring uses can be deployed on a wide range of platforms which ultimately opens up our potential customer base.

I had the chance to present Database Change Management as the InnoTech conference. It was pretty good to get out and see a group of people that I have not seen before. I sometimes feel like I am in the Austin .Net bubble so while I presented some code samples in .Net using SQL server I really tried to present the information as being agnositc of you database platform. Good development techniques can transcend technology and platforms.

Building a Service Gateway Using MassTransit, Part 2

Chris Patterson — Thu, 29 Oct 2009 22:34:21 GMT

This post is the second in a series on building a highly available service gateway. The implementation will be built in C# using MassTransit, StructureMap, ASP.NET MVC, and NHibernate.

Continued...

In part one, I discussed two exchange patterns that are often exposed as a web service. In this installment, I'm going to cover a more complex exchange pattern that including makes a request to an external system in response to a request on our web service.

One of the comments on the previous post raised the question of using messaging for queries. Udi, Ayende, and I agree that this so-called "Data-SOA" is a bad thing. The queries I am presenting in this article are made against an external service and not an internal database. In many cases, this service might have limited availability and limited throughput, making it important to isolate our service to increase availability.

Complex Request

In the simple request, order status is a small enough data point that caching it at the web service makes sense. The details of an order, however, are much more involved and it is not practical to keep the details cached in case they are requested (which in this case, is much less often that the order status). To support the order detail request, we will apply a pattern that separates the inbound request from the outbound request, ensures that requests are performed only once (minimizing the click, click, click refresh mentality of some users), and even retain requests if the outbound service is unavailable until the request expires or the outbound service becomes available.

The first thing we want to build is the service that will be responsible for calling the external web service. To start, we'll define two messages representing the happy path of calling the service. The first message contains the criteria of the request (such as the order id and perhaps the customer id) and the second message contains the details of the order that were returned by the web service. In addition to the messages on the happy path, we may also define additional messages used to publish exception information related to the request.

Once the message contracts are defined to call the web service, we need to build a service that will handle the request messages. To do this, we'll build a message consumer for the request message, host it inside Topshelf, and subscribe to a service bus bound to the input queue of the service. The message handler will use the request message properties to prepare the request to the external web service and produce the response message once the request returns. In this example, the external web service only supports synchronous requests (in a later article I will try to cover calling remote services that support asynchronous requests).

At this point, we have a service that responds to a command message (the request) and produces a result depending upon the outcome of the request. What we need now is a way to coordinate the requests to the service to ensure that requests are not lost due to an unavailable service, errors are retried once a failed service becomes available, and duplicate requests are ignored.

To coordinate the service requests, a saga will be created to manage the state of each request. A new message will be created to initiate the saga containing the same data that is needed to produce the request message that is sent to our gateway service. And as with the service, messages will be created to return the results of the request to the consumer of the external service. The saga will use our state machine driven saga syntax, making the business logic understandable at a glance. The saga will also define the retry parameters that should be used to recover from service outages by specifying an exception policy. The messages that are used by the requester to interact with the saga will become the interface that is used by our system to make requests to the external service. This includes the web service we are in turn providing to our customers. The internal messages used to communicate between the saga and the gateway service are not intended for use outside of our gateway service component.

A nice side benefit of this architecture involves the actual call made by the gateway service. Since the interface is defined by the messages that are orchestrated by the saga, the backing implementation can be changed without impacting the consumers of the saga-based front end. This can be a huge benefit when it comes time to change service providers or bring an external service in-house either through acquisition or new product developments.

With the saga in place (and for this example, we will go ahead and host it in the same process that we are hosting the gateway service), we are now ready to build the web service request handler. Rather than go straight into that now, I think I'm going to stop here and save that for part three. After part three is finished, I'll start posting some of the code for the examples and include it in the samples folder to make it easy to pull down and experiment with on your own machine.

Building A Service Gateway Using MassTransit, Part 1

Chris Patterson — Thu, 29 Oct 2009 03:18:58 GMT

This post is the first in a series on building a highly available service gateway. The implementation will be built in C# using MassTransit, StructureMap, ASP.NET MVC, and NHibernate.

Introduction

A common way of applying a messaging solution to an existing system is to find a tightly coupled service reference in the system and insulate the system from that dependency. For example, the system may utilize a web service via a call from the user interface, blocking the user interface until the web service request completes. Another situation might involve a web service provided by the system which in turn calls another web service to complete the request. While these two examples are common in systems today, it is important consider how the system is using a dependent service when determining how best to insulate the system from that service.

To help make that decision, allow me to share some common exchange patterns implemented using a web service. While most web services are modeled after the request/response pattern, there are actually several types of exchanges that can occur within a web service method.

Drop Box

If the web service is called by the client to provide information to the application, availability is likely the most important concern. Clients may have limited connectivity (such as a wireless client), limited resources (such as an embedded system that is unable to store data for future delivery), or a combination of these and other factors. To deal with these limitations, the web service should be designed for maximum availability to avoid failed requests due to an outage behind the web service boundary.

To support this high availability, the web service should count on the only thing that is available at the time web service method is invoked -- the local machine. If the web service attempts to write the information to a database located on a server across the network that is not available, the information in the request may be lost. In this case, it would be a better choice to write a message that contains the request information to a local queue. By doing this, the request information is retained and can be processed separately from the web service request. This allows allows the web service method to return to the caller, keeping resources available for other clients to report.

Once the data from the request is safely stored in the queue, a separate service is built that consumes the messages from the queue and sends the information to the internal system, which in this case might be a database server. If the database server is unavailable, the message is left in the queue until the database is available. This new service can also be stopped and even updated without disrupting the web service from receiving requests.

Simple Request

If we look at another scenario, one where the request contains search criteria and the response includes the data specified matching the request. In order to break this down in different way, we will look at two different types of requests. First, we will define a request to retrieve the status of an order. Second, we will define a request to retrieve the details of an order.

When only the status of the order is requested, we are dealing with a relatively small amount of data per order -- in this case, the order id and the status of the order. If this consisted of maybe 50 bytes of data per order, we could easily store the status of 100,000 orders in only 5 megabytes of RAM. Since users might check the status of an order often, at least once an hour, this could generate thousands of requests an hour. Since the status of an order may only change once or twice a day, the need to query the database each time the order status is checked can put an unnecessary strain on the database server. In addition, the order status request service is now tightly coupled to the database, making the availability of the service dependent upon the availability of the database. This dependency chain can get even longer as more complex systems are designed, so limiting the dependencies of a service is a key parameter in increasing availability.

To reduce database load, eliminate the dependency on the database, and increase the availability of our order status web service, we can design our service to subscribe to order status updates. When the status of an order is updated, the update will publish a message containing the new status. As the status of orders are updated, our service would update an in-memory cache containing the status of every order in the system (well, every is not necessarily every -- it could just be the orders placed over the last week that have not yet been received by the customer). On startup, the web service would query the database for the status of all orders placed and use the results of that query to seed the cache. Once the cache is seeded, as new orders are added and order status updates occur, the cache would update dynamically in response to the update messages. Since the cache is local to the web service, requests need only check the status by querying the cache and immediately returning the status of the order to the caller. In the case of an status request for an order that does not exist in the cache, the service could queue a request to get the order status from the database which would then publish that orders status so that it could be returned. If the order is not found, the service could return an unknown order response to the caller with instructions to perhaps try their request again later.

Up Next

The two exchanges described above are relatively easy to implement using messaging (and likewise, using MassTransit). The next exchange pattern I'm going to cover is the more complex request where the dependent service must be called to complete the request. Due to that complexity, I'm going to wait until the next installment to describe that in greater detail. After that, I'll start to share some code as we build a solution to these exchanges.

MVC Best Practices

bogardj — Wed, 28 Oct 2009 12:45:26 GMT

Simone has a great post (as usual) on 12 ASP.NET MVC Best Practices:

Controller:

1 - Delete the AccountController

2 - Isolate controllers from the outside world

3 - Use an IoC Container

4 - So NO to “magic strings”

5 - Build your own personal conventions

6 - Pay attention to the verbs

Model:

7 – Domain Model != ViewModel

8 – Use ActionFilters for “shared” data

View:

9 – NEVER use code-behind

10 – Write HTML each time you can

11 – If there is an if, write an HtmlHelper extension

12 – Choose your ViewEngine carefully

This is a fantastic list, and of course I have my own additions/suggestions :) First, I’m not a fan of action filters for shared data. If we’re doing strongly-typed views, using action filters for shared data puts us back into the magic string land, where we have to either use inheritance in our ViewModels, or the dictionary part of ViewData. I don’t like either approach, I’d rather go with something like RenderAction to truly enforce SRP in my ViewModels. Not everyone appreciates the elegance of this approach, but I’m sure they’ll agree eventually.

I also don’t like writing an HtmlHelper extension just because I have logic in the view. For a half-decent view engine, it shouldn’t be that problematic to have view logic in the view. HtmlHelper extensions enforce the Helper object anti-pattern – a bunch of procedural logic hanging off one static class. Instead, we went the route of building intelligent input builders for input elements, and only really using HtmlHelper when we want to eliminate duplication between multiple views. I see these constructs:

HtmlHelper
Partials
Master Pages

All as means of eliminating duplication in our views, but not much more. If there’s logic in a view, that’s fine by me as long as it’s not duplicated. It’s not my fault that C# looks downright bizarre mixed in with HTML markup, but that’s what other view engines are for.

Finally, if there was an award for How Not to Design a Controller, the AccountController would be the runaway champ. I still can’t imagine why it’s necessary, and the idea of portable areas would be a better fit for its functionality than the Thing We Always Delete with Extreme Prejudice.