Suppose for now we have the a very simple application that sends messages from one endpoint to another. We want to extend the NSB framework to do the following:

• Outgoing message should be checked for confidential information – like credit card numbers – and censored if necessary.
• Messages of some specific kind should be augmented with copy right notices.
• Some messages should be prevented from being send. As an alternative, a simple notice message should be sent instead.
• The outgoing bytes should be compressed and encrypted.
• The incoming bytes should be decrypted and decompress before dispatching to the message handlers.

To achieve those requirements I started analyzing the UnicastBus class, which is responsible to taking the message processing them before sending them using the configured transport.

The first extensible point I found was the IMessageModule interface. Classes that implement this interfaces should implement three functions that represents the three different points in the lifecycle of a message handler. Those functions are executed before processing a message, afterwards and afterwards if a problem happened. The biggest short come of this approach  is that the functions take no parameters. You don’t know which messages are being processed, you don’t have access to the bus to send messages on your own. If you know the Alexandria application you will be very disappointed.  Cool things like caching responses and sending them when a matched request is sent is not possible. I’m not sure when to use IMessageHanlder interface. The only usage I’m aware of is managing NHibernate sessions.  The plus side of this interfaces is its easy to use. Just inject all implementations in your IoC Container of choice I you’re cool. NServiceBus will retrieve and call them for you. As I figured out, almost all extensibility points works in same way. Just implement an interface, inject the implementation in the container and you’re done.

The second interface that could be used to grant you access to  message processing pipeline is the interface pair IMutateIncomingMessages and IMutateOutgoingMessages.

The IMutateOutgoingMessages interface has only one method:

IMessage MutateOutgoing(IMessage message);

As you see you don’t just have write access to the message, you have to return the modified you. This can be used to return a wholly different message.

The following snippet shows how to watch sent messages and censor all messages containing the word “Visa”.

public IMessage MutateOutgoing(IMessage message)
{
    var msg = (message as EventMessage);
    if (msg == null)
        return message;

    if (msg.Text.ToLower().Contains("visa"))
    {
        return new CensoredMessage
                   {
                       Text = "Message has been censored!"
                   };
    }
    return msg;
}

Even though you couldn’t prevent sending the message, you can send a replacement message that do pretty nothing. If you have multiple mutators and you would like to execute them in defined order, just introduce them to the container in the same order. IoC containers usually return items in the same order they we configured in.

Accordingly  IMutateIncopmingMessage do the same to the received messages.

What we could not do with those interfaces is compressing the outgoing bytes to spare bandwidth.

To do this you have to use the IMapOutgoingTransportMessages interface. Having single method

void MapOutgoing(IMessage[] messages, TransportMessage transportMessage)

it grants you access to original logic messages as well to the serialized bytes.TansportMessage give also an access to the message headers. Those headers could be used to set meta data that can be useful when decompressing the messages.

Using this method we could implement the following compression service

void MapOutgoing(IMessage[] messages, TransportMessage transportMessage)
{

    if (!ShouldCompressMessage(transportMessage))
        return;

    transportMessage.Headers["zipped"] = "1";

    byte[] output;
    var inputData = transportMessage.Body;

    using (var inMemoryStream = new MemoryStream(inputData))
    using (var outMemoryStream = new MemoryStream())
    {
        using (var zipStream = new DeflateStream(outMemoryStream, CompressionMode.Compress,leaveOpen: true))
        {
            inMemoryStream.WriteTo(zipStream);
            zipStream.Flush();
        }
        output = outMemoryStream.ToArray();
    }
    transportMessage.Body = output;
}

private bool ShouldCompressMessage(TransportMessage transportMessage)
{
    return transportMessage.Body.Length > 300;
}

Now to the other side of communication. All we have to do to decompress the messages is implementing IMapIncomingTransportMessages, isn’t it? I was very surprised to discover that there is no such an interface. I can’t imagine why there is an IMapOutgoingTransportMessages  interface but no IMapIncomingTransportMessages. I guess it must be on the way. Nevertheless NSB is open source and you can add this interface by your self. Simply by adding the interface and calling its implementations using the container in the right place I could undo the compressing and continue process the message, as if there were no compression at all.

void MapIncoming(TransportMessage transportMessage)
{
    if (!transportMessage.Headers.ContainsKey("zipped"))
        return;

    byte[] output;
    using (var inMemoryStream = new MemoryStream(transportMessage.Body))
    using (var zipStream = new DeflateStream(inMemoryStream, CompressionMode.Decompress,leaveOpen: true))
    {
        using (var strea = new MemoryStream())
        {
            zipStream.CopyTo(strea);
            output = strea.ToArray();
        }
    }
    transportMessage.Body = output;
    transportMessage.Headers.Remove("zipped");
}

That’s all, folks.  Using those techniques you can hook your own processing pipeline to the default pipeline of NSB without (big) modifications to the NSB source code.

The Pro .NET 4 Parallel Programming book is written by Adam Freeman and published by APress.

The book is divided by into 8 chapters:

The first chapter is, what a surprise, the introduction. It answers such questions like why should you care about concurrency and parallel programming.

The second chapter introduces the Task class of the Task Parallel Library (TPL) and handles almost all aspects of using it; like staring , cancellation, coordination and handling exceptions of tasks. This chapter makes you love tasks and wish you have never worked with threads.

The third and fourth chapters are about sharing data and coordinating tasks in more details. Those two chapters are what make this book a book about real world problems. You see in this chapter how solve real problems with shared data and control flow. You don’t do solve just embarrassingly parallel problems. Free lunch is over! The subjects in those chapters include locking primitives, signaling other tasks /thread to do work. All of that in great details.

The second two chapters are about parallel loops and parallel Linq. It handles the specialties of those aspects like breaking from a parallel loop or managing order while using parallel Linq.

The seventh chapter is about testing and debugging. It shows the new capabilities in Visual Studio 2010, its strengths and weaknesses and how to make use of them.

The last chapter is a great prologue. It shows how to use the TPL to implement well known algorithms like quick sort and MapReduce.

After this small summary of the book I will show the things I liked and things I didn’t.

Lets start with the positive side of this parallel coin.

Pros:

1. All mentioned aspects are handled in great details. If its tasks coordination, you will know how to wait for tasks, how to start a new task after another task finished successfully or with an exception and how to cancel scheduled tasks. Synchronization Primitives are handled in all details. How to use them, when to use the slim version of them, etc.
2. At the end of each chapter there is a section about common problems and their causes.  You learn which conditions produce such pitfalls and how to avoid them. Actually I have never heard about some of them. I found this chapter to be invaluable.
3. The practical last chapter helps you solving the jigsaw and orchestrating the small parts of the previous chapters to build real world solutions.

Cons

1. I’m not sure if this behavior is intended, but this book is merely about the TPL. All introduces techniques, with exception of the Synchronization Primitives maybe, are from the TPL. Threads are mentioned only sparely as way to manage tasks scheduling. Did threads become useless (at least to work directly with)? Are ThreadPools obsolete?  I don’t know.  I wish the book mentioned that explicitly.
2. The Monitor class has only been mentioned in conjuction with locking objects, but you could use it for other things, like wait and pulsing. I’m not sure if you could simulate this behavior with [(a Manual)|(an Auto)]-ResetEvent. But this should have been mentioned.
3. The book has its own convention in naming namespaces, classes and private methods.  The namespace Speculative_Cache and class name  Use_Speculative_Cache do not conform to the established conventions. Starting private methods with small letter is also not very common in the .Net world.

Summary

A great book about the TPL. If you are a .NET programmer  then you should read it.  If you think you don’t have to because you will not be needing parallel programming in your daily job then you are the most suitable audience for this book. Your programs will perform better if you learned how make independent small chunks  run independently.

Happy reading!

Modern software is build as a pyramid of abstraction layers. When writing a an application in C#, for instance, you are using an abstraction layer above  the intermediate language (IL). The IL is an abstraction layer  above the the native environment, which, in turn, is  an abstraction layer above the von Neumann computation model. The von Neumann computation model, is an abstraction layer above the electronic gates, which hide the details of current and electrons.  It your software uses the Internet you are very likely to use a framework that abstract the whole abstraction layers of TCP protocol (technically your application resides in the 7th layer – the application layer – of TCP layer. Nevertheless, you have multiple abstraction layer inside it).

It is safe to claim that, without those abstraction layer, writing modern software will be kind of self torturing.

Anyway, this article is not about how useful  are abstractions. It’s about how harmful then can be, when used wrongly. The reason why abstracts could be dangerous lies in the fact, that they may hide important details. Such details could cause the system to stop behaving as expected.  When ignoring details, you are saying “I don’t care about the details because my system will work the same, regardless of the actual detail. Actually this is a very important principle of the S.O.L.I.D priciples: the “Liskov substitution principle”.  This principle states “Let q(x) be a property provable about objects x of type T. Then q(y) should be true for objects y of type S where S is a subtype of T.”

In this definition T is an abstraction over S.  Assuming S´is another subtype of T, this principle doesn’t say that q(x) <=> q(x’), for all instances x from S and s´from S´. To use other words, when defining an abstract you are defining the properties you want to have in all subtypes. Mixing defined with assumed common behavior leads to serious problems. A funny Poster about the LSP states: If it looks like a duck, quacks like a duck, but have batteries – you have properly the wrong abstraction.

Is the rest of this post I will show some harmful abstraction that led/ could lead to dangerous behavior.

Databases are memory collections

This abstract has been usually pushed by the raise of the Domain Driven Design (DDD) which encourage persistence  ignorance (PI).  The famous pattern of the PI is the repository pattern which hides the database access details behind a slick interface with few simple methods like : GettAll(), GetById(), etc. This pattern is really great for unit testing. Abstracting the data access layer enables  to replace the database with an in-memory collection of the entities which results in a huge performance hit compared to the accessing the database and, more important, to predictable behavior.

Performance problems

The problems start to appear usually after you have established the infra structure of you architecture, usually too late.The problem with this abstraction is that databases don’t work like memory. Navigation from an object to another in the database is not nearly the same as in memory. Round trips to database are far more time consuming than in memory.  Getting too much object from the database is too expensive. All serious  ORM provide lazy loading to load dependencies of some object on demand. To be mainstream conform I will demonstrate this behavior using the blog post with comments example. Loading the blog post from the database doesn’t mean you want to have all comments. This could be worst if the post has attachments in the database. Loading them  is not only slow but could also  be memory exhaustive. Such sub objects could be lazy loaded, i.e. on demand. Now assume you want show the blog post with all its comments. Lazy loading them is known as the select N+1 Problem. The ORM has to query the database N times, that is one time for each comment, and the first time to retrieve the blog post.

If you ignore this issue it bite you later. And it will bite hard!  Not wanting ti discard your beautiful repository and the nice unit tests with the memory collections to try to solve this dilemma adding new specialized functions to the repository. For retrieving all post with comments you add:

GetAllWithComments()

The same problems happen again and again and each time you add a new method to the repository.  The beautiful slick is ain’t slick and beautiful any more. It get bloated with many methods like:

GetAllWithCommentsPaged();

GetAllWithCommntsAndAuthorsPaged();

Get ByTag();

GetByTagWithComments();

GetByTagWithCommentsPaged();

You name it.

Context problems

Many ORM embed the database context inside persisted object to keep the connection to the database to retrieve sub object and persist updated value. The context provides further context features like  object lifecycle, transactions, etc.  The consequence of this coupling between objects and content causes the persisted object to depend strongly on the the context. It is not enough for a service class to receive a list of blog posts to calculate the average comments count per one post. I has to assume that the comments are all loaded eagerly, or the context is not disposed. It means also that you can not keep the database object in the memory between requests if you wish you modify them later of navigate in the object graph.

Remote calls

Remote calls are calls to a resource on a remote process, remote computer on the same network or a call over the internet. Such calls are afflicted  with long latency.  If you wish to avoid slowing down you application you can all them asynchronously, i.e. starting the call and registering a callback to be executed when the call has terminated. Problems start when  you try to hide the distributed nature of your call. If all what you are doing is calling a webservice to get the weather data and show them on you home page your cool. You can hide the webservice call behind an IWeatherIno interface and nevermind how it works. Of course the latency of you site with get at least as big as the called webservice latency, but no problem. It works!

Now imagine a more complicated situation with many distributed processes.  If a communication partner is waiting for few partners to answer its calls the system will get very slow. And that’s not all. It can get even worse. Deadlocks could render the system unusable.  To demonstrate this situation let us consider the starbucks example. This example demonstrate how multiple process could communicate with each other over a long term to accomplish a business transaction.

A transaction at starbuck involves three actors: a customer, a barista and a cashier. The customer starts the transaction by ordering a drink from the barista. The barista starts making the drink an notify the cashier to bill it. The cashier asks the customer to pay for the drink. The customer pays for the drink. After receiving the payment the cashier notify the barista, which deliver the drink to the customer.

Executing this transaction in the same process yield no problems. The execution flows from object to other as method calls until the initial call of the customer to order a drink terminates.  All needed is one thread.

Now trying to hide the distributed nature could will lead to the following deadlock:

The clients orders a drink and the thread blocks waiting for the asynchronous call to return. As described previously the cashier request the customer to pay. The customer cannot reply because its thread is blocked waiting to the ordering call to terminate, which in turn will not terminate until the payment is done. Every one is blocked waiting for some call to return.  Of course you could assign a second thread to the customer. But what he is ordering two drinks simultaneously? This is a fairly simple example. In a real world  distributed application you will have many of such service calling each other. Abstracting the asynchronous calls behind an interface would harm your system.

Hiding the stateless nature of the http protocol

The http Protocol used for the  web communication is stateless by nature. That is, each http connection is new connection that contains no information about previous connections. Classical ASP.NET tried to  get rid of this limitation by using an abstraction layer over the protocol that enables stateful web control.  preserving the state of web page by rendering a huge amount of redundant data the get exchanged each timer between the browser and server.

ASP.NET Also tried to abstract the html behind an object oriented, component based model which failed frequently. The hidden details about the true nature of html could not be ignored without losing of control over the feel and look of the rendered pages

I’ve tried in this article to demonstrate how abstract could be harmful if used inappropriately. Consider you abstraction carefully and never assume. And remember if it looks like a duck, quacks like a duck, but have batteries – you have properly the wrong abstraction

I came across this Problem after watching the "Functional Programming Fundamentals” series on Channel9, with Erik Mejer. Graham Hutton (the author of Programming in Haskell) guest-hosted the 11th episode and showed how to solve this problem using brute force with Haskell. As I’m learning F# currently I started to implement the program in F# as well. Despite being heavily based on the Haskell solution from that episode, this code is not an accurate translation of the Haskell code. It just uses the same ideas.

At first let us make a structure to hold the possible operations:

This kind of types is called discriminated unions which means that an instance of this type is one of the named cases. An operation could be any of the following cases: Addition, Subtraction, multiplication or division.

The second type is an arithmetic expression, which may either be an integer or two expressions combined using an operation from the previous ones:

Boy, I love discriminated unions

Now we have the structures to hold the possible combinations, we still have to generate then in some way. For a set of numbers  we have to generate all possible subsets, and for each of those subsets all permutations have to be generated. Its obvious that this task can be composed from two separate functions: building subsets and permutating them.

The first function generate all subsets of the set of numbers:

This exploits the simple idea that for each element of the set there are two almost identical categories of subsets;  subsets that include this element, and the matching subsets that don’t. This could be exploited using an inductive approach to generate the subsets of small sets and then adding new elements successively to those sets.  This function returns a sequence instead of an instance of Set because the number of subsets is exponential to the number of elements in the original set.  Sequences are lazy evaluated, and thus consume less memory. There are 2ⁿ subsets of any set of n elements.

The Second function generates for a giving set  alls possible permutations:

This function returns n! different permutations for each set of n elements.

Now we can compose both methods to retrieve a method that generates all permutations of all subsets of a given set:

Given a set of n elements, this function returns n!.2ⁿ different choices, which is a huge number. For n=6 choices yields  46080  elements.

After building all possible permutated subsets of the number set there is still many ways to combine them into an arithmetic expression.  An arithmetic could be represented as a binary tree. Each node in this tree is an operation and the leaves are numbers. Building a binary tree could be done by splitting combining both binary trees into one with a single root node. This could be repeated recursively for all possible sub trees and possible splitting points to generate all binary trees:

This yields an enormous number of binary trees. For a fixed number of leaves n one could construct (2n)! /((n+a)!n!) different trees.   We will see in a coming article how to reduce this number by avoiding redundant expressions because of the commutatively of adding and multiplication.   “split” is the function that splits a list into two non trivial list without changing the order of elements.

Now we move to the evaluating arithmetical expressions.  The function “eval” takes an expression and returns an list of integers. This is a common technique when programming in Haskell to express Null. If the expression is valid the function will evaluate it and return a singleton listing containing the result otherwise it will return an empty list.

“valid” is a simple function that ensure that all intermediate results are natural numbers. If valid of sum expression returned false all tree containing the sub tree of this expression are invalid.:

Now we have all necessary tools to iterate thru all possible combinations and returning the  ones which matching result:

Running this function on the same input as the one presented in the Haskell lecture of Hutton returns the same result:

The only problem with this Algorithm is the runtime. It takes too much to calculate the result.

Finding one solution is relative fast, but finding all results takes for ever. In the coming article I will try to reduce the runtime by avoiding redundant arithmetic expressions.

The code exists on http://github.com/mouk/Countdown and will be updated to reflect the planed changes.

Since I already have a yearly subscription to Tekpub and because tomorrow is my 26th birthday I’m willing to give my coupon away

As I have been using A LOT of open source projects lately and not participating with coding I want to give something back. If you are an active committer  in any active open source .Net (or jQuery)  project, just leave a comment with your OS project and wished series. I will get the coupon and forward it to you.  Active OS projects are … note very unknown project, you know?

== UPDTAE ==

A coupon of “Mastering NHibernate” has already been sent to Krzysztof Koźmic. Have fun with it, Krzysztof

The host of Mastering LINQ, Justin Etheredge,  posted a challenge on his site  promising the first five people to code a LINQ  query returning all primes of a sequence a single Tekpub screencast of choice.

I gave it a try! It is really easy:

var n = 100;
var primes = Enumerable.Range(1, n)
    .Where(i => i > 1)
    .Where(i =>
               {
                   for (int j = 2; j <= Math.Sqrt(i); j++)
                   {
                       if (i%j == 0)
                           return false;
                   }
                   return true;
               }
    );

Having a yearly subscription of Tekpub I don’t really need the price. I will give it away to some co-worker probably.

UPDATE:

The closure inside the where function could also be formulated using the “Range” and “All” method of the (I)Enumerable:

var primes2 = Enumerable.Range(1, n)
    .Where(i => i > 1)
    .Where(i => Enumerable
                    .Range(2, (int) Math.Sqrt(i) - 1)
                    .All(mod => i%mod != 0)
    );

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. This article is a quick overview of Continuous Integration summarizing the technique and its current usage.

I advice you to read the whole article about Continuous Integration before you go on.

For the sake of CI you need the following tools:

1 – A Build Server

TeamCity is very popular CI server from JetBrains, the  creator of ReSharper, IntelliJ, etc. I have had also used CruiseControl.NET, a free CI server from ThoughtWorks, but I liked TeamCity much more because of its friendly UI and distributed agents principle.

2 – Build Tool

MSBuild is an xml-based build platform from Microsoft that comes with the .NET framework. A build platform is some kind of DSL, that helps you to compile your code and do all other related stuff in one action. You could compile source code, execute tests, generate documentation, generate a deployable zip file, publish your web site to the server and almost everything else you may need to do with your code.

Technically, you could write a batch file as you your build script and I have no doubt it will work and do every thing you wish. It just won’t be that convenient.

NAnt is another popular build platform ported from Ant for the Java platform.

Recently I have heard a lot about using Rake, the Ruby build tool, in the .NET world. Psake  is another build automation tool written in PowerShell .

In this and few next posts I will configuring TeamCity to do a nightly build for Stutali , the task list generation tool, I introduced last in this article.The task list generation task will be also included in the nightly build to process its own code and generate the task list.

Building the Build Script

The first step in configuring the build server is to add a build script to the project.  Because TeamCity supports sln files directly, a build script is technically not necessary if you just want to compile the project. Anyway if you want to perform some further building steps like processing  source code or deployment you will need the script.

As already mentioned, MSBuild is the  technology that will be used in this article. The first and most important task is the bare compilation. For this task MSBuild needs only to know  where the sln file is.

<PropertyGroup>
  <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
  <ProjectDir>src</ProjectDir>
  <ProjectName>TasklistGenerator</ProjectName >
  <ProjectFile>$(ProjectName).sln</ProjectFile >
</PropertyGroup>

<Target Name="Build" DependsOnTargets="Clean">
    <MSBuild Projects="$(ProjectFile)" Properties="Configuration=Debug" />
</Target>
  

The Property group section is the place where you define you constants. This could include the project name, tools path and every thing you may use at different places.

TeamCity

After testing the build script and pushing the code to the repository, we can start configuring TeamCity to do the CI work. At first we have to create a new project in the admin panel. This task is as easy as clicking through the wizard steps, filling out some text boxes and ready you are. TeamCity makes really pleasant. Out of the box is Git not supported. Anyway JetBrains has developed a plug-in to enable Git repositories. You have just to download this file from JetBrains and it to the place described.

For the source code repository you should have been able to choose Git from the list after getting the Git plugin installed properly.

In the build triggering section you can specify when you want the build process to be started. Plausible options are either when the repository changes, i.e. when you push modification, or scheduled on a regular base, usually in the nightly, the so called nightly build.

Artifacts

Artifacts are any data items generated during the build process. Those could be binaries, documentation, reports, etc. Because of distributed nature of TeamCity, the actual compiling process doesn’t have to happen physically on the save machine as the TeamCity server. Thus, the artifacts you want to keep have to be specified explicitly, so that they be copied from the machine of the build agent and kept on the server.

Suppose for now the build script generates an html file containing the task list parsed from the source code and save it to CHECKOUT_FOLDER\doc\tasklist.html.  If you want TeamCity to copy this file and make available for download in the artifacts tab of the build report, you have to specify it in the artifacts section

If you want to add an alias to the copied artifacts, just use the => syntax

doc\tasklist.html => tasks.html

You can also copy a whole folder of artifacts.

Now we are ready to go. Every thing is configured and we can trigger a test build. Just click the “run” that you see on almost every project page. If every thing went well you should see a new green line in the result page of the project.

Outlook

It was just the start. What you can do in the CI process is only limited by your imagination and time. You could generate code metric reports, what we will do in the next posts, generate release binaries, push them to the download server, generate documentation,  push the new ASP.Net files to the server and precompile them, etc.

For the next posts, we will see how to integrate NDepend, NCover, FxCop and code documentation with the build process.

See you later!

<book>
	<autor>The Autor</autor>
</boo>

Now you can access the auto properties of each book in the following way:

dynamic book = GetBookAsDynamic();
string autoName = book.autor

Having been using Json intensively  the last few weeks and seeing how easy it is to consume Json in JavaScript, I thought dynamic object would be a fantastic way to make Json more natural in the .Net world. My favorite Json library is the Json.Net. Since, I started to code a wrapper about it.

Suppose now you have such an object, that you have Json serialized:

var obj = new
{
    Name = "mouk",
    FavoriteNames = new[] { "mouk", "dermouk", "mouk9000" },
    NestedValues = new { First = 1, Second = 3, Nums = new[] { 1, 2, 3 } },
    NestedArray = new object[]{ 1, 3, new[] { 100, 2, 3 } },
    Age = 25,
    Height = 180
};

var serializedObject = JsonConvert.SerializeObject(obj);

After deserializing it back to a dynamic object you can access as easy as:

dynamic _deserializedObject = JsonDeserilizer.GetObjectFromString(serializedObject);

string name= _deserializedObject.Name;
int age = _deserializedObject.Age;

int nested = _deserializedObject.NestedArray[1];

Source Code

The source code of this wrapper can be downloaded from http://github.com/mouk/JsonToDynamic/tree/master:

The Project

After some googling with both Google and Bing I decided to do it myself. The only similar program I found, was the Task Scanner plug-in from the Hudson project.  Unfortunately, this plug-in is written in Java and I didn’t want to assume the presence of a Java runtime on the build server. Besides, I want to integrate this job as build task (MSBuild build for now but maybe also for NAnt, etc..). For all other purposes a command line interface is essential. All those requirements make the amount of code I could reuse very minimal. So I went for the own solution and started summarizing the requirements.

Wish List

If you prefer the scrum terminology, you can call this wish list the product backlog. It contains all items that COULD be implemented.

Important feature for me are:

• Recognizing all lines including // TODO token
• Flexibility of defining pre configured and custom tasks
• Assigning priorities for the the different types of tasks
• Returning for each task a TaskItem object with the description, file and line number.
• Grouping the tasks by file name or  priority.
• Integration with TeamCity
• Generating reports in both xml and html formats
• being able to define thresholds for the occurrence of the different type/priorities  to consider a build failed

To generate a task list to you MSBuild script all you need the the following xml snippet:

<UsingTask AssemblyFile="$(ToolPath)\TasklistGenerator\TasklistGenerator.dll" TaskName="Tasklist" />

<Target Name="tasklist" >
	<Message Text="==== Generate tas klist ===" />
	<CreateItem Include="src\**\*.cs">
      <Output TaskParameter="Include" ItemName="Sourcefiles" />
    </CreateItem>
    <Tasklist Files="@(Sourcefiles)" OutputFile="doc\tasks.html" />
</Target>

The Code

Currently, still the project in the pre alpha phase of the CTP (PACTP as I call it since two minutes ago). It Uses a Regex to scan all specified cs files and generates a simple xml representation. A html reporting facility is also in pre stone age look and feel  html (Did I mention I don’t do UI?)  I will try to find out how does usually build tasks, like NUnit, integrate with build server. not sure if there is a common format. The only interface available currently is the MSBuild task.

You can download the most recent source code from GitHub  at the following link http://github.com/mouk/Stutali/tree/master.

Because I have never developed an integration task, I don’t know what are the requirements exactly such a thing. More than ever, every idea or notes will be highly appreciated.

Pysor Series

• Using IronPython to configure Castle Windsor I : the basic functionality
• Using IronPython to configure Castle Windsor II : parameters and references
• Using IronPython to configure Castle Windsor III : arrays and lists

In the first two articles I introduced Pysor, the Castle Windsor configuration tool using IronPython. Now I have added some exciting functions to exploit the nice hash table and list syntax feature in IronPython.

Since the second part of this series is possible to add (named) parameters to component registration. It accepted only both literal and referential scalars. In the current revision you can exploit the nice list syntax of Python using the square brackets to add arrays and list parameters.

A parameter is a string literal or a reference to an already registered service. This value is supplied either to a constructor parameter with the same name or a property.

A example used in the last part was to provide AdditionalMessage property like this:

add( "retriverWithParam", HtmlTitleRetriever, HtmlTitleRetriever,
	{'AdditionalMessage': "Test"})

Now suppose that we have a class that accepts an array of strings in the constructor.  We could provide them using the Python list syntax

add( "MessageStorage" , MessageStorage, MessageStorage,
	{'messages':['first message', 'second message' ]})

This syntax works not only for array but also for ILists.

Things get more interesting when you want to add an array of registered services. The method add returns a hook to the service. You can use this hook to reference the service in the parameters :

ftp = add( "ftp", FtpFileDownloader, FtpFileDownloader)

add( "MultipleDowloaderStorage", MultipleDowloaderStorage,
	MultipleDowloaderStorage, {'dowloaders' : [ ftp] })

Comparison

In this section I will compare the same configuration using traditional xml syntax and the Pysor syntax. I am sure not every body would prefer Pysor. I am pretty comfortable with xml, yet I don’t like it. I will choose Pysor anyway. In the next section I will make a small project and use both configuration mechanisms and let you find by yourself, which solution is more elegant and appropriate for you.

Not breaking the tradition of almost all IoC tutorials, we will need a logging service to be located dynamically. Additionally we have two implementations: ConsoleLog  and FileLog. FileLog expect a file name in its constructor..

public interface ILog
{
    void Log(string message);
}

public class FileLog : ILog
{
    private readonly string _fileName;
    public FileLog(string fileName)
    {
        _fileName = fileName;
    }

    public void Log(string message)
    {
        Console.WriteLine(_fileName+ " -> " + message);
    }
}

public class ConsoleLog : ILog
{
    public void Log(string message)
    {
        Console.WriteLine(message);
    }
}

In additional we have an IAlgorithm interface with following members:

public interface IAlgorithm
{
    event EventHandler OnOperationDone;
    void Run();
}

Algorithm is a class implementing this interface. AlglorithmRunnner is a class that takes an algorithm with an array of loggers, starts the algorithm and notify all loggers when an event is fired.

Configuring this setup of dependencies in App.config would look much like:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
    <section name="castle" type=
"Castle.Windsor.Configuration.AppDomain.CastleSectionHandler, Castle.Windsor"/>
  </configSections>

  <castle>
    <components>
      <component id="fileLog"
                 type="CastleComparison.FileLog, CastleComparison"
                 service="CastleComparison.ILog, CastleComparison">
        <parameters>
          <fileName>log.txt</fileName>
        </parameters>
      </component>

      <component id="consoleLog"
                 type="CastleComparison.ConsoleLog, CastleComparison"
                 service="CastleComparison.ILog, CastleComparison">
      </component>

      <component id="algorithm"
                 type="CastleComparison.Algorithm, CastleComparison"
                 service="CastleComparison.IAlgorithm, CastleComparison">
      </component>

      <component id="runner"
                 type="CastleComparison.AlgorithmRunner, CastleComparison">
        <parameters>
          <loggers>
            <array>
              <item>${consoleLog}</item>
              <item>${fileLog}</item>
            </array>
          </loggers>
        </parameters>
      </component>

    </components>
  </castle>
</configuration>

On the other hand, configuring it with Pysor is as simple as

clr.AddReference("CastleComparison")
from CastleComparison import *

fileLog = add("fileLog", ILog, FileLog,
	{"fileName":"log.txt"})
consoleLog = add("consoleLog", ILog, ConsoleLog)

add("algorithm ", IAlgorithm , Algorithm )

add("runner", AlgorithmRunner, AlgorithmRunner,
	{"loggers": [consoleLog, fileLog]})

Clearly, I will always choose the second configuration.

To-do’s

The updated to-do list is now

• Adding a nicer API for referencing assemblies and importing namespaces (I have now idea how to do it).
• Adding parameters to be passed to the constructor.
• Passing parameters as lists or arrays.
• Dictionary based parameters
• Referencing already registered implementation inside the same configuration script.
• Documenting and signing the assembly
• Lifestyle management
• Considering turning Pysor into an Interpreter to be used as the built-in XmlInterpreter

Source Code

Please don’t forget to download the code from GitHub and try it yourself.

All remarks , ideas, bug reports, etc. will be appreciated.

For the CI I use MSBuild script that compiles, runs test, generates documentations, etc.  For test projects I have the habit of postfixing them with "Tests”  to simplify fetching all test dlls in MSBuild scripts using wild cards (Who dares to claim that Convention over Configuration is new in the .Net world? ). Running all NUnit tests in some project would look like :

<CreateItem Include="tests\*\bin\Debug\*.Tests.dll">
  <Output TaskParameter="Include" ItemName="TestAssembly" />
</CreateItem>
<NUnit Assemblies="@(TestAssembly)" ToolPath="$(NUnit-ToolPath)" />

Adding new test project will require me to add nothing to the build script. I have just to follow the convention.

Unfortunately this is not possible with xUnit task because it accepts only one assembly. Thus I have to specify each assembly separately. The xml result will be also split. @bradwilson nicely wrote me via  Twitter, that I can use a project file to specify multiple assembly.  This, while solving the second problem, doesn’t enable you to use wild cards. You have to add each assembly manually to the project.

Gallio is a tool  or, may better a framework, for running many test frameworks in the same way and provides as unique results format. It integrates also very well with some build servers.

To integrate Gallio in my MSBuild script I downloaded the bits from here and extracted them to the tools folder inside my project root folder.

Running the tests was as simple as

<UsingTask AssemblyFile="$(ToolPath)\gallio\Gallio.MSBuildTasks.dll"
	TaskName="Gallio" />

<Target Name="Test" DependsOnTargets="Build">
	<Message Text="==== Starting Gallio to run tests ===" />

	<CreateItem Include="tests\*\bin\Debug\*Tests.dll">
      <Output TaskParameter="Include" ItemName="TestAssemblies" />
    </CreateItem>
    <Gallio RunnerExtensions="TeamCityExtension,Gallio.TeamCityIntegration"
        Assemblies="@(TestAssemblies)"  />
</Target>

RunnerExtensions is an extension for Gallio to provide TeamCity with status messages. Integrating Gallio with TeamCity will be discussed in a following post.

NUnit   and running multiple assemblies from the console

Even though NUnit MSBuild task supports multiple assemblies, if you want to use NCover with NUnit  you have to do some manual work to enable NCover running NUnit console tool. The @() operator of MSBuild join all paths with a semicolon. On the other hand NUnit console runner expect all assemblies to be separated with a white space. Fortunately you can tell @() how it should concatenate the paths.

<NCover ToolPath="$(NCoverPath)"
        CommandLineExe="$(NUnit-ToolPath)\nunit-console.exe"
        WorkingDirectory="."
        CommandLineArgs="@(TestDll->'%(Identity)', ' ')"
        CoverageFile="$(CoverageFile)"
        LogFile="$(CoveragePath)\Coverage.log"
        AssemblyList="@(CoverageDll)" />

In the fourth line @ concatenate the relative path of each file using a white space. You may need to surround each path with a quote if it contains white spaces.

Pysor Series

• Using IronPython to configure Castle Windsor I : the basic functionality
• Using IronPython to configure Castle Windsor II : parameters and references
• Using IronPython to configure Castle Windsor III : arrays and lists

In the last article I introduced a small Castle Windsor configuration tool using IronPython. This tool enabled us to add service implementation in an easer to read way. On the other hand advanced usages like optional and constructer parameters were not possible.

In this article I will continue developing Pysor (As I called it!) to accept parameters. Before introducing the new functionality I will  show what are parameters and when and how would you want to use them. For the sake of demonstration I will borrow the demo application from the very good article series from Simone Busoli about Castle Windsor. If you didn’t read it then go read it all and come back.

The developed application is an html title retriever. It downloads an html string and then extract the title tag from it.  For downloading the the html document it uses an I Downloader service  that accepts an Uri object and downloads it if it can handle the scheme. For Example we have an HtmlDownloader, FileDownloader, etc. The other needed service is ITitleScraper which extract the <title> tag contents.

public interface IFileDownloader
{
    string Download(Uri file);
    bool SupportsUriScheme(Uri file);
}

public interface ITitleScraper
{
    string Scrape(string fileContents);
}

For the purpose of accelerating the unit tests I changed the downloader to not really download the files but return a fake text instead.

Having the constructor :

public HtmlTitleRetriever(IFileDownloader downloader, ITitleScraper scraper)
{
    AdditionalMessage = "";
    Downloader = downloader;
    Scraper = scraper;
}

with AdditionalMessage as a property indicates a short message, that will be concatenated with retrieved tile, we can now configure the container:

add( "parsingScraper" , ITitleScraper, StringParsingTitleScraper)

add( "HttpFileDownloader", IFileDownloader, HttpFileDownloader)

add( "retriver", HtmlTitleRetriever, HtmlTitleRetriever)

And everything works like expected. Now suppose we want to set the AdditionalMessage for each initiated object. Using xml configuration this could be achieved using a parameters tag  containing all parameters in a dictionary-like fashion.

The most appropriate data structure for this purpose in Python would be a hash which is equivalent to a Dictionary in the .Net  world.

Because C# in the current version doesn’t support optional or named parameters and because I don’t  know whether Python supports method overloading I added a Python method with supply the C# method with default value for missing arguments.

def add(name, service, impl, params={}):
	addComponent(name, service, impl, params)

Using this new method you can now set the AdditionalMessage value for each object:

add( "retriverWithParam", HtmlTitleRetriever, HtmlTitleRetriever,
	{'AdditionalMessage': "Test"})

We test it with

[Test]
public void CanProvideOptionalParameters()
{
    var obj = container.Resolve<HtmlTitleRetriever>("retriverWithParam");
    Assert.AreNotEqual("", obj.AdditionalMessage);
}

And of course it works.

Notice that in line 4 we specified the name of the configuration node to use because have added the service HtmlTitleRetriever twice.

The other use of parameters is to specify an implementation for some service to be used.  If you have for example another IDownloader implementation, that retrieves files sing the ftp protocol and we want to use this implementation for the constructor.  In xml we could do it using ${name} to reference the name of an already registered service. In Pysor we can use this notation as well. But to make it more like a usual program I modified the add function to return a string to be used whenever you need to reference this implementation.

Func<string, Type, Type, IDictionary<object, object> , string> action =
     (name, service, impl, parameters) =>
         {
             var pairs = parameters.ToList();
             var reg = Component
                 .For(service)
                 .ImplementedBy(impl)
                 .Named(name);

             if (pairs.Count > 0)
             {
                 var param = (from pair in pairs
                              select Parameter
                                 .ForKey(pair.Key.ToString())
                                 .Eq(pair.Value.ToString())
                             ).ToArray();
                 reg = reg.Parameters(param);
             }

             container.Register(reg);
             return "${" + name + "}";
         };

 //Inject this function into IronPython runtime
scope.SetVariable("addComponent", action);

And the Python wrapper function looks now like:

def add(name, service, impl, params={}):
	return addComponent(name, service, impl, params)

We are now ready to register a retriever that uses a concrete implementation:

ftp = add( "ftp", FtpFileDownloader, FtpFileDownloader)

add( "ftpRetriver", HtmlTitleRetriever, HtmlTitleRetriever,
	{'downloader': ftp})

[Test]
public void CanProvideSpecificImplimentationParameters()
{
    var obj = container.Resolve<HtmlTitleRetriever>("ftpRetriver");
    Assert.IsInstanceOf<FtpFileDownloader>(obj.Downloader);
}

We are finished for now.

To-do’s

The updated to-do list is now

• Adding a nicer API for referencing assemblies and importing namespaces.
• Adding parameters to be passed to the constructor.
• Referencing already registered implementation inside the same configuration script.
• Lifestyle management

Source Code

The source code of Pysor is available to download from GitHub

https://github.com/mouk/Pysor/

Castle Windsor is a very popular IoC container in the .Net world. Like almost all other containers it can be configured using either a fluent interface or an xml-based configuration file.

The fluent interface has the advantage of being strongly typed, what spares you a lot of errors caused by typos. On the other hand, it is hard coded and can’t be changed easily without recompiling (Actually you could use an IoC container to load you IoC container configuration dynamically but it give a rise to the question: “How do configure the container to load its own configuration?” )

The other option is to use an xml file. Despite being the most used solution in almost all containers it is really a very ugly solution. The configuration file can get very big and very complicated.

As I am reading IronPython in Action from Manning Publications, I thought I could configure Windsor using Python and a very tiny DSL. IronPython is an interpreted language for .Net framework. It combines the elegance of Python with the strength of .Net. Since it being interpreted it is a suitable solution for configuration.

For the beginning I will start with a simple solution, that can configure basic service without an enough for me to be able to configure just basic services without parameter zing. The syntax for configuring a simple implementation in Castle looks like:

container.AddComponent("service.name"
				typeof(IService>),
				typeof(Implementation));

Of course there is a whole bunch of other  settings like lifestyle and parameters, but I will ignore that for now.

Analyzing this one line I realized that all I need is a function with three arguments: the name of the coupling, the service type and the implementation type. SoI coded thus a function in C#:

private static void ConfigureContainer(IWindsorContainer container, string scriptPath)
{
	var engine = Python.CreateEngine();
    var runtime = engine.Runtime;
    var scope = runtime.CreateScope();
    scope.SetVariable("__main__", "__main__");
    Action<string, Type, Type> action =
			(name, service, impl)=>
				container.AddComponent(name, service, impl);

	//Inject this function into IronPython runtime
    scope.SetVariable("add",action);
    var script = engine.CreateScriptSourceFromFile(scriptPath);
    var code = script.Compile();
    code.Execute(scope);
}

To test this DSL I wrote a simple interface with a single implementation to bind them with Python.

namespace PythonConfig
{
    public interface IDependency
    {
        string Name
        { get; }
    }
    public class Dependency : IDependency
    {
        public string Name
        {
            get { return "Default"; }
        }
    }
}

And the Python configuration is just as simple as

import clr
clr.AddReference("PythonConfig")
from PythonConfig import *

add("name" , IDependency, Dependency)

Now everything is ready. You can configure the container using this file:

static void Main(string[] args)
{
    var container = new WindsorContainer();
    ConfigureContainer(container, "script.py");

    var depend= container.Resolve<IDependency>();
    Console.WriteLine(depend.Name);
    Console.ReadLine();
}

And … it works as expected!

Todo’s

I am very aware that this implementation is very limited and incomplete. It is just enough for my needs (That’s how DSL ought to be). Nevertheless, I will try to add some of the missing features. In particularly following  functionalities will be considered very soon:

• Adding a nicer API for referencing assemblies and importing namespaces.
• Adding parameters to be passed to the constructor.
• Referencing  already registered implementation inside the same configuration script.
• Lifestyle management

Links

• IronPython’s page on Codeplex.
• IronPython in Action an excellent (and currently the only the first) IronPython book by Michael Foord.
• Binsor: a Boo based DSL for Windsor by Ayende.

Because each time I wanted to use this service, I wondered which one to use, I decided to do some comparison and  outline the differences.  Thus picked the ones I have most seen or used and compared them with each other. My chosen pastebins are:

• paste-it.net
• pastie.org
• codepaste.net
• gist.github.com

Paste-it.net

Paste-it.net supports a very large number of languages.  Almost all popular languages can be chosen from the list. Just as an example : C#, VB.NET Ruby, Java, Django, xml, etc.  There are even some languages I have ever heard of. I Didn’t know for example that vim is a language.

Beside the big number of supported languages I like ability to set an expiration date at most. If you need to send some co-workers a small piece of code of some internal project and don’t want it to be persisted this option helps you minimizing this risk.  Paste some code and make it disappear after one day.

The other interesting option is the hidden check box,  with which you could hide your paste. I am not really sure how this works. You still can access this post without a password. I guess, it just will prevent you past from being listed.

When viewing a piece of code on paste-it.net you can change the language any time you wish. Beside this ability, you can’t change the theme.  You have always the basic theme with a light background. For my this is almost always just fine. You could also view the raw code, download it as a text file of tweet it.

Right to the code you get a whole bunch of Google ads. All ads are in text format with no blinking or moving party, yet a really whole bunch of them.

Pastie.org

Pastie.org is another simple and easy to use pastebin.  The first thing took my attention was the absent of C#  support (and actually all other .NET languages too) . I don’t thing adding support for C# will cause the developers any more effort. They could just the use theme of java for now.  On the other hand, you can choose Pascal if you wish. Ruby, RoR, C++, etc. are some of the supported languages.

Pastie allows you as well to hide you pastes. On the other hand, it not possible to make the pastes expire after some period of time.  When making a paste private, it gets a very harder name to guess instead of the usual serial number.

In addition to posting a link to your paste you can also embed it in your html. Pastie gives you a small html snippet to a JavaScript file, that inject the code inside of the Dom. Oddly enough when  you make a paste private you still get the snippet to embed, but it does nothing. It’s Pretty confusing the first time you see it. The public ones will work and display the code properly if you have a lot of horizontal space and don’t have complicated style sheets.  For this blog it failed miserably.

Another nice feature is the ability to change the theme on the fly. Not very crucial for me, but it still yet nice feature to have.

For making a bundle of code you Pastie provides you with the ability to define different sections inside your paste and giving each section a title.  One limitation of this feature is that you cannot chose different language for each section. That means you cannot paste the controller code and the mark up of some mvc page.

One graphical ad is shown decently on the right bar. Nothing else.

Codepaste.net

Codepaste.net is a .Net oriented pastebin.  It supports all major .Net languages with some other languages, that may be used during .Net development like JavaScript, Powershell, xml, etc. Support for C++ and Java is also available.

Because of it lack of hiding and expiration settings it is really easy to go. Just paste a snippet, give it a title, choose the language  and push the button. Some other optional information like author name and tags can be provided. After posting the code I couldn’t see anyway to edit the paste. Anyway you can register and log in to manage your pastes.

Beside “recent posts” codepaste.net provides also feed to pastes. I have no idea why anyone would like to subscribe to some bunch of non-related code snippets.

A very interesting feature is its API. It provides access to looking up, searching and posting.  This may be very useful for integrating in other application like twitter clients. just like the URLs shorteners.

Unfortunately codepaste.net doesn’t provide a link to the raw source code, what makes copying some piece of the code a bit harder.

gist.github.com

gist.github.com is a service from Github, the popular project hosting service. When using gist you get all benefits of Github. Each snippet you post to gist become a Git repository with all the known capabilities like cloning, editing and pushing. Taking the embedding facility into consideration, gist.github.com is a very powerful way to post code into some html page and keeping it up to date.

Links

Wikipedia has also an article comparing some other pastebins.