Frans Bouma's blog

"Cloud Cloud Cloud, if you're not in it, you're out!"... or something

FransBouma — Wed, 18 Nov 2009 10:16:48 GMT

After I graduated from the HIO Enschede (B.Sc level) in '94 I have worked with a lot of different platforms and environments: from 4GL's like System Builder, uniVerse and Magic to C++ on AIX to Java to Perl on Linux to C# on .NET. All these platforms and environments had one thing in common: their creators were convinced their platform was the best and greatest and easiest to write software with. To some extend, each and every one of them were decent platforms and it was perfectly possible to write software with them though I'll leave the classification whether they were / are the greatest and easiest to the reader. I'll try to make clear below why this dull intro is important.

Yesterday I watched the live stream of the PDC '09 keynote and in general it made me feel uncomfortable but I couldn't really figure out why. This morning I realized what it was and I'll try to explain it in this blog.

Cloudy skies

If one word was used more often than anything else in the keynote it was likely the word 'cloud'. Cloud, cloud, cloud, azure, cloud, cloud, azure, cloud, azure... and so on. Perhaps it's the weather in Seattle which made Microsoft fall so in love with clouds, I don't know, but all this cloud-love made me a little uneasy. This morning I woke up and realized why: it's too foggy. You see, the whole time I was watching the keynote, I had the idea I was watching the keynote of some conference about some science I have no knowledge about whatsoever.

"Cool, another guy talking about azure clouds with yet another set of fancy UIs I've never seen, giving me the feeling that not using those is equal to 'doing it wrong', but what the heck azure clouds are and what problem they're solving is beyond me". That kind of thing.

A long line of people were summoned on stage to tell something about some great tool / framework / idea / wizardry related to clouds and with every person I more and more lost grip about what problem they all wanted to solve. All I saw was a long line of examples of Yet Another Platform with its own set of maintenance characteristics, maintenance UIs, maintenance overhead and thus maintenance nightmares.

More UIs, more aspects about things which were apparently new to software engineering nevertheless utterly essential to writing good software... more UIs I've never seen before, more cloudy weather, more azure flavors, more UIs I've never seen, more...

"Aaaaarrrgg!"

As I've tried to explain in the first paragraph, I've been around the block a couple of times. I have lived through internet bubbles, read McNealy's 'The Network is the computer' articles / propaganda, shaked my head when I heard about Ellison's Java client desktop idea, waded through the seas of SOA and SOA related hype material, so I have a bit of an idea what "Big computer with software somewhere + you" means. In this 'modern age' it's dubbed 'Cloud computing', though to me it looks like the same old idea that has been presented by various people in the past but with new labels. With all these platforms presented in the past, there was really one issue: what was the problem they all tried to solve? Why would one want to use it? With Cloud computing, that same old issue hasn't been solved.

"I built it, you run it"

One aspect all these 'big computer with software + you' systems tried to sell was that they could run the software you wrote for you and you didn't have to worry about a thing. Well, not to worry about a lot, but still you had to worry about things, as the system was still Yet Another Platform with its own set of characteristics, flaws and weaknesses and most importantly: differences with the development- and test environment the software was written with.

The problem with software once it is written, tested and ready for deployment is that last stage: will it run in the environment on-site the way it runs locally in the test environment? And is that on-site environment easy to maintain?

In other words: the problem is that the environment the software has to run in isn't necessarily the same as the environment the software was written with / tested in, which could cause a lot of problems during deployment and after deployment. Other aspects like updating the environment due to security flaws, bugs in software etc. are also factors which add to the overall unpleasant experience of deploying and keeping software running.

So the answer to that problem should be a system which provides the following things:

The environment equal to the one the software was written and tested with
The resources to keep the software running when the software requires them.
The security that the software keeps running, no matter what.

In other words: the software engineers built the software, tested it and defined the environment (as they've done that for development and testing anyway) and shipped that in one package, and at the place where the software has to run, that exact same environment is provided, together with the resources required (like memory, cpu, a database connection). So "I built it, you run it". How the environment is re-created isn't important, the important thing is that the exact same environment is provided to the software, 24/7.

Are EC2, Azure and other cloudware solving the problem?

No. They provide Yet Another Platform but not the same environment. As they're yet another platform, you've to develop for that platform. The most typical example for that is that the newly announced application server from Microsoft 'AppFabric', has two flavors: one for Windows and one for Azure. Why would anyone care? Isn't it totally irrelevant for a system in the 'cloud' what software (or what hardware) it is running? All that matters is that it can provide the environment the developer asked for so the developer knows the software will run the way it was intended.

Let's look at a typical example: a website of some company with a small database to serve the pages, a small forum and some other data-driven elements, not really complex. Today, this company has to hire some webspace somewhere, database space, bandwidth and most importantly: uptime. To make the web application run online, it has to match the rules set by the hosting environment. If that's a dedicated system, someone has to make sure the system contains all software the web application depends on, that the system is secure and stays that way. If it's a shared hosting environment, the web application has to obey the ISP's rules of hosted web applications, e.g. can use 100MB memory max., can't recycle more than 2 times in an hour etc.

When Patching Tuesday arrives, and the web application runs on a dedicated server (be it a VM or dedicated hardware, doesn't matter), someone has to make sure that the necessary patches are installed, and that those patches don't break the application. Backups have to be made so if disaster happens, things can be restored. These all count as 'uptime' costs.

With a VM somewhere on a big machine this doesn't change, you still have to make sure the VM offers the environment the application asks for. You still have to patch the OS if a patch for it is released, you still have to babysit the environment the application runs in or hire someone to do that for you, but it always involves manual labor to make sure the environment online is equal to the environment during development and testing.

In the whole keynote I didn't hear a single argument how Microsoft Azure is doing this differently. Sure I can upload some application to some server and it is ran. However, not with the environment I ask for, but inside the environment Azure offers. That's a different thing, because it requires that the developer has to write software with Azure in mind. If I have a .NET web application running on a dedicated server which uses Oracle 10g R2 as its database and I want to 'cloudify' () that web application with Azure, I can't because I have to make all kinds of modifications, for example I have to drop the Oracle database for something else and also make other changes as the environment provided by Azure isn't the same as the one locally.

EC2 and other cloudware do the same thing, they all provide 'an' environment with a set of characteristics, but not your environment. So in other words, they're not solving the problem, they only add another platform to choose from when writing software. Like we didn't have enough of those already. Sure, they offer some room for scaling when it comes to resources, but what happens when the image has to reboot due to a security fix that had been installed? Is the application automatically moved to another OS instance? Without loss of any data in-memory, so it looks like the application just ran along fine without any hiccup?

So what's the solution? What should Cloud computing be all about instead?

It should be about environment virtualization. I give you a myapp.zip and an environment.config and you run it. And keep running it. All dependencies on software of my application, like 3rd party libraries, are enclosed in the application's image. That's not an image of an OS with the app installed, it's just the application. The environment.config file is a file which contains the description of the environment that the software wants, e.g. .NET 3.5 sp1, Oracle 10g R2 database, 2GB ram minimum, IIS7, domain name example.com registered to app, folder structure etc. etc. So I outsource any babysitting of the environment of my application.

That is incredibly complex. It might not even be doable. But it's the only way to make cloud computing something else than a new name for an old idea, despite the long list of well-known names who showed an even longer list of UIs and tools during a keynote.

Can Azure do what I described above? I honestly have not the faintest idea, even after watching the keynote yesterday and by reading up some marketing stuff. That doesn't give me confidence, as it's in general not a good sign if a vendor has a hard time explaining what problem a product solves.

LLBLGen Pro v3.0 sneak-peak video

FransBouma — Mon, 19 Oct 2009 11:27:23 GMT

I created a small video (flash movie) of a neat feature of the upcoming LLBLGen Pro v3.0 designer: creating a typed list definition from search results obtained in the designer by running a custom piece of code (C#, with Linq to objects. VB.NET is also supported)! So any query you want to run on the model meta-data is allowed.

Please click on the screenshot below to open the page with the video. You need flash to play the video. No sound included.

LLBLGen Pro v3.0 is scheduled to go beta at the end of 2009 and will support the LLBLGen Pro runtime framework, Entity Framework, Linq to Sql and NHibernate.

Please click the screenshot to view the small video. (opens in new window)

Update: uploaded a better html file, so the video isn't resized improperly.

Happy 6th anniversary, LLBLGen Pro!

FransBouma — Mon, 07 Sep 2009 08:46:30 GMT

Today, it's been exactly 6 years ago we released the first version of LLBLGen Pro, v1.0.2003.1 after a development period of roughly 9 months (Sunday september 7th 2003, late in the evening). It was a big gamble, would it succeed or fail? We got our first customer within 9 minutes after release and we then knew it would be a success. And it still is, with thousands of companies using it world-wide, from small mom & pop shops to the biggest banks on the planet. Honestly, we hoped for success but that it took off this big was beyond our expectations. A big thank you! to all of our loyal customers who trusted our work in the past 6 years and who are keep trusting it.

Needless to say, we're still going strong and are looking forward to v3.0 which is scheduled to go beta at the end of the year. It will actually be our 10th major version (1.0.2003.1, 1.0.2003.2, 1.0.2003.3, 1.0.2004.1, 1.0.2004.2, 1.0.2005.1, 2.0, 2.5, 2.6) since the initial release, and will be the first release which will support other frameworks besides our own runtime framework and will also add another major new approach: model first.

Looking back at those 6 years, I think the biggest asset we deliver is quality you can count on. From the get-go we strived for that aspect, with top-notch support which is free and bug-fixes which are usually delivered within 24 hours. A data-access technology isn't something you just pick out of a pool of tools, it has to fit your way of how you want to write software and work with data, what you want to do in your application and above all, has to be rock-solid so you don't run into surprises, unexpected lack of support for common features or a wall of disbelieve when you ask for help or support or a bugfix. So in other words, a data-access technology is one of the pillars your software has to count on. From the start we realized this and with every feature we added we made sure that indeed, our customers could indeed count on our work and the quality we deliver.

During these 6 years, we worked full time on implementing more features, like a new paradigm (Adapter), support for more databases, multiple ways to do inheritance, more powerful code generator engines, template editor, linq provider etc. and it was and still is simply great working on this every day. On to the next 6 years!

LLBLGen Pro and SQL Azure

FransBouma — Sat, 05 Sep 2009 13:48:04 GMT

LLBLGen Pro works with SQL Azure, that is, the generated code and the runtime library. There are a couple of things you should be aware of, and I'll enlist them briefly below. The thing which doesn't work is creating a project from a SQL Azure database, as SQL Azure has no meta-data tables publicly available to the connected user (also a reason why for example SQL Server Management Studio doesn't work with SQL Azure at the moment)

The things to be aware of are the following when you want to work with SQL Azure and LLBLGen Pro are the following:

SQL Azure doesn't support catalog names in the queries. As LLBLGen Pro supports multiple catalogs per project, and thus cross-catalog queries, you can only use one catalog in your project.
To avoid catalog names in the queries, you should use the feature called 'Catalog Name Overwriting', which simply means that you configure the runtime to use a different string than the catalog name. You should configure the runtime to overwrite the catalog name of your project to "", so the catalog name is not emitted into the SQL query.
Our tests and those performed by some of our customers showed that if you use a schema which isn't the default schema, it also seems to make SQL Azure throw errors. So to be safe, either use 'dbo' as the schema, or if you must: define the used schema as the default schema of your user using:
ALTER USER username WITH DEFAULT_SCHEMA = schemaname

That's it. If you make sure of that, which are a simple couple of steps to check, you can use LLBLGen Pro generated code on SQL Azure. Happy azuring!

I'm now also on Twitter

FransBouma — Wed, 19 Aug 2009 16:16:37 GMT

Direct profile url: http://twitter.com/FransBouma

I don't promise to follow everybody, but for the few people who want to follow what I have to say, I'll try to use it for more smaller blurps than this blog, as my blogposts here seem to be pretty big (and time consuming to write) overall.

Think first, 'doing' is for later

FransBouma — Sun, 26 Jul 2009 15:44:24 GMT

In the comments section of Ayende's blog, I recently debated the usage of principles like the ones in SOLID and argued that these principles aren't really the important thing to focus on. Instead, people should focus on thinking. In the Netherlands we have an old saying: "Bezint eer ge begint", which translated to English is something like "Think everything through before you start". Now, before I wake up the anti-Waterfall people, I'd like to add that this post isn't about Waterfall at all. Instead, I'd like to line out how I write my software, how thinking is an essential part of every step I take in the whole process and will illustrate it with an example which hopefully will illustrate that some extra time spend on the thought process before writing any code is very valuable.

The main thing to understand about this post is that it's not a guide to design a whole system as it is about how to approach and successfully implement features. Features are pieces of functionality the software needs to have at various abstraction levels. TDD people might describe them as user stories, I stick with feature because I don't use TDD so I use a different name to avoid confusion. The example feature I'd like to use in this case is the following: say you are working on an O/R mapper designer and you have the ability to define value types. Value types are types which have one or more attributes (fields) but don't have an identity of their own like Entities do.

A good example is 'Address', with the fields 'StreetName', 'City', 'Zipcode' and 'HouseNumber', but you could also define value types with just one field, like 'EmailAddress' with the field 'Value'. A value type with just one field can be useful if you want to place logic related to that single field (like in this case the check if the value for the email address is valid) with that field and do that just once: every time you now need to define an EmailAddress field in an entity, you can set its type to 'EmailAddress' instead of 'string' and validation is built in, as you already did that once for the value type 'EmailAddress'.

This feature will give a couple of problems, where one being pretty complex: if you want to add the feature to add new fields to a value type definition, you might run the risk of creating an infinite loop: a field in Address which also is of type Address. Or worse, you could have a field Foo inside Address which has a field which is of type Address. How do you prevent that from happening? I'll show you how to solve this in a general way and also how to get there.

Let's go back to our feature: adding a field with a valid type to a value type. Which steps to take to add this feature successfully? The steps to take are the same steps I'll always take with every feature, I've summed them up briefly in the list below:

Think first. 'Doing' is for later. Think everything through, use reasoning to learn more about the feature, try to think of all possible problems related to the feature and possible solutions.
Analyze what you need based on step 1. Try to find generic, proven algorithms and data-structures which might be what you need, and build on top of that.
Document the results of step 1 and your analysis result from step 2: write which alternatives you've investigated and which one you've picked and most importantly, why.
Prove your algorithms first. This is not that hard and doesn't require any code
Implement algorithms / data-structures as designed. Then check whether you implemented the algorithm correctly. This is doable by simply looking at what you wrote. How you implement it, is dictated by the algorithm: which steps are there to take.
Test the implementation of what you wrote. As the algorithm is already proven, the tests prove the implementation. If you want, you can also start with this before 'Implement algorithms' by implementing it using mocking and tests. That's not the point, the point is that the code is the end-result, not the start.

I gave them numbers in the list above to easily refer to them. Let's address them one by one and along the way see how our example plays a part in this.

Step 1: Think first

This might sound like pretty common sense, but it's the most important step. You should spend considerably time in this step, thinking things through, what is the feature all about? Which side effects can be recognized? Should you split it up in various sub-features/parts? If you don't feel comfortable with what you have discovered during this step, like you don't really have a good feeling about how to approach it further, you shouldn't proceed yet. This step is the foundation of what you will do in future steps. What's also important to notice: no coding. This is all thinking, not doing. The doing is for later isn't there for nothing. The most important mistake people can make is act before thinking things through. You'll see why in a bit.

In the feature at hand, adding a field to a value type, we can discover several things: we have to validate on the name within the value type, the field has to be correct by itself (e.g. has to have a valid type, name etc.). We also recognize the necessity of avoiding an infinite loop within a value type as described above: we can't add a field F to value type VT if by adding that field, an infinite loop is created inside VT, or indirectly in a containing type of VT. A field can get a different type, which also should avoid this infinite loop. We can also recognize the requirement that when we add a field to a value type VT, this field is then also required to be mapped onto a target, but for the sake of simplicity, we don't go further in the mapping scenarios here.

I also leave a notification system for other subsystems and undo/redo outside the scope of this post but the requirements for these aspects are the same: they too have to be thought through, how to solve them and have we solved them before already, if so, how did we do that? An experienced software engineer solves things often in the same way as s/he already knows what to do when different kind of small problems occur. Software engineers which aren't that experienced are often faced with these decisions and don't really know what to do. If you find yourself in such a position, don't be arrogant and deny it, but simply take the steps of thinking it through, try to look if (based on theory!) you have made previous attempts to solve the same problem elsewhere.

After thinking the feature through, we look at our findings, and see that for the most part there's some straight-forward validation needed for the name and the correctness of the field and also a complex validation for the infinite loop protection. As we have thought about our feature and the implications, we can proceed with the next step. Though, don't have the illusion that the thinking is now done, the next steps still require a lot of thought.

Step 2: Analyze what you need

When analyzing the results of step 1, we see there are three kinds of validation we have to solve:

Validation of the field itself (does it have mandatory properties set?)
Validation of the field within the value type it is added to (does it have a unique name within the value type?)
Validation across all value types (does it create directly or indirectly an infinite loop?).

The validation for a) is straight forward, and we have several standard approaches for that: add a validation method to the class or a property itself which signals that the field is valid, and internally the field maintains the value for this by checking itself whenever it changes. There might be others as well, all have their strong points and weak points. As this is a common problem, it's likely that in an earlier feature it already has been solved, so we can leverage that analysis and see if it applies here too.

The validation for b) is similar to a), we simply apply it at the value type level instead of at the field level. We can re-use our analysis results and verify whether it applies here too. Validation for c) is something else though: how to approach this? It depends on what you like most: visual approaches on a white board, or reasoning in theory without images. The key is that you avoid writing code. This is perhaps going to be considered a bad thing in some people's eyes, but it's key here, as writing code makes you fall into the trap that you'll think that the code is what you are trying to express in executable form, but it's not, as you haven't thought of what you want to express in executable form (as you haven't decide what algorithm / data-structure to use!) so you can't possibly write code which represents that.

The problem of c) is at first pretty straight-forward until you discover that it is actually less simple because you have to investigate a lot of different paths and different situations, as the value type VT1 you add the field to might be inside another value type VT2, which is inside another value type VT3 so the field to add isn't allowed to be of type VT1, VT2 and VT3. To find all those paths requires an algorithm which keeps track of all kinds of things, which makes it less easy to implement, maintain and test. So what's there to do?

We first step away from the idea that we're the first to solve these kind of problems. To be able to find a general theoretic solution which is already proven to be correct, we have to make our own problem more generic and realize that others before us have already solved this same problem with theory (not code! This isn't about copying pieces of code from Google, it's about re-using theory). In this case, we have to analyze when an infinite loop occurs in our situation, in short: what are the criteria for that. It comes down to: which types should a field F not have if we add it to value type VT1? Obviously VT1, but which value types should we also avoid? If you draw a picture of this on the white board, you'll see that every value type known in the system (so every value type the field F can have) which is directly or indirectly referring to VT1 is not allowed, as by setting the type of F to one of these value types will automatically create an infinite loop.

If you don't see it directly, draw a picture on a piece of paper or whiteboard of the following: VT3 points to VT2. VT2 points to VT1. VT4 points also to VT2. VT5 points to VT6. How to find the value types referring to VT1 directly or indirectly? Look at your drawing at the whiteboard. It will likely look like a graph, a directed graph. Graphs are one of the most important data-structures you'll need in software engineering. One of the key aspects of graphs which makes them so great is that for graphs (directed, non-directed) a lot of algorithms have been discovered and described (and proven to be correct) which we can re-use without doing any effort. The nice thing about these algorithms is that they often solve problems we face every day (like ordering elements which are related to each other, or this one: which paths are there?).

The algorithm we need here is Transitive Closure. Transitive closure gives all pairs of vertices ('nodes') in the graph which are directly or indirectly connected to each other. So in our graph example above, it will give VT3 -VT2, VT3 - VT1, VT2 - VT1, VT4 - VT2, VT4 - VT1, VT5 - VT6. This means that I can travel from VT3 to VT2 and VT1, from VT2 to VT1, from VT4 to VT2 and VT1 and from VT5 to VT6. This also means that all value types from which I can travel to VT1 are therefore not allowed, as these all refer directly or indirectly to VT1! This leaves us with VT5 and VT6 which are the only value types allowed for a field F added to VT1.

We're not done yet. If you dare to look at the wikipedia page I linked to above, you'll likely stall at the complex math formulas you're faced with. Don't worry, these are part of the mathematical background of the concept. As we're software engineers, we need to look for theory about an algorithm which describes transitive closure of a graph G so we can implement it. As this is a problem which has been solved before us, some very clever people have already come up with what we're looking for: there's a very efficient algorithm designed and proven for transitive closure: the Floyd Warshall algorithm.

You might now wonder how you would have gotten to the same conclusion. One of the most important steps is to avoid home-brew algorithms unless you really really have to. A typical bad habit of software engineers (and don't worry, I also still make that same mistake once in a while) is that they think they're the first to face a problem and also that there's nothing generic discovered yet which could possibly help them out so they have to cook something up themselves: the home-brew algorithm. The downside of home-brew algorithms is that they're not yet proven to be correct. For general algorithms like the Floyd Warshall algorithm, it is proven to be correct so we can skip that important step: we don't have to think about if it will work with whatever graph we throw at it, it will. We just have to type in the three nested loops and we're done.

Now that we've done our analysis and have come up with good candidates of how to solve the problems we faced with this feature, we can proceed with the next step.

Step 3: Document the results

Software engineers tend to dislike to document their work, however it's very important that we document our thought process, and especially why we took decision A and not B. The key of documenting the design decisions and which alternatives were not chosen, is that if we have to face a similar choice again, for example in two years time we have to alter this piece of code and have to look up why the feature was designed with a graph and a Transitive Closure algorithm, we will learn from the design decisions that the graph approach was a good one because it was a proven path: we don't have to worry about the fact if the algorithm would give us all the paths we would be interested in, it will. So we can keep that implementation and don't have to worry about alternatives being better: we can learn from the documentation we made, the alternatives are not better. This documentation for this feature doesn't have to take ages to write, it might even be half a page, as long as it contains the information that explains why which alternative is chosen.

After documenting our findings from the first two steps, we proceed with the next step.

Step 4: Prove your algorithms first

Here we'll see that what we have invested in pays of. We use a proven algorithm so we don't have to do any work, it is already proven to be correct. Would we have chosen an algorithm we designed ourselves, we would have to prove if it works. This can be a bit time consuming and, I'll admit, boring, but it is worth it. One of the key aspects of proving an algorithm is that you think each step through, you define the pre-/post conditions, what will make it go wrong, and perhaps even you'll see flaws in the algorithm and have to start over. It's easier to do this without code, because code can contain bugs due to bad implementation. If you write tests for your code (or start with tests when implementing code), your tests not only test the implementation but also the algorithm. If the test fails, is that due to an algorithm error or due to an implementation error or both? If you prove your algorithms first, you know it's an implementation error.

Some say that this step is not doable or it's even worthless. However it's easier than you might think: write pre/post conditions down for each step, and reason about the algorithm you designed, when will each step break and why not? It doesn't have to result in hard-core math, it's often already enough to think through each step of the algorithm what the pre/post conditions are and when a step will fail to spot problems.

As we've already proven our algorithm by pointing to the work of others (Floyd and Warshall) we can move on to the next step.

Step 5: Implement algorithms / data-structures as designed

This is the step where we actually will write code. For the people using TDD, you will likely combine this step with the next one by writing tests first and using mocking to work towards a working implementation, but that's really semantics. In step 2, we decided to use a directed graph algorithm, so we need a graph data-structure which can handle directed edges. Writing a graph class is straight forward, the only thing you have to make a decision on is how to store which vertices are connected: using adjacency lists or by using an adjacency matrix. Both have strong points and weak points, you can learn more about them by reading the linked wikipedia articles. You can also decide to use a prefab graph class, as there are several graph classes already written for .NET, it's up to you.

Once we have the graph class, we can implement the Transitive Closure algorithm. As this algorithm is really about three nested loops, it's straight forward. Below is the Transitive Closure implementation of Algorithmia's DirectedGraph class:

/// <summary>
/// Returns the transitive closure of this graph using the Floyd-Warshall algorithm.
/// See http://en.wikipedia.org/wiki/Transitive_closure and http://en.wikipedia.org/wiki/Floyd-Warshall_algorithm.
/// </summary>
/// <returns>The transitive closure of this graph.</returns>
public DirectedGraph<TVertex, TEdge> TransitiveClosure()
{
    DirectedGraph<TVertex, TEdge> result = new DirectedGraph<TVertex, TEdge>(this, this.EdgeProducerFunc);
    if(this.EdgeProducerFunc == null)
    {
        throw new InvalidOperationException("The EdgeProducerFunc isn't set to a value.");
    }

    foreach(TVertex i in this.Vertices)
    {
        foreach(TVertex j in this.Vertices)
        {
            foreach(TVertex k in this.Vertices)
            {
                if(!j.Equals(i) && !k.Equals(i))
                {
                    if(result.ContainsEdge(j, i) && result.ContainsEdge(i, k) && !result.ContainsEdge(j, k))
                    {
                        result.Add(this.EdgeProducerFunc(j, k));
                    }
                }
            }
        }
    }
    return result;
}

Is this implementation correct? We can of course test this with some unit tests. We can additionally to these tests, check whether we have indeed implemented everything correctly by looking at the steps in our algorithm and then go to the code to see if we made a proper projection of the step to the executable form: the code. If not, we have to change the code, not the algorithm. Don't think lightly of this and be careful not to cut corners by 'assuming' it is OK. A human isn't a very good source code interpreter but the developer of the code should at least try hard to check whether the implementation is indeed how it should have been. That already will catch obvious bugs and mistakes, as the algorithm is correct so we have to worry only about the implementation.

There are several implementations possible of a given algorithm A. It depends on how you interpret each part of the algorithm and how you think it's best to implement these parts. This might cause problems but these are caught by this step as well because you've to verify what you wrote is indeed what you should have written. This is also the place where code reviews come into the picture and the reason why they work: other people will look into how an algorithm is implemented exactly, and as each person will actually make the same projection from the algorithm to code, any difference in what they would have done themselves vs. what they are reviewing will trigger discussion and will in the end result in better code.

One might wonder if this is really worth the effort. Yes it is. The exact same algorithm implementation above for example is also used to find inheritance loops when inheritance is defined between entities (as it's in fact the same problem) and you can solve other problems with the same algorithm as well. Furthermore, the more time you spend on making sure the code is actually of high quality and correct, the less time you'll spend on maintenance later on or bug-fixing, as the code and more importantly, the reasoning behind it, is well understood and debated, analyzed, documented and considered the best alternative of the list of possible options.

For this particular feature we picked a data-structure and algorithm which were already well-known. As we've implemented them in our code-base, we can re-use these general building blocks whenever we need to solve similar problems. The advantage of using well-known data-structures and algorithms is that they're not subject to change because they don't evolve: their definition is well-formed, the problems they solve are well defined and therefore it's safe to use precisely these kind of building blocks to base your own code on instead of home-brew data-structures and algorithms. Always try to use general, well-known data-structures and algorithms. The more the better, and once implemented, you can re-use them without worry: a transitive closure is always a transitive closure. A topological sort is always a topological sort and so on.

Let's move on to the final step.

Step 6: Test your implementation

If you're using TDD, you likely already have the tests written as part of the previous step. If you're not using TDD, you should now test your implementations of the parts in the previous step. As the algorithm or algorithms and data-structures, are already proven to work, we can focus on if we have written the right code to implement them. Because of the work in the previous steps, we have in-depth knowledge of how things should work, what each step in the algorithm should do, and thus what happens in each step of the feature. This leads to tests which test on cases where we can expect problems, for example around the pre/post conditions we found. Keep in mind that in our example, there is an unlimited amount of graphs to test, and you can't write tests for every single one of them. Using proved algorithms helps tremendously in this case, as the proving of the algorithms already solves you from the quest to test for every possible input. Let me violate DRY and repeat myself here: Always try to use general, well-known data-structures and algorithms. The more the better.

When our tests succeed, we can be pretty confident that our feature works. The way we implemented it from start to end has given us a lot of valuable assets: we have documented design decisions for later use, for ourselves but also for others who have to maintain our work, we have written generic data-structure and algorithm classes of proven algorithms which we can re-use in a lot of other situations, and we have a working feature which is based on theory and the reason why the code is the way it is can be tracked back to the theory and information we formulated and collected in the early steps.

Conclusion

In this post I tried to shed some light on a different form of software engineering, which is based on a thought process to base code on a solid theoretic foundation. I deliberately avoided any pattern, fancy methodology or principle like the ones from SOLID, because I wanted to focus on the thought process behind writing code, why we write the code we write and that that process doesn't started within a code editor but that it ends within the code editor and starts somewhere else. I have tried to illustrate how I write my software on a daily basis and hope it is valuable to others.

Follow-up on the 'Firefox v3.5 fiasco'

FransBouma — Sat, 11 Jul 2009 09:02:28 GMT

(Follow up to: The Firefox 3.5 fiasco)

I'd like to inform the audience that the people over at NSS, the sub-system which is responsible for the disk-trashing behavior of Firefox 3.5 (and the accompanying delays on startup) on some systems, has worked on a fix for this which appears to be scheduled for FF 3.5.1. You can read the discussion by starting here (which lands in the middle of the bug comments, but the comments above the one linked are basicly bickering comments over what to do to the symptoms instead of really fixing it at the root)

It's good to know that the NSS folks finally listen in and will use CryptGenRandom when available (it's a windows subsystem method) and will only revert to disk-based entropy collecting when CryptGenRandom isn't believed to be as solid as it is on 'modern' OS-es like Windows XP and up. I still think that MS has patched Win2k's kernel code enough to make CryptGenRandom (which is essential for the TCP stack as well) solid enough, but it's their call (IMHO, people should make choices based on evidence based arguments, but as Win2K is rather old and no longer supported by MS anyway, it's not such a big deal)

Let's see whether this patch will turn out to be as good as it looks today. I'm glad Mozilla is keen on fixing this pronto, as FF 3.0 is scheduled to be non-supported software starting in January 2010.

So what can be learn from all of this as a developer? In my opinion, the true lesson to learn here is that no-one is perfect and that it's key to keep listening to what our users experience when using the software we wrote, so problems can be solved better and choke points can be dealt with. It's all too easy to simply close the eyes and ignore problems reported by perhaps a minority of the users by cooking up excuses for not dealing with them, but that's not the solution: the problems won't go away by ignoring them, the vocal minority might actually be representing a big non-vocal group or worse: a big non-vocal groups of ex-users. That's not to say that every problem ever reported by a user should immediately be fixed: unless you have unlimited time and resources, it's practically not doable to achieve that, but we should at least try and investigate whether these reports might cover bigger problems, might affect bigger groups than a sole individual.

The Firefox 3.5 fiasco

FransBouma — Thu, 09 Jul 2009 08:59:38 GMT

(updated: replaced 'trashing' with 'thrashing' as indeed, I meant 'disk thrashing').

As a Firefox user, I was delighted when Mozilla released Firefox v3.5. It was advertised as a new milestone in browsing, with more standards being supported, new engines for javascript and web content rendering and the intarweb would appear to me faster than ever before. As a person who has a bit of a blind spot for marketing and everything related I was a tad skeptical, but I thought "What the heck!".

So I went ahead and downloaded the installer at release day and after fighting with the usual plug-in upgrade mess, I was able to run the browser for the first time and lo' and behold, the web felt like I was back in 1994, when no-one but the Real Geeks had web sites and everything was lighting fast. Live was good.

The next day, with a fresh cup of coffee in my hand I started my beloved Firefox 3.5 browser on my freshly booted system. I was expecting to see the browser dialog within seconds to re-experience the web at light speed, but nothing happened. Well, something did happen, my PC's hard-disk was busy like I was running three virus-scan sessions at the same time. After 35 seconds or so, it finally managed to find all the bits and pieces it apparently needed and showed me the familiar face of the Firefox browser dialog and I was on my way to the outside world!

Suddenly, a small, screechy voice in the back of my head tried to make a point. That voice, which sometimes cries through a developer's head when s/he writes a piece of code which isn't in the format the voice owner likes, at which point it desperately tries to convince you to do something else instead by giving unwanted advice like "Wouldn't it be better if... " and other lovely comments no-one wants to hear, that voice made a remark in it's usual dull way about those 35-something seconds before the browser really started. As with similar occasions, I didn't pay much attention to it. Every Firefox instance I started was lighting fast, and showed up 2 seconds or even faster, it must have been something else which had caused the delay at startup. I know, it didn't sound very convincing the first time either.

That afternoon, I had no browser window open and started a new one. Again I was rewarded with a long pause, disk crunching and a blank screen, until 30-35 seconds had passed and Firefox 3.5 was awake and ready for duty. "Hmmm..." I thought. Voicy in the back of my head was awake again too, with random babbling about "Told you so", "I'm not gonna repeat myself" and similar wisdom, the usual. Could it be windows or some service caused all this disk thrashing and the delay? I more and more got the feeling Voicy was right (I hate that feeling) and there was something fishy about all this.

I didn't want to wait 35 seconds every time I started a browser, so I wondered what to do. Then I realized I was an end-user of this application, this browser. And what do end-users do? That's right, they go to the support offering of the vendor. Mozilla has a nice forum system so I searched it a bit to see if fellow Firefox users had similar delays. Well... you could say... yes indeed. And not only delays of 30-40 seconds but some had to wait minutes or even worse: after starting Firefox, it went into a coma and never truly woke up. Mozilla also found out that more and more people had the same problem and added a sticky thread to their forum. You can reach it here.

That forum thread revealed what the true cause was of this disk thrashing and delay at startup. I have to warn you though. If you're a developer, your software engineering fire will die a little when you read the true cause and from then on you will have to fight off thoughts of giving up development altogether and apply for a job in marketing or HR. So what was it, what's the cause of this slowness? It's NSS. What? The Network Security System. It turns out that NSS needs to do all kinds of encryption and other security related tasks (which seems kind of logical), and for that it needs random numbers. Sounds reasonable, right? Well, it kind of does.

True random numbers are hard to produce, because in a computer system, nothing is really random, it all is a result of some action which was a result of some action etc. etc. The clever boys and girls of the NSS team had to crack this problem: how to get 'true' random numbers which are as random as possible? Instead of using the randomization functionality of the underlying operating system (which has this feature build-in as every TCP stack for example needs it), they did what Mozilla in general always does: they re-invented the wheel. Nothing against re-inventing stuff, don't get me wrong, not every wheel is as equal as the other one, and you can never have enough good, re-invented, shiny wheels. Though, the downside of re-inventing wheels is that along the way you can't make mistakes, it has to be better than the previous invented wheels. No-one wants to use your square new wheel for example.

To solve the problem of the randomization, the NSS team came up with something clever, something so great, that no-one else had ever thought of that before: they decided to read the files in all possible temp folders on disk with multiple threads so these files can be used as seeds for the randomization. Brilliant. Temp folders! Why hasn't anyone else thought of using a disk-based resource for random number generation! I mean, these folders change every couple of milliseconds, have immediate access, no latency to read their contents and are never filled to the brim with useless cruft!

That is, if you're on the NSS team. In the outside world, things are a tad different. You see, Firefox v3.5 reads the Internet Explorer Cache and the central Windows temp folder in your user profile, through its NSS subsystem. Not only is it, in my humble opinion, not done to read another application's caches or temp folders, it's also amazingly ignorant towards the real bottlenecks of our modern computers: hard-drives. If you're using a virus-scanner which is set to paranoia mode, this whole temp folder traversal by NSS will be even slower because every file accessed will be scanned by the virus scanner. Over and over and over again. And what happens if the user doesn't do anything else but browse with Firefox, so these temp folders will not change (or are empty)? Isn't using file reading the worst way to obtain a seed for randomization?

I used sysinternals' Filemon tool to check which folders and files Firefox was reading and along the way I also saw they read all fonts up front. All of them. That too seems rather odd for a browser who claims to be the fastest browser. How many fonts do you need on a random (pun intended) webpage? Besides the default ones and a few common ones? 2, 3? Would it hurt anyone if these are read 'on the fly'? Not compared to the delay in startup time for a browser dialog when you have many fonts installed.

NSS is open source, but it's not something you can fix yourself, unless you compile the browser as well. The problem is that NSS is a security component and therefore needs to be signed by Mozilla to be used in Firefox. This means that recompiling the NSS dlls won't work, Firefox won't accept them (which is logical, it's a vital part of the security system, heck it is the security system!). Though, why should I even bother? It's 2009, for crying out loud. After 15 years of web-browser development, the human race should have produced a web-browser by now which is worth using, without silly startup delays which last minutes or even longer. After all, in this case, I'm an end-user.

Mozilla on their forum says 'a' developer is working on a fix and they 'hope' that this developer is able to fix it. That's not sounding very promising to me. This is a top priority issue, Mozilla, unless you want droves of people drop your browser for the competition. There's already a fix available, Firefox v3.0 didn't do this disk thrashing and is able to communicate security over the internet, at least that's what you always told your users. In other words: the NSS version in Firefox 3.0 was capable of creating random numbers and doing encryption without the necessity of reading a competing browser's disk cache nor the OS' temp folders. In case you wonder, Mozilla, no, I'm not going to advice friends and family members to use Firefox 3.5 anymore till this is fixed. Not that you nor your droves of developers lose any sleep over that, at least I hope not, but with me I'm pretty sure more people will do the same: move away from Firefox or revert back to an older version and wait with advice to friends and family about Firefox 3.5.

I'll revert back to Firefox 3.0 till this is fixed, or move to another browser (although I find Chrome a bit too much Google in one package). If you're planning to upgrade to Firefox 3.5, be aware of the issue I described above and do realize that it's not something you can learn to live with, as the delay will occur randomly (pun intended) during the day: sometimes starting a browser is fast, however an hour later it can take again 30-40 seconds or longer.

Yes Voicy, I'll listen to you more. At least more often.

Linq: Beware of the 'Access to modified closure' demon

FransBouma — Thu, 25 Jun 2009 08:32:21 GMT

If you're using Linq and Resharper, you've probably seen the warning Resharper shows when you use a foreach loop in which you use the loop variable in a Linq extension method (be it on IQueryable<T> or IEnumerable<T>). In case you don't know what it is or what damage it can do if you ignore the issue, I'll give you a database oriented query (so on IQueryable<T>, using LLBLGen Pro's Linq provider) which creates a dynamic Where clause based on input, the typical scenario you should be careful with when it comes to this particular problem.

var customers = from o in metaData.Order
		join c in metaData.Customer on o.CustomerId equals c.CustomerId into oc
		from x in oc.DefaultIfEmpty()
		select new { CustomerId = x.CustomerId, CompanyName = x.CompanyName, Country = x.Country };

string searchTerms = "U A";
var searchCriteria = searchTerms.Split(new string[] { " " }, StringSplitOptions.RemoveEmptyEntries);
foreach(var search in searchCriteria)
{
    customers = customers.Where(p => p.Country.Contains(search));
}
var ids = (from c in customers select c.CustomerId).ToArray();

The above code snippet has the demon embedded into itself, likely without you noticing it. Can you spot it? (Ok, I already gave it away a bit with the foreach loop hint).

The problem with the above query is that it will produce a WHERE clause in the SQL query with two LIKE statements which both filter on %A%. How's that possible? The cause is in the 'Access to modified closure' problem: search is a local variable. The first time the foreach is ran, search will have the value "U". The .Where() extension method will add a MethodCall expression with a call to Where(lambda) with inside the lambda among other things a ConstantExpression referring to the local variable search for the value. And that's precisely the problem: when the foreach loop is looping again, search will get another value: namely "A". As there are no more values, the loop ends and the query is executed.

Well, executed is more complex than it sounds: first, the expression tree has to be converted into SQL. When the linq provider runs into the two .Where() extension method calls, it evaluates the argument, which is a LambdaExpression which contains a ConstantExpression which refers to... a local variable called search. It can't do anything else but reading that variable, which has the value ... "A" for both, as it reads the same variable. So it's not storing the constant value search has when the call to Where and Contains is made, it's storing a reference to the local variable.

How to fix this? It's pretty straight forward: create a new local variable:

foreach(var search in searchCriteria)
{
	var searchTerm = search;
	customers = customers.Where(p => p.Country.Contains(searchTerm));
}

With each iteration, it creates a new local variable, and thus each Contains call will refer to a different variable and thus the SQL query will contain the two LIKE predicates the way it should, one with %U% and one with %A%.

This subtle issue pops up with Linq to Objects as well, so beware when you pass the foreach loop variable to a Linq extension method: if the query doesn't run at that same spot, you likely will run into this problem and will have an obscure bug to track down.

Happy hunting

Multi-value Dictionary C# source code (.NET 3.5)

FransBouma — Mon, 18 May 2009 08:52:00 GMT

By popular demand, I've published the C# source code of my Multi-value Dictionary class, which can also merge dictionaries into itself and which implements ILookup<T, V> as well. It's part of Algorithmia, our upcoming data-structure and algorithm library which will ship with LLBLGen Pro v3.0 later this year. The code is released under the BSD2 license, see the enclosed readme.txt. The class comes with its own general purpose Grouping<T, V> class as well and of course its own ToMultiValueDictionary() extension method.

I hope this is useful to others.

Update: it seems that if you run a Linq query (Linq to objects) over the MultiValueDictionary, the compiler and intellisense get confused as there are now two enumerators and both work with the linq operators, which means you either want to remove the ILookup code from the class (which is not that hard) or explicitly state the generic arguments. It's not a big problem, though in case you run into this problem, you know the reason.

Update 2:A user pointed out that I forgot to include ArgumentVerifier, a simple class to make life easier wrt verifying arguments, so I've included that one as well.

The desperate quest for doing it 'right'

FransBouma — Mon, 04 May 2009 11:00:02 GMT

This morning I ran into an interesting design decision. The problem at hand isn't that interesting, I've solved it a lot of times before. The interesting thing is that this problem isn't always solved the same way. It goes like this: do you tell an element which is inside a container (which can be inside another container) to exclude (remove) itself from its container or do you tell the container to exclude (remove) the element? This might sound simple enough, but what is the right thing to do here? And if one is chosen, on what ground is that approach the right thing and is that always the case, no matter what the scenario might be? No, "It depends" doesn't cut it, for the sole reason that every single day probably millions of developers around the world are, in any state of desperation, searching for the right thing to do, be it for this or other problems. Check the various Q&A sites, the various newsgroups and above all, the wide range of developer blogs, articles and twitter channels, and you'll see that a lot has been, is and will be discussed about that single concept: the right thing.

When I was confronted with the decision outlined above (more on that below), I wondered how a developer with decades of experience in the trenches like myself still has to wonder about this somewhat small decision and isn't capable of instantly choosing one over the other. Is it the big fear deep in all of us that if we make the wrong decision it might haunt us and eventually will bring us down? Looking at myself, with the massive code base this decision will be part of taken into account, it does bug my mind: if I pick the wrong decision, it might hurt the system, my company, and everyone depending on that. If I have these kind of questions, there must be others with the same question, wondering the same thing: what's the right thing to do. Looking at all the blogs, articles, answers given to similar questions on various Q&A sites, indeed, there are many many people out there wondering that same thing: either showing that by giving advice how to do the right thing (dear reader, if you now start to wonder if this is a recursive blog post, you're probably right), or by asking what it might be.

So I wondered: isn't this quest to find what the right thing is actually haunting our profession and why exactly is this? Why do we care so much? And more importantly: can we ourselves solve this?

This post was partly triggered also by a blog post I read this morning by Patrick Smacchia, where he shows that a toolkit written by Jeremy D. Miller called StructureMap has cyclic dependencies between namespaces and Patrick tries to make a case that this kind of coupling is apparently not that great to have. Reading the post I wondered why anyone would give a hoot about such a thing. Don't get me wrong, I like solidly written software which allows great maintainability and extensibility without a lot of effort, but I couldn't help wondering who decides what's right and wrong and why we should care about these kind of 'rules'. A lot of these rules make sense simply because they are based on common sense, however I still have the feeling that the vast majority of these rules only work in a given scenario, however in many situations the boundaries of these scenario's are omitted, be it deliberately or by mistake. The pitfall is that if these scenario boundaries aren't given, the rule at hand starts to look like a rule which can be applied always as it apparently is a rule which is one of the ones based on common sense and is the right thing to do, as it has no boundaries/scenario given where it does work so it should work always.

With the rewrite of LLBLGen Pro's designer for v3.0 using .NET 3.5, I'm trying to do some things differently compared to what I did in v2.x. One of these things is a completely different set of data-structures to store meta-data. These data-structures give a lot of freedom to reason about the meta-data and as everything is event/observer controlled, it's very loosely coupled. However, I too ran into cyclic dependencies of namespaces (inside a common root namespace in the same assembly). When I detected this, I wondered... "should I correct this" ? But then I thought: "Why? Will the sky fall down if I leave these few cycles in?". So I did what I always do in the case when I have to make a design decision: make a pro/con list and decide what's the better option based on that list, document that decision and why you took it (so you can always read back why a decision was made, the most important thing about design documentation), move on. In this particular case, I didn't see any advantage of refactoring the code to obey some rule which is only important if you're going to split up assemblies (which I'm not planning to do in this case) so I left it in.

You might wonder why I didn't present to you right away the pro/con list of the decision I started this post with. The main reason is that I wanted to show you that there is no such thing as the right thing if there's no context given, or better: if your situation isn't known. This is very important. Today, and in the years to come, you'll likely be exposed to articles, blog posts, books, lectures and what not, written by generous people who simply want to help you out, which will tell you what's the right thing to do. I'd like you to keep one single thought in mind, whenever you read such an article, post etc.: Does the scenario this rule, this good advice, applies to actually match my scenario? If not, be careful to apply that advice without proper thinking it through. Software engineering isn't an exact science: there's no such thing as a formula where you put something in and the result is calculated, our profession is about building an executable form of what's been described as the functionality of a system, however there are no turn-key solutions to make that possible: every situation is different.

Let's go back to the decision I started with and give it some context, a scenario, the situation it occurred in. This might help you with what you thought instantly what the right thing was when you read the first paragraph. In LLBLGen Pro v3, the user can add meta-data obtained from multiple databases to a single project and choose which elements to obtain from these databases, e.g. which tables, which schemas etc. See the screenshot below for an impression.

LLBLGen Pro v3 alpha: Step 2 of meta-data retrieval wizard.

The screenshot above shows how the user obtains the meta-data, a simple click-through wizard with some fancy selection/filtering (not shown). The ability to select elements, also raises the question: "what if I selected a couple of tables I don't want to see anymore in my project?". Or in other words: I want to be able to exclude elements (tables in this example) from the project later on without actually removing them from the database. In the scenario presented to the user this looks something like this:

LLBLGen Pro v3 alpha: Context menu (incomplete, not all features have been added yet!)

The above screenshot shows the ability to exclude the selected tables (Customer and Employee in this case) from the project. This means that the objects representing these two tables are removed from the meta-data data-structure of the Project (through a controller which asks the Project to exclude the elements at hand which will delegate the call further) and all mappings to these elements are cleared. This feature works, it removes the tables from the Tables collection in the schema object representing the schema the tables are in and as everything is observer-aware, events are raised which are picked up by mapping objects which clear themselves if they have the elements excluded as their target, all undo-able thanks to Algorithmia.

When I wanted to implement the feature on the Schema node I ran into a slight problem: a Schema isn't a Schema Element like a table, view or stored procedure, it is a container, although it is contained in a Catalog. I had to refactor the code I had to also support these different elements (catalog, schema, database meta container, which all aren't a schema element). This gave me the hint that the whole setup might not be correct and I should simply create an interface like IExcludable or something, implement that on all elements which are excludable and call an Exclude method on that. Sounds logical and feng shui-compliant with Common Sense Software Engineering (CSSE), don't you think?

However this runs into a tiny problem: does every element know its container? Is the container of a table the schema (its logical container) or the Tables collection in the schema object (its physical container). To be able to work with meta-data, it's essential that a table knows its schema. However it doesn't need to know that it's in a Tables collection. A schema knows its parent catalog, but a catalog doesn't know it's parent container as it's not something it should be aware of (it doesn't have a logical container, although it has a physical container in the Project). Is it better to tell the element that it should remove itself from its parent's container, or is it better to tell the parent that a contained element has to be removed? Example: do you tell the schema that it should remove itself (which in turn will force the schema to tell its logical container (catalog) to remove the schema) or do you tell the catalog that a schema has to be removed?

The interface idea sounds great, but requires that elements know their container (also for the catalog). It gives the freedom to implement the logic inside the elements which it is all about instead of in code outside the data-structures. On the other hand, code outside the data-structures and placed in a method in the Project class, which contains the meta-data, which controls the calls to the right parent is also tempting as it also knows the container of the catalog, something the catalog itself doesn't know. Yes, the Catalog owns the Schemas collection, but does it own the Schema elements inside the Schemas collection?

So it comes down to:

Interface route. This route requires little code in Project, as an IExcludable is passed into a method on Project by the controller and the project simply calls the Exclude method on the object and the object has to take care of it being excluded (removed) from its parent. We've to make sure the Catalog knows its container as well which is outside its reach at the moment (other assembly) so logically not the right thing to do. Instead for the catalog, this requires some if/else code to call the container of the catalog as well.
Manager code route. This route uses some switch/case statement in Project and based on the element to exclude, it calls the proper container to exclude the element. This looks straight forward, but places knowledge what the parent of which element is, inside the Project class instead of the element itself. This could be less ideal when for example the parent type changes and you have to hunt for all the references to that parent and change that code everywhere.

One thing to note is that, as you can see in the second screenshot, multiple target database types can be in the same project. Telling a project to exclude a given table object isn't a matter of asking the single container of catalogs to remove the given table object, one first has to find the proper database specific store which contains the catalog with that table. A table itself doesn't have this knowledge of course, its parent schema's parent catalog does, at least the ID of the database.

Still convinced what you picked as the right thing to do in the first paragraph is the right thing, or is it more complicated than what you initially thought? I know this scenario is very specific but that's precisely the point: the question presented is very simple and likely you've made this same decision a lot of times before, as I have too. Yet, the specific scenario, the specific context the decision has to be made in makes things less trivial than it initially looked like.

I don't believe in do this and it will be all right kind of advice without a firm context description, so I'm not going to give you one. Instead, I'm giving you advice on how you yourself might be able to find the right decision in similar and other situations you will run into: make a pro/con list of each alternative, eventually prioritize these pro/con items if you will, and simply look at the lists and make a decision based on them, make the decision which makes rationally the most sense, considering the pro/con lists. That's it. Make a decision, based on rational reasoning and cold hard facts and document it, implement that decision and move on. Don't let your decision be lead by what looks like a turn-key 10-step process to get it right which is applicable to any (and thus also yours) scenario/context. There's no such thing, every situation, every scenario is different, yours is too. Perhaps a guide tells you to componentize everything which will take you 2 weeks to complete and in the end 99% of the components are used by only one other component. Did you gain anything by that? It might be you actually made things more complex than the situation you had before you componentized everything. Who has to deal with that extra complexity? The people who gave you advice in a generic "Use ABC with XYZ and everything shall be great" article, or you?

In the end, what matters is that you a) made a decision and b) you documented the reason why you made that decision and didn't take one of the the alternative(s). If for example, after a year or two it turns out your decision wasn't the best, based on the knowledge you have at that moment, you can always check the design decision documentation you've made of the decision, and conclude you did make the right decision back then, but based on other, perhaps incomplete (compared to the situation after two years) knowledge/information. That's life. As my wise mother always says: "If you'd know everything up front, it's not hard anymore to get rich".

What I decided? For this particular case I chose the interface route with the if/else for Catalog in the Project method. Yes, it's perhaps not that pretty due to the if-statement but the alternative isn't that great either and based on the pro/con items of both alternatives, the interface route seems the best choice. In this context, at this moment, with the information at hand.

Should you do the same thing in the situation where you have to make the same decision? That's not a conclusion you should draw from this post, instead you should take my advice, make the pro/con list and decide for yourself what to do. It might be you make the same decision, it also might be you pick an alternative. That's ok, you have the pro/con list plus the reasoning to prove you made the right decision at that moment. That's what matters.

inc(me.MVP)

FransBouma — Thu, 02 Apr 2009 08:03:03 GMT

Yesterday I received the MVP award for C# again, thanks Microsoft!

The Undo-Redo paradox

FransBouma — Fri, 20 Mar 2009 14:10:31 GMT

In July 2008 I started development on LLBLGen Pro v3's new designer. The first thing I realized was that I needed a good, solid, generic framework to base the new designer on, especially because v3 would introduce a new big feature: model-first entity model development. In short, model-first means that the user starts the designer and can build an entity model from scratch (so no meta-data available whatsoever) and create meta-data and mappings from that entity model, or modify an existing or reverse engineered model by adding new elements. So the user will edit, delete, and do other things which aren't based on any meta-data, but based on theory, thought processes and perhaps trial/error. In short: the user will make changes to a live model in memory and will try to undo and redo these changes during the process. Everywhere. Always. So undo/redo has to be present everywhere, and always in every situation. Removing an element, like an entity definition, should remove all its related and depending elements or at least make them update themselves and undo-ing that removal should restore the original state.

The framework I had in mind would need to be able to undo any edit action, any change. I also needed a new set of data-structures to store the entity model in. In v2.x of LLBLGen Pro, the entity model is stored in an 'enclosed way': if an entity E has a relationship with entity F, it has a relationship object R which is stored inside E (if F also has a relationship with E, there's a relationship object representing that relationship and which is stored in F). While this might be a natural way of storing object graphs (the graph edges are the references between the vertices), it leads to a problem: you can't reason over the entire model, as it always requires traversal of the object tree in a way where you need to dig through an object (e.g. the EntityDefinition instance which contains the instance of the entity relationship) to get to other elements. A graph object with vertices and edges (so the entities would be the vertices and the relationships would be the edges) would be easier to do reasoning over the model.

To be able to undo any change to a model, you need to have some kind of mechanism to perform the change in the first place and then simply revert the action the mechanism performed. This is solved with the Command pattern. In short, it describes a way to perform actions (the 'commands') onto a data-structure or other element and as you have described the action through a command, you can extend it to perform another action when the command action has to be 'undone' or better: rolled back. However, my v2.x code base of LLBLGen Pro doesn't use the command pattern to do its actions, as it never needed to: to get things done you call methods on objects which call other methods, set properties etc., the basic OO style of maintaining an object model in memory. Implementing everything through commands now seemed like a lot of work: imagine every property get/set action has to be done through commands so the change is undo-able, every method call made might change internal members and these changes have to be undoable as well.

Algorithmia

I decided to solve this properly and from the ground up so I started working on a separate project: Algorithmia. Algorithmia started as a .NET 3.5 class library I wrote in my spare time to learn .NET 3.5's new lambda stuff and which contained some well-known algorithms and data-structures which weren't in the .NET 3.5 BCL (or not implemented in a useful manner). So I implemented in-place sort algorithms (so these sort a data-structure in-place, not like Linq's OrderBy() methods which return a new enumerable) as extension methods, a couple of priority queues and heaps like a full Fibonacci Heap. Algorithmia seemed (and still is) perfect to add my general purpose algorithms and data-structures to, and the undo-redo algorithms and related classes are no exception.

After some long, deep thinking I realized I needed two fundamental things to meet LLBLGen Pro v3 requirements: a general purpose undo/redo mechanism and a set of data-structures, like a graph structure which are undo/redo aware. The two separate areas should have 1 thing in common: undo/redo should be transparent to the user (the developer). With transparent I mean:

someObject.Name = "Some String";

where the property set action in the statement above should be undoable (and redo-able). The traditional command pattern approach would have forced one to write such a simple statement with a command, so the action (setting the property) would be undoable by setting it again to the original value. I wanted it solved differently so I didn't need to write command calls everywhere and I also could leverage databinding for example or events and other things build into the .NET framework.

Commands, Command queues and their overlord manager.

To understand what undo/redo really means and how complex it can get, let's look at an example. Say I have a graph with two entity definitions: Customer and Order, and a one to many relationship R between them. Furthermore I map a foreign key field onto R in Order (so it points to Customer's identifying fields, which happens to be CustomerID). All nice and dandy. I feel a bit bold today and I select Customer and hit the DEL key. Obviously, the Customer entity definition is deleted from the model. But that's not enough. To remove it from the model, I have to remove it from the graph and because I do that, I have a dangling relationship (R) which has to be removed from the graph as well. If R is removed, the foreign key field in Order also has to be removed as it's based on R. Pressing DEL sounds rather complex all of a sudden.

The traditional command pattern approach suggests that you issue the action to remove Customer from the entity model graph through a command however that immediately gives a problem: what has to happen to the actions which follow immediately after the removal, like the removal of R and the foreign key field? Do we have to add these commands to the command which did the removal of Customer from the graph or not? If we don't, undo-ing the removal of Customer doesn't automatically undo the follow up actions as well, as these seem to be unrelated. But if we do add these commands to the initial command, it will create a complex piece of code which is also immediately unmaintainable as it has to know about all things which could happen after we've removed Customer from the graph.

The Command
To undo an action, you can take several approaches. For example, you could use the transactional approach where you make changes to a temporary space and finalize it when you commit the transaction. Another approach is to read the initial state right before the Do action is performed and Undo simply restores that state. I've taken the second approach as it is more flexible: there's no transaction to commit: a change is a change and it's final however, it's always undoable. What makes this easy is the introduction of lambda's in .NET 3.5. Do and Undo are simply lambda's. The command has support for lambdas which read the state before the Do lambda is called and the Undo action is simply passing in the original state into the Undo lambda and the action is undone. There are various bells and whistles added to that of course, but that's the basic idea.

The Command Queue
To be able to manage when to undo what, commands are placed in stack-like data-structures: the last command placed into the data-structure is the first to undo. However, that's a 1-dimensional data-structure. In my example above, undo-ing the removal of Customer requires the undo-ing of the removal of R and the removal of the foreign key field. So I created a Command Queue. A Command Queue is internally a Linked List (with a more flexible implementation than BCL's as concatenating these Linked Lists takes O(1) as it should instead of BCL's LinkedList class) with a simple pointer where the last command is. Commands are placed in a Command Queue, one after the other. This gives the flexibility of undo-ing and redo-ing them by simply moving a pointer along the Linked List inside the Command Queue.

To be able to undo a command which spawned other commands, I placed a Command Queue inside every Command. This gives the advantage that when I undo a command, it first calls Undo on all commands in its own queue and then it performs the Undo of itself. Undo-ing a command inside its queue could mean that that command also will perform an Undo action on several other commands first. And here we have our multi-dimensional structure we needed for the situation of our example. However it of course gives another problem: how do we get all these commands neatly nested into each other without any hassle?

The Command Queue manager
I created a thread-safe singleton class which manages the command queues, the CommandQueueManager. This manager is fairly straight forward and it's the interface for the developer to undo/redo anything, to enqueue and execute commands and to keep everything working in the right order. There are some static helper methods on the Command class to easily enqueue itself, but in general the manager is the one to talk to (ain't that always the case? )

The bare-bones mechanism comes down to this: it has an active stack of Command queues and a command which comes in to be executed is simply placed in the command queue at the top of the stack. If a command is executed its queue is placed on top of this stack and every command that gets created while this command is executed is thus placed inside the queue of the command which originated it. When the command is done with its Do method, its queue is popped and the previous queue is now at the top of the stack, which can be the queue of the previous command or the main queue of the manager.

Scopes and threads
Singletons have the side-effect that there's just one instance at runtime, which is nice because that's the reason they're there. The downside is of course that multi-threaded applications have to deal with a shared resource and that's always a sign trouble is ahead if you're not careful. The manager is thread-safe, which means only one thread can queue and work with commands at any given time. Per thread there's also one stack, so different threads can't add commands to each other's command queues. In a way, per thread there's a unique scope. Such a scope consists of a command queue stack. It might be handy in some cases in single-threaded approaches as well: what if you want to create a boundary in which a user can undo/redo actions but when the user closes the form for example the actions are final? That requires a unique scope for that edit form. The Command Queue manager can deal with that, you simply ask for a scope with a new ID and you get it. If you ask for the scope of an ID which was already known, the scope of that ID becomes active.

Back to our example...
So let's go back to our Customer, Order, R and the foreign key field. The user selected Customer and pressed DEL. The UI controller calls into the main system and asks the Project to remove entity Customer. The Project then starts working, but what exactly does it have to do? Remember that I needed a graph to be undo/redo aware. The Entity Model is implemented using Algorithmia's graph class where entity definitions are vertices and relationships are edges (non-directed edges). The graph is undo-redo aware, it manages itself through commands. So removing the Customer entity definition from the Project is as simple as telling the graph to get rid of the Customer instance it has inside itself as a vertex. The UI controller called the single Project method through a command. That command's Command Queue was placed onto the stack of the current scope and its Do lambda was executed. All commands added to the Command Queue Manager will end up in this queue or in a nested queue, so undo-ing the removal will undo all these commands as well.

The graph removes the vertex Customer from itself through a command, which is placed inside the UI call command's queue. The graph notices a dangling edge, R. It removes it too, also through a command, and this command is also placed in that same queue. And now things start to get interesting: when R was removed from the graph, the graph called a method on the edge which raised the edge's event ElementRemoved. Is anyone listing to that? Yes, the object which is used inside the foreign key field inside Order. As the relationship has been removed (as been told through the event), the foreign key field has no purpose anymore, and has to remove itself as well. As it is placed inside a command-aware list, called CommandifiedList<T>, it simply removes itself from its container though that container does the actual removal through a command. That command ends up in... the queue of the removal of R, as that was the active command in progress and that command's queue is on the stack.

So after all this, we have a nested set of commands which we can undo, in the right order, and also which we can redo, in the right order, without the complexity of requiring command creation everywhere, being aware of which command is spawned from where... none of that at all: it's straight-forward .NET code like you and I are used to write.

Undo-ing this Customer removal starts by calling the Undo method on that command. As that command contains a queue with two commands (removal of Customer from the graph and removal of R from the graph). It starts with the last command, which is the removal of R and calls Undo on that. The removal of R command has also commands in its queue, namely the removal of the foreign key field, and starts undo-ing that command first. This makes sure everything is played back in the right order.

But what about that simple property setter example we started with? Let's look at the logic behind that simple statement and how things are made transparently undoable.

The little worker class under the hood: CommandifiedMember

The following code snippet shows a simple test class used in some unit-tests for the command functionality:

public class HelperClass
{
    private enum HelperChangeType
    {
        Name
    }

    private readonly CommandifiedMember<string, HelperChangeType> _name;

    public HelperClass()
    {
        // create a new commandifiedmember instance and set the default value to empty string
        _name = new CommandifiedMember<string, HelperChangeType>("Name", 
                      HelperChangeType.Name, string.Empty);
    }

    public string Name
    {
        get { return _name.MemberValue; }
        set { _name.MemberValue = value; }
    }
}

To combine a lot of functionality around a single member which was needed in a lot of cases I created a class called CommandifiedMember. CommandifiedMember does a lot of things: it sets the value of the member using commands, so setting the value is undo-able. It checks whether the value to set is equal to the current value of the member, so it doesn't issue unnecessary commands. It raises events when the value changes so observers can subscribe on these changes and act accordingly. It has awareness of interfaces which might be implemented on values set as the member value. This is important in the case of the foreign key field of our example: if the identifying field's type changes, the foreign key field's type also changes. To be aware of that, it needs a signal from the identifying field it relates to. Simply changing the identifying field's type will raise an event which will end up in the foreign key field's member which notices this as it automatically subscribed to the event as it recognized it. The member then simply raises an event so the foreign key field notices this and can act accordingly. Similar to removing the relationship R for example: R is removed, so it raises an event that it's been removed. Observers, like CommandifiedMember instances which refer to it, can now act accordingly and set themselves to null or raise an event for example.

The code snippet above doesn't show it, but there's more built in: it is also IDataErrorInfo aware. This is done through an object which is pluggable into a CommandifiedMember and which is also part of Algorithmia, called ErrorContainer. CommandifiedMember is aware of validation and calls a virtual method before it continues to call the Do action. It takes care of logging the error in the ErrorContainer and if a correct value is accepted, it clears the error accordingly. The code snippet above also shows the usage of an enum which is used for the change-type specification. This is useful if you want to use undo/redo to its full potential and implement a lot of logic through events using the Observer pattern: HelperClass could sport an ElementChanged event which propagated the HelperChangeType to its subscribers, which could then easily determine what exactly changed in HelperClass without the necessity for a lot of events and also avoiding string-based approaches like INotifyPropertyChanged.

So with the CommandifiedMember in place, I can create the following, undo/redo aware code:

HelperClass h = new HelperClass();
h.Name = "Foo";

By setting the property, I indirectly create a command which sets the actual member, compile time checked. I can undo this action by simply asking the Command Queue Manager to undo the last command. However, I'm not even aware that setting the property is an undo/redo aware affair nor do I care. I simply write code like I used to do, without the hassle of creating commands to make sure things are undoable later on: missing one spot makes some things suddenly not undoable, with the CommandifiedMember, that's not possible. As it's transparent, I can bind the Name property of an instance of HelperClass to a control and have undo/redo awareness without even writing any code: if the control sets the value of Name, it will be undoable. Of course, to make the control become aware of the fact that Name has been rolled back, I have to implement INotifyPropertyChanged on HelperClass, but that's pretty easy to do: I get an event when _name changes so I can anticipate on the change by a simple event handler:

public class BindableHelperClass : INotifyPropertyChanged
{
    public event PropertyChangedEventHandler PropertyChanged;

    private enum HelperChangeType
    {
        Name
    }

    private readonly CommandifiedMember<string, HelperChangeType> _name;

    public BindableHelperClass()
    {
        _name = new CommandifiedMember<string, HelperChangeType>("Name", HelperChangeType.Name, string.Empty);
        _name.ValueChanged += new EventHandler<MemberChangedEventArgs<HelperChangeType, string>>(_name_ValueChanged);
    }

    private void OnPropertyChanged(string propertyName)
    {
        if(this.PropertyChanged!=null)
        {
            this.PropertyChanged(this, new PropertyChangedEventArgs(propertyName));
        }
    }

    private void _name_ValueChanged(object sender, 
            MemberChangedEventArgs<HelperChangeType, string> e)
    {
        switch(e.TypeOfChange)
        {
            case HelperChangeType.Name:
                OnPropertyChanged("Name");
                break;
        }
    }

    public string Name
    {
        get { return _name.MemberValue; }
        set { _name.MemberValue = value; }
    }
}

I introduced a switch for checking on the change type, which is a little overkill as there's just 1 member, but you get the idea. It's not really more code than one would write in the case of a simple normal class, however you get value checking, event raising, undo/redo etc. all for free. Binding the Name property of an instance of this class to a control, say a TextBox, will make it possible to edit this instance with undo/redo awareness.

So where does this 'Paradox' of the title come into play exactly? Well I think you now know enough information to understand the following example of it.

The Undo/Redo 'Paradox'

The Undo/Redo 'paradox' as I dubbed it (probably a bad name, so forgive me), is the contradiction between what the user thinks what's being undone and what the system thinks the user means that should be undone. I've specified 'paradox' in quotes as sometimes people call things a paradox while they clearly aren't a paradox and I'm not yet sure if this is a true paradox, though I have a feeling it unfortunately is.

I've created a real-life example of the paradox in the following screenshot. It's a screenshot of a part of the LLBLGen Pro v3 GUI (where I moved everything close together so it fits in a tiny area):

There's a lot of info in this tiny screenshot and I'll describe briefly what's important to understand the problem. The project shown is a dummy test project with a couple of random entities. At the left you'll see the Project Explorer which shows the groups, the entities, the value types and the typed lists (some elements are still not there, in the case you're missing something. It's not done yet ). At the right of the Project Explorer you see the editor for the Customer entity which is a subtype of Person, and below it a debug panel for the command queue manager where I can see which commands are in the queue and inside which other commands they're stored. As you can see, after I've loaded the project, I created a typed list called Test which spawned one command, the addition of adding a new item to a CommandifiedList. The arrow suggests it's the current command, so pressing cntrl-Z or clicking Undo in the toolbar will undo that command.

So, what's the problem? Well, it's at the top: I typed a space in the entity name and tabbed away from the textbox. The validator plugged into the CommandifiedMember kicked in and denied the value and reported an error: names can't have spaces. So the cursor stays in that textbox.

What will happen if I press cntrl-Z or click Undo? Will that undo the change I made inside the textbox by undoing the insertion of the space, or will it undo the last command it knows, creating the typed list?

The 'paradox' is that the system isn't aware of any command setting the Entity Name to an invalid value (as that would make the project become erroneous: what if I entered a name which is already taken?) however the user is. The textbox has a cntrl-Z mechanism, where pressing cntrl-Z will undo the changes in the textbox, which in the case above would remove the inserted space and everything would be normal. However, what does the user mean: local undo or global undo when issuing the undo command and when are local undo's all of a sudden global undo's?

In general: there's a global undo/redo system with a global access mechanism (cntrl-Z/cntrl-Y) and there are two different scopes in play: the local editor scope and the global model scope: issuing an Undo action raises the question: do you want to undo a local action which might not be propagated to the global model scope (e.g. the change hasn't been processed yet) or do you want to undo the last change at the global model scope level? This isn't an easy question to answer, as I hope to illustrate in the explanations below.

A perhaps more well-known example of this problem is the issue you run into with the Windows Forms designer and after that when you change code in the form class: after you've made some changes to a form in design view, you switch to the class and add some code, like a member declaration. Then press cntrl-Z a couple of times till you've undone all your changes to the code and you'll likely see a message box pop up which tells you that you can undo one last thing which can't be redone. Why is that?

It's the same issue: suddenly the local scope you were working in (the code editor) has no more commands to undo and pressing cntrl-Z again then raises the question: does the user want to undo more commands in the editor (though there aren't any left) or does the user want to undo things on a model /global scale, like the changes made to the design view of the form? That's unclear and can't be solved by the undo/redo system by itself: perhaps the user simply only wants to undo/redo the changes in the editor (like the textbox or code editor) and stop undo-ing commands if there aren't any left, in that scope. However, perhaps the user wanted to undo things on a global scale after all commands in the local scope are undone and to do that the user has to leave the editor to signal that the undo action is not for a local scope. This is of course confusing and unclear for a user as the user isn't aware of the length of command queues or even local / global scopes.

In the specific situation of the screenshot above, there are a couple of obvious things which one might want to try to solve this paradox, like disabling the global undo/redo mechanism when an error occurs, however that doesn't solve the situation where I don't create an error but simply append a couple of characters to the name and then press cntrl-Z. One could think of introducing a scope used only for the textbox, but it then gets tricky to get rid of that scope once the value is indeed valid as that action has to be in the global scope to be able to be undone on a global scale (so I don't have to go back to the textbox to undo the name change). Another solution might be to store the invalid value in the model and simply use the mechanism available so pressing cntrl-Z will undo the change which caused the error. The downside is that if the user presses cntrl-S after the change, the erroneous value is saved which could cause a problem, for example if the file format is in an XML format and elements are referenced by name, so what happens if I specify a name which is already in use, which is an error trapped by the validator, however I still save the project?

I can't find a simple solution for this 'paradox', and I fear there isn't one either, but perhaps some solution pops up soon.

LLBLGen Pro v3.0 is slated for release later this summer/autumn, with support for LLBLGen Pro Runtime Framework, Entity Framework, NHibernate and Linq to Sql, and Algorithmia is shipped with LLBLGen Pro v3, very likely in sourcecode form and a flexible license so you can use it in your own applications as well.

[Dutch] Devnology Code Fest

FransBouma — Tue, 17 Mar 2009 12:47:54 GMT

(Sorry English speaking visitor, this post is in Dutch, as it's about a Dutch user group meeting)

In Nederland hebben we een aantal gebruikersgroepen die op gezette tijden meetings organiseren voor developers. Aan dit aantal is een nieuwe toegevoegd, Devnology (http://www.devnology.nl). Devnology is niet zozeer gericht op het houden van meetings waarbij 1 persoon een praatje houdt en de rest poogt niet in slaap te sukkelen, het is meer gericht op discussie en interactie tussen developers, samen bezig zijn met code, software engineering en andere aan ons vak gerelateerde zaken. Ook is Devnology niet gelimiteerd tot .NET alleen maar zijn andere talen en platforms even welkom. Het gaat tenslotte om software engineering en niet om de laatste truuks voor een random MS product.

Op 1 april houdt Devnology haar eerste meeting, een Code Fest: je krijgt vooraf een opdracht en wie de beste implementatie maakt wint. De opdracht deze keer is: programmeer de Game of Life, een bekend concept, en de keuze van platform en taal is vrij. Deze opzet komt me bekend voor van vroeger uit de demoscene, waar op demo-parties ook dit soort compo's gehouden werden. Het leuke aan dit soort dingen is dat enerzijds de competitie je toch dwingt je best te doen en anderzijds de discussies met mede-developers ter plaatse altijd wel wat nieuwe kennis en info opleveren waar je wat aan hebt.

De opdracht van Game of Life is op het eerste gezicht wellicht wat een dood spoor maar je kunt op zoveel manieren dit probleem aanpakken dat het nadenken daarover, het uitpuzzelen welke benadering juist die originele oplossing oplevert, juist je tot ideeen kan brengen waar je als software engineer baat bij hebt, in je vak en dus in je dagelijks werk.

Devnology gaat in juni ook een Open Spaces meeting houden en daar kijk ik nu al naar uit.

Ik ben van de partij op 1 april. Zie jullie daar!

Choose .Concat() over .Union() if possible

FransBouma — Wed, 04 Mar 2009 10:25:28 GMT

Update: I've reworded a sentence as it was too vague. Sorry for that.

Here's a simple performance tip which can benefit you without doing any effort. Linq to Objects has two methods to combine two sequences together, both with different characteristics: Union() and Concat(). The difference in characteristics makes it possible to gain performance without doing anything difficult. Let's look at a simple example first:

Say we have two lists of integers: A: {1, 2, 3, 4} and B: {1, 2, 5, 6}. When using A.Union(B), a set union is executed, which results in { 1, 2, 3, 4, 5, 6}. When A.Concat(B) is used, the sequences are simply concatenated and { 1, 2, 3, 4, 1, 2, 5, 6} is the result. Pretty straight forward stuff. If you do not want duplicates in the second sequence to appear in the resulting sequence, Union() is necessary. However, in the case where it's impossible to have duplicates in the second sequence or you don't care if duplicates in the second sequence appear in the resulting sequence, Concat() is a better choice.

It seems obvious that Union() is more performance intensive than Concat(): Contact() simply makes sure the enumerator returned enumerates over the two sequences, Union() filters out duplicates in the second sequence. If your sequences have a lot of elements, using Union() will make the operation become significantly slower.

In the past 8 months I've written a lot of Linq to Objects queries and today I saw:

/// <summary>
/// Gets the entity mapping targets in this meta-data store
/// </summary>
/// <returns>all tables/views, ordered by catalogname/schemaname/tablename unioned with 
/// all views ordered by catalogname/schemaname/viewname</returns>
internal IEnumerable<IEntityMapTargetElement> GetEntityMappingTargets()
{
    return from c in this.PopulatedCatalogs
           from s in c.Schemas
           from e in s.Tables.Cast<IEntityMapTargetElement>()
                     .Union(s.Views.Cast<IEntityMapTargetElement>())
           orderby c.CatalogName ascending, s.SchemaOwner ascending, e.Name ascending
           select e;
}

It turned out I happened to have used Union() in many cases in the code where two sequences had to be merged into one sequence, however it was impossible to have duplicates in the second sequences in these queries. Must be an old strain of SQL-itis, I think: "Oh I have two sets to combine to one set: UNION". However, in the query above, it's not possible to have duplicates in the second sequence: there aren't views in the set of Tables and vice versa. So this same query could be written with a Concat(), saving performance as the second set doesn't have to be filtered from duplicates.

If you too have the habit to use .Union() to combine sequences, pay attention to that second sequence: if it can't have duplicates (make sure it also doesn't contain duplicates in the future!), it's better to use Concat() instead of Union().