Gáspár Nagy's Blog

SpecFlow 1.8

2011-10-26T14:41:00.001+02:00

We have released SpecFlow v1.8 recently (just published 1.8.1 with a few fixes) and I would like to take the opportunity to give a subjective summary of the new features and also some outlook for the upcoming releases.

Run & debug scenarios from context menu in VS2010

Maybe the most interesting (most wanted) feature is to make scenario execution more convenient. Now it is here: in the context menu of the editor, the project, folder or file nodes of the solution explorer, you will find new commands to run or debug the scenarios of the current context: the scenario of the cursor position (or all from the file if you stand in the header or between scenarios) or the scenarios in the files matching to the selected solution explorer node.

Since SpecFlow does not run the scenarios directly, but you run them with a unit test runner, this feature is a little bit tricky and needs a short explanation.

When you select the “Run SpecFlow Scenarios” command, SpecFlow tries to run the selected scenarios with your favorite unit test runner. You can either tell SpecFlow which one is your favorite or let it find out automatically based on some built-in ranking. This can be done in the SpecFlow section of the Visual Studio Tools/Options dialog.

From the picture you can also see the list of supported unit test runners (ReSharper (5 and 6), MsTest and SpecRun). The TDD.NET is in the list, but the gateway is not yet implemented. Contributions for the TDD.NET gateway or any other that is not listed yet is welcome.

Cucumber harmonization

As you’ve probably heard, Richard Lawrence has announced that the Cuke4Nuke project has been retired and we decided to merge our efforts to further improve SpecFlow.

We have been working for a while with Richard, Paul, Christian, Jonas and Darren for the preparation of this and we have started to implement features to be better compatible with Cucumber.

Currently we are at half-way. We have consolidated the terms (e.g. using “hook”) and validated/fixed some obvious compatibility problem. The remaining stuff is mainly concerning the table manipulation API, that is currently covered by the Assist namespace in SpecFlow. Regarding this, we would like to

make the table API extensions better discoverable by merging them to the main “TechTalk.SpecFlow” namespace
consolidate the different CompareToX methods into one
improve the output of the table comparison methods

Improvements in the SpecFlow architecture

SpecFlow has an evolving architecture so there are always improvements here as the features grow. 1.8 is maybe sill an interesting milestone, because we have fully refactored the class-dependency model of the runtime code and have replaced the custom object factory solution with a DI tool.

This step was important, because this consolidated and solved issues with the context injection feature (e.g. disposing injected objects) and also because this has opened up new ways for customizing SpecFlow. Here is an example for the customization:

Let’s say you want to implement a custom way of reporting SpecFlow user errors (e.g. for doing something special for pending steps). For this you need to create a class that implements the IErrorProvider interface. You can let SpecFlow use your custom provider by changing the default dependencies in the configuration like this:

Also the context injection feature can resolve the SpecFlow classes as well, so for example if you need to apply the same conversion logic as SpecFlow uses in your code (including step argument conversions), you can just let an IStepArgumentTypeConverter instance be injected to your step definition class like this:

SpecFlow tests for SpecFlow

Also a new improvement in the SpecFlow solution that now the specifications and the integration tests have got a new platform. The projects TechTalk.SpecFlow.Specs and TechTalk.SpecFlow.IntegrationTests contain SpecFlow tests for SpecFlow itself. The existing non-unit tests have been converted to the new model and I have also added a couple of additional tests, but the coverage is not yet close to complete. My hope is that with the established environment, it will be easier for the contributors to also provide SpecFlow end-to-end tests for the new features.

There is a similar work also in Cucumber, the cucumber-features. The goal is to make the necessary step definitions that SpecFlow can also run (and pass) these scenarios. But this is something to do still.

SpecRun

If you have followed the posts and tweets of TechTalk, you might know that we are also working on an integration test runner tool, that provides smarter (better, faster, more integrated) feedback for integration test execution.

The SpecRun beta has been updated to be compatible with SpecFlow 1.8.1. Take a look at the website: http://www.specrun.com or just grab the nuget package into your project:

Install-Package SpecRun -ProjectName MyProject

SpecFlow next steps

This post is really getting too long, so just a few bullet point for the future ideas:

continue with Cucumber harmonization
improve VS2010 step analysis (performance, stability)
improve step definition skeleton generation (based on the pull request of pmchugh)
.NET 4.0 specific enhancements
simplify customizations with plugin infrastructure

Download it now!

Don’t forget, that the VS integration is independent from the project dependencies, so you don’t have to wait until all your projects can upgrade to 1.8. Go ahead and download the installer for 1.8.

Please also take a look at the documentation. It’s getting better and better, but if you see any glitches in it: it’s a wiki, so just fix it!

Have fun!

SpecRun – because integration tests are not unit tests

2011-09-02T16:05:00.001+02:00

We have started the SpecFlow project in 2009 to provide a tool for .NET developers to perform BDD-style functional testing. SpecFlow uses unit testing frameworks, like NUnit, MsTest or MbUnit to execute the Gherkin scenarios. While this was a pragmatic choice, it has turned out that for executing functional integration tests, these unit test frameworks are not optimal (quite understandable, as they are unit testing frameworks and not integration testing frameworks). While they can execute the tests very well, they lack features like

(html) report as primary output
fast integration test execution by parallelization
advanced execution metrics: execution time, memory usage, memory leaking, benchmarking
detection of “random failures”
test variations (e.g. running the same suite for IE and Firefox)
support for applying different test configurations
provide fast feedback about failures (prioritize failing tests first, stop after a number of failures, etc.)

In addition to that, we were missing a test runner that treats Gherkin features, scenarios and scenario outlines as first-class citizens.

To address these problems, we have decided to launch a new product that fits into the SpecFlow-SpecLog context. It has been publicly announced today, so it’s here: SpecRun

We have been working for this tool for a while now and have tested it in our projects. It is really promising. It seems that with parallel execution, the local test execution time (that is more critical from the cost perspective, as developers can’t progress while the tests are running) can be dropped to ca. 60%, that is almost half time. This and the smarter feedback about the errors were received very positively by the developers on the selected projects. They were more willing to run the tests locally reducing the long round-trips and they were able to detect the root cause of the errors better.

SpecRun is also using SpecRun for testing itself obviously, and our own dog food tastes pretty good!

SpecRun will be a commercial product (with free versions for OSS projects of course). This was a hard decision. We have a wonderful community around SpecFlow and the support of TechTalk, but even this way we could not afford the costs of building up so many new things. I think this is a good combination. While we believe that SpecRun is a great help and provides quick ROI for the users, it is not the main concern of SpecFlow. With the separation of the two aspects, we can have still our SpecFlow efforts focused on the main concerns (editor, runtime API, documentation) and also provide an extension for the smarter execution experience.

SpecRun is currently in a closed beta stage, but we are planning to have the commercial release soon. Have a look at our website: http://www.specrun.com, stay tuned on our twitter account: @specrun and contact info@specrun.com if you are interested in participating in the beta program.

Enhanced upgrading options in SpecFlow v1.7

2011-08-01T10:42:00.001+02:00

We have just released SpecFlow v1.7 over the weekend. In this release there are extended integration options for the Silverlight async testing framework and also for SharpDevelop 4.

In addition to that, we have tried to make the upgrade process easier for the SpecFlow users, especially if they work in bigger teams or on different projects in parallel.

The decision to upgrade a tool for a project is usually made carefully. You have to consider your close deadlines, check the breaking changes, and discuss it with the team.

If you work on multiple projects in parallel (e.g. you have to support and older project too), it can be quite hard to synchronize that all projects upgrade at the same time.

As SpecFlow is (was) a single msi installer and the different versions cannot be easily installed side-by-side, the upgrade process was sometimes challenging.

From v1.7 we have managed to have better separation of the components that are directly used by the project (the generator and the runtime) and the IDE integration. With this, you can upgrade SpecFlow on your machine and defer the upgrade for the individual projects.

For that to work, please ensure that your project is using SpecFlow 1.6 or later and it is setup as one of the following (see further advanced options in the documentation):

SpecFlow is configured in the project as a NuGet package (this is the easiest and the recommended option).
SpecFlow assemblies (not only TechTalk.SpecFlow.dll, but (almost) all the assemblies from the SpecFlow installation folder) are copied into your project folder structure and the reference for the TechTalk.SpecFlow.dll is taken from this folder. See the detailed list of required assemblies in this documentation page (last section).

If you have this setup, you can safely install a newer SpecFlow version on your machine. The projects will still use the configured generator and runtime.

When you feel confortable to upgrade SpecFlow on your project as well, you can do that project-by-project (the easiest it to use NuGet upgrade).

Note 1: Please make sure that the SpecFlow version on the machine is the same or newer than the SpecFlow version used by the projects.

Note 2: This feature works currently with Visual Studio 2010 only.

SpecFlow is and open-source project sponsored by TechTalk. You can also support the project by contributing on github or by “adopting a feature” ;-). TechTalk also provides commercial support for SpecFlow.

Learn SpecFlow at the the Progressive .NET Tutorials: http://skillsmatter.com/event/open-source-dot-net/progressive-dot-net-tutorials-2011

Tracing Web Automation Test Errors with SpecFlow

2011-07-18T09:23:00.001+02:00

If you try to run your automated UI tests on the build server, you can quickly run into problems that are quite hard to analyze.

The most common scenario is that one of your assertion like “Then my name should be displayed” fails, because the web driver does not find your name in the resulting page. It works locally, but what’s the problem on the build server? Permission problems? Timing issues? Browser compatibility? Different locale? Web compilation or configuration errors? All of these are potential candidates, but it is hard to find out quickly which one.

We are using the following SpecFlow event binding to trace the content of the current page in case of an error. In many cases from the page content it’s easy to say what the problem was.

[AfterScenario]
public void AfterWebScenario()
{
    if (ScenarioContext.Current.TestError != null)
    {
        Console.WriteLine("Page src for failing test: {0}",
            BrowserContext.SeleniumWebDriver.PageSource);
    }

    //... (if you stop your browser session after
    //     each scenario, this comes here probably, 
    //     to ensure that the tracing runs before)
}

Git & Dropbox for preparing live demos

2011-06-14T11:51:00.001+02:00

I have arrived from Oslo where we did two sessions with Jonas Bandi on NDC 2011 about SpecFlow.

As Jonas lives in Switzerland and I live in Hungary, we have used Dropbox to synchronize the content we are working on. This was fine for slides and other notes, but for the code demo we did not have the right toolset so far.

Actually I like to do demos where I modify and evolve the code during the session. But if you do so, you need to maintain different versions of the code base. You need a baseline to start the demo from and probably also a “done” version if something goes wrong. Additionally you need to be able to revert back to the baseline while practicing the demo. This begs for a SCC and Git is a perfect choice.

To have a little bit of “surprise” effect, I don’t like to publish my demos upfront so did not wanted to put it into a github repository. To be able to share the code with Jonas, this time we’ve tried to combine Git with Dropbox like this:

create a “bare” git repository into a dropbox folder
clone it to your working folder
work on the code & commit changes
push the changes to the dropbox folder (the origin)

Though I’m pretty sure that this is not a well scaling solution for real development, but for two persons it was just fine & simple.

Agile Developer Course in Vienna

2011-05-31T17:06:00.001+02:00

In summer everyone is on holidays, the projects are running on low frequency, so maybe it’s a good time to meet others and improve your agile development skills.

July 6-8, 2011 there will be a three-day course in Vienna instructed by me (in English) that focuses on agile engineering practices and team collaboration. The attendees can also practice behavior driven development (with SpecFlow) and TDD. This is all done through a lot of hands-on exercises and group discussions.

There are two days left for the discounted price!

Details & registration: http://www.techtalk.at/scrum-trainings/certified-scrum-developer.html

(Successful completion of the course can be credited towards the Certified Scrum Developer (CSD) certification of Scrum Alliance.)

Testing manually vs. SpecFlow

2011-05-06T16:12:00.001+02:00

No, I cannot answer this general question. Just had a nice story in our team and since I’m sure that Mátyás and Viktor will never blog this, I do it now on behalf of them.

We are working on a small project, where we have to implement a smaller module for an existing system. Interesting project, but not BDD at all. Especially there are no automated tests.

Since the project is really small and it has to fit into this bigger existing system, we decided not to introduce SpecFlow there (was a hard decision, you can imagine :).

We are near to finish the first sprint (well, we have introduced Scrum and user stories to the project still), so we started to think about the demo and that we should some manual testing. But how?

The majority of Sprint 1 was a background job that had to process some data depending on some complex rules. So we sat down after the daily standup yesterday and brainstormed the different cases we want to test. “Surprisingly” the cases were pretty much look like acceptance criteria illustrated with examples. As there were too many cases (11) for manual testing, we have selected the 5 most important to be tested.

We had two choices: either we try to find matching data in our test database and test the cases with them or we create test data for our own (with SQL scripts?).

Finally we have chosen the third option: decided to create our own test data, but through Gherkin & SpecFlow. It was a risky decision (budget, deadline, etc.), so we decided for a 1 dev, 1 day time-boxed spike to do it. This was at 9:30am.

To make the story short, at 17:11, the Gherkin files were created and bound for all 11 tests… and they were green (like Cucumber).

And the funniest thing is, that we’ve found a bug… a very tricky one that would not have shown up in the manual tests most probably!

This was very convincing! To be honest, I did not expect to be finished in one day. I was asking Mátyás today morning in the standup carefully, what happened. He said: “Yeah, I’m sitting next to Viktor and doing some pair programming”. I was asking back (trying to hold back my excitement): “And what happened with the SpecFlow tests?” He said very calmly (he is a cool guy): “Ah, yeah the SpecFlow tests are done already. They are all green”.

Not even mentioning the extra benefit, that we can show the automated tests to the customer so they can also create new test variants quickly.

So, thanks for the good job, guys!

P.S.: Just to improve your C# skills, I also share buggy code (without comment):

a ?? 0 + b ?? 0

Re: Rapid Feedback

2011-04-11T10:10:00.001+02:00

I’ve read yesterday evening the post of Attila Domokos about rapid feedback that made me thinking. Here is my reaction that didn’t fit to a regular blog post comment.

I think no company should afford wasted time and having a development environment that is determined to waste time is a usually forgotten, but important factor in this. The scenario that has been described by Attila is known to all of us. Well known. Though the story can end many different ways. For Attila, Rails was the “solution”. For me it is BDD and good practices. (I’m not against Rails at all. Rails is fine, but these problems have to be solved within .NET for most of us.) I’m absolutely agreeing with the idea however, to use the right tool (framework) for the right job, but it would be very sad if it would turn out that WPF development cannot be efficient. (I’m not sure though… I know another Attila who would immediately tell me that it cannot, but I’m still hoping.)

[BTW: I’m pretty sure that there are a lot of devs (not the ones working with me fortunately), who actually love this situation and use the compilation times to read the daily newspaper.]

As a developer and developer coach, I always try to ensure that the team is taking the time (early enough) to setup a development environment where you can get rapid feedback. For a WPF-frontend .NET app, here are my suggestions (I’m not saying that we can always do all unfortunately):

Do BDD/ATDD in a test-first, outside-in way, so that usually you don’t have to start up the application for testing the functionality. (We do this with SpecFlow of course. :-)
Consider testing the functionality at the controller level (as well?), in order to get feedback about the implemented functionality without the slow UI automation. (At least until we have a good test framework where the automated UI testing is really easy and fast.)
Do TDD for your inner components. Ensure that the unit testing environment and practices are confortable and helpful. If the tests are executed only after the run-and-test round has been done, you did not win too much.
For WPF, (try to) make sure that your controls can be seen in the VS designer or blend (aka “blendable”), so that you don’t have to start the application for pixel proofing. There are infrastructure for design-time data, but introducing it late is very hard.
Implement shortcuts to your application that the run-and-see test can be also done more efficiently. For example, you can make a stub authentication/authorization provider that bypasses the login screen.
Optimize the build process to be fast - review and remove projects, post-build events, slow signing process, etc. from the build configuration that you use for development. (Be very careful with post-IL tools, like Code Contracts or PostSharp.)
Measure the difference and argue for faster development machines, solid-state drives.

Still, it can happen that you have to wait. Personally, I’m using this waiting time for “development”: Even if I’m not doing TDD, I always follow the practice of “making a small piece work” and then “making it better” (refactoring). So while it is building, starting up, etc. I’m already thinking about the refactoring possibilities. Try it! It’s a little bit exhausting, but fun!

SpecFlow 1.6 and after…

2011-04-05T15:54:00.001+02:00

After a lot of work we have finally released SpecFlow 1.6. This was an important milestone of SpecFlow for several reasons:

Altogether 11 people have contributed (in compare to 1.5, where “only” 6)
Effective usage of the public build server, for the contribution and for preliminary results
NuGet package published together with the release

There are many smaller improvements and fixes, but the most important new features are these (see the announcement and the detailed change log for further details):

Visual Studio 2010 integration improvements (improved intellisense, go to binding, automatic table formatting)
Windows Phone 7 support
Assist improvements (dynamic getter for ScenarioContext,
Table.FillInstance())
Step argument transformation for tables and multi-line text
arguments
Integration with Gherkin parser v2.3.5

I have posted recently about the VS2010 improvements and automatic table formatting. If you experience any problems with this features, you can enable SpecFlow debug tracing or disable the problematic features in Tools / Options / SpecFlow. (The trace is emitted to the output window.)

And what will come after 1.6? We have of course several ideas how to improve SpecFlow. Here is a (probably incomplete) list of directions we want to go:

1. Using the generator from the project’s lib folder

The generator interface have been prepared now for running from a different AppDomain, so we want to change the VS2010 integration in a way, that it detects the generator in the project’s lib folder (that was placed there through NuGet for example) and perform the test generation with that. The benefit: the VS integration would be more independent of the runtime and the tools. You can then decide project-by-project when to upgrade to a newer SpecFlow release. Less problems with breaking changes - more frequent releases.

2. Share VS2010 improvements with MonoDevelop

The core infrastructure we use for intellisense and navigation is Visual Studio independent, so we want to reuse that also for the MonoDevelop integration. For that the implementation has to be restructured a bit and the VS-independent parts have to be extracted. We do this together with Dale Ragan, who is the master of the SpecFlow Mono/MonoDevelop integration.

On the CukeUp! I had a good discussion with Aslak Hellesøy about standardizing the data gathered for the autocomplete (e.g. in a json format). This data could be generated build-time and used for other tools in the Cucumber-family providing rich editing experiences. Interesting, isn’t it? Let’s see.

3. Remove duplicated binding-match logic

Currently the logic that decides whether a scenario step matches for a binding is duplicated in the runtime and in the VS integration. This should be cleaned up.

As a part of this, we will establish a dependency between the generator and the runtime (generator depends on runtime), so the file-based sharing can be eliminated.

4. Better dependency handling

I have started to setup a better DI solution for SpecFlow, for

better support for parallel scenarios (eliminate static stuff)
better extensibility – being able to set/override any dependencies from the config file
better support for unit testing

Currently I’m using a mini DI framework, that can be embedded as a single source file (called MiniDi). This is a huge refactoring, so I try to do it step-by-step.

5. Runtime platform improvements

We do want to further improve the support for alternative platform runtimes, like Silverlight or Windows Phone 7 with the help of the contributors. Currently Matt Ellis is working on the asynchronous Silverlight support.

6. Visual Studio 2010 integration improvements

There are a lot of ideas still to improve the editing and navigation experience, like showing the binding status of a step, commenting out scenario blocks or running the focused scenarios from context menu.

7. Documentation, documentation, documentation

Yes, I know. This is something to be improved. But I hope that by using the github wiki it will be easier to complete the docs and keep it up-to-date.

---

Like before, we are collecting feedback about SpecFlow also through a (new) survey – please fill it in (anonymously or with your name) to help our work.

Once again, thanks to the contributors for their great work in SpecFlow. Also thanks to TechTalk for sponsoring the development and the infrastructure of this open-source project.

---

At TechTalk we are developing another product, SpecLog, that supports managing your requirements in agile projects. Please have a look at our website: http://www.speclog.net/.

Automatic Table Formatting in SpecFlow

2011-04-05T12:51:00.002+02:00

In the new SpecFlow v1.6 release, besides other Visual Studio 2010 integration improvements there is one more that I did not mention in my previous post: the automatic table formatting.

I have spent a lot of time by aligning the table columns in the Gherkin files. This new feature tries to address this problem: whenever you type a pipe (|) character in a table that has at least two rows, it realigns the table columns. Here are the rules for the aligning:

the column widths are calculated based on the cell texts (header and body) without starting/ending whitespaces
the cell values are aligned to the left with a single padding space
there is at least one padding space on the right
the indentation whitespaces of the header row are repeated for the subsequent rows
comment lines between table rows are handled

The formatting is applied only to the table (parts) before the typed pipe character, simply because of technical reasons: it was easier to preserve the current cursor position this way. Though this sounds a bit weird, this should be still natural if you type in the table row-by-row. If you want to reformat an existing table, just simply re-type the last pipe character.

A short video showing this feature

If the automatic formatting is not something you like, you can either press “Undo” after you typed in the pipe, or you can disable this feature from Tools / Options / SpecFlow.

Building Windows Phone 7 Applications on build server without installing the SDK

2011-03-29T15:16:00.001+02:00

I don’t like installing SDKs on build servers, especially ones that are relative new and gets updated frequently. In the new SpecFlow release, we provide support for WP7. In order to build this on our build server, I had to solve this problem. Here are my findings.

I have based my work on the post of Jeff Wilcox. The solution for WP7 is almost the same as he describes, but there is a small trick.

So I have created the following folders in the project’s lib collection:

lib\Silverlight\v4.0\Reference Assemblies – with the content of “C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\Silverlight\v4.0”, except the Profile folder. This can be also reused for normal SL4 projects.
lib\Silverlight for Phone\v7.0\MSBuild – with the content of “C:\Program Files (x86)\MSBuild\Microsoft\Silverlight for Phone\v4.0”.
lib\Silverlight for Phone\v7.0\Libraries – with the content of “C:\Program Files (x86)\Microsoft SDKs\Windows Phone\v7.0\Libraries\Silverlight”.
lib\Silverlight for Phone\v7.0\Reference Assemblies – with the content of “C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\Silverlight\v4.0\Profile\WindowsPhone”.

Unfortunately there is a special step in the WP7 Microsoft.Silverlight.Common.targets file that sets the reference paths without any condition. Because of this, you cannot specify the paths explicitly. I have duplicated this target (GetFrameworkPaths) and decorated it with a condition. This I’ve saved to a separate targets file inside lib\Silverlight for Phone\v7.0\MSBuild called Microsoft.Silverlight.Common.OverridesForBuild.targets. You can download the entire file from here, but the actual change is really just a condition:

<PropertyGroup 
  Condition="'$(UseCustomWindowsPhoneSDKFolder)'!='true'"> 
  <_FullFrameworkReferenceAssemblyPaths>… 
  <TargetFrameworkDirectory>… 
</PropertyGroup>

After this preparation, you can change your project file to use the dependencies from the lib folder. For this you need to open the project file in an XML editor and change the Import sections to the following:

<PropertyGroup>
  <WindowsPhoneBuildResources>..\lib\Silverlight for Phone\v7.0</WindowsPhoneBuildResources>
  <SilverlightBuildResources>..\lib\Silverlight\v4.0</SilverlightBuildResources>
  <TargetFrameworkDirectory>$(WindowsPhoneBuildResources)\Reference Assemblies</TargetFrameworkDirectory>
  <TargetFrameworkSDKDirectory>$(WindowsPhoneBuildResources)\Libraries</TargetFrameworkSDKDirectory>
  <_FullFrameworkReferenceAssemblyPaths>$(SilverlightBuildResources)\Reference Assemblies</_FullFrameworkReferenceAssemblyPaths>
  <UseCustomWindowsPhoneSDKFolder>true</UseCustomWindowsPhoneSDKFolder>
  <SilverlightRuntimeVersion>4.0</SilverlightRuntimeVersion>
</PropertyGroup>
<Import Project="$(WindowsPhoneBuildResources)\MSBuild\Microsoft.Silverlight.$(TargetFrameworkProfile).Overrides.targets" />
<Import Project="$(WindowsPhoneBuildResources)\MSBuild\Microsoft.Silverlight.CSharp.targets" />
<Import Project="$(WindowsPhoneBuildResources)\MSBuild\Microsoft.Silverlight.Common.OverridesForBuild.targets" />

You might need to change the lib folder reference in the first two properties according to your setup.

With these I was able to build our solution on the build server.

Highlights of SpecFlow’s new VS2010 integration features

2011-03-19T08:51:00.001+01:00

The new SpecFlow release will come with a few nice improvements for Visual Studio 2010 integration. I have made a few screenshots to show the most interesting ones.

Intellisense for keywords

Improved intellisense for steps, including binding information and examples from the feature files in the project (you can also see if the step was used in a scenario outline)

Go to binding - We have been waiting for this for a long time…

Re-generating code-behind after changing the configuration

Re-generating code-behind after upgrading the project

SpecLog Comes!

2011-02-02T09:07:00.001+01:00

In November 2009 we have started with the SpecFlow project to support .NET community with a BDD tool that is nicely integrating to the development environment. Through SpecFlow and the projects we delivered, we have learned a lot about agile requirement analysis and BDD.

And now, after 14 months we also have something new to share! At the BDD eXchange, we have already talked about the development process of our new product, SpecLog, and now we are really close to release the first beta. On the right hand side, you can see our iteration burndown from today morning.

SpecLog is a collaborative tool for agile requirement engineering with strong integration to SpecFlow and ALM tools (initially TFS 2010). Stay tuned to http://www.speclog.net, where we will publish detailed information and the beta download this week.

Detailed Report for PartCover in TeamCity 5.1

2010-09-20T11:42:00.011+02:00

I was configuring a build with TeamCity (for SpecFlow that is hosted on github). After the initial success, I thought I would also enable code coverage in the build. I don't really believe in code coverage numbers, but if it is easy (and free) to configure, than why not?

TeamCity supports NCover and PartCover out of the box. As I wanted to go for a free solution (SpecFlow is a free, open-source product), I used PartCover (v4.0.10801). PartCover seems to be a great tool, but it is not so easy to set it up with TeamCity. (You can find a lot of questions and discussions about it.) As I can see, there are some improvements in the later versions of PartCover and TeamCity, but still there are some problems.

If you just start with the build server configuration, you should read first Jeremy Skinner’s post. (On TeamCity 5.1, you don't have to worry with the assembly renaming.) The rest comes here:

Problem: Sometimes you see no results at all

Maybe there are many reasons that can cause this, but for me the problem was an x86/x64 issue. On an x64 build controller, you have to make sure that the test execution and the coverage are all running with the same processor architecture. I've achieved success with x86 compilation only, with x64 I always encountered a "...\tmpE1BE.tmp exited with code -1" error. But I did not play with this so much. These are my working settings:

Problem: The reports provided with PartCover don't work with TeamCity

You can find fixes for the reports in stackoverflow and also on Jeremy's post, but altogether these reports are not too detailed, e.g. they don't go into method level. Without knowing which method causes the low class coverage, the numbers are not too helpful (except for your boss). So this leads me to the third problem...

Problem: There is no detailed HTML report available

As I have mentioned above, this was quite a pain for me. So I refreshed my old XSLT skills (again) and tried to put together a report that shows what I need:

global & assembly-level summary
top 10 uncovered method
detailed method-level report with source code line numbers (it is not possible to embed the source code file from XSLT unfortunately)
coverage values excluding the very simple members (simple property get/set, empty constructor, etc.)

The detailed report (for a small solution):

Class details with method-level numbers:

How to setup the detailed report

download the XSLT from here: PartCoverFullReport.xslt
copy the XSLT file in the "xslt" folder of PartCover on the build controller(s) (e.g.: C:\Program Files\PartCover\PartCover .NET 4.0\xslt)
in the build configuration, specify the "Report XSLT" field as:C:\Program Files\PartCover\PartCover .NET 4.0\xslt\PartCoverFullReport.xslt=>index.html

SpecFlow Workshop at Skills Matter / Progressive.NET Tutorials

2010-05-17T20:41:00.001+02:00

Luckily escaping the BA strike and the newest volcano ash cloud, on Friday I safely arrived home from the Skills Matter Progressive.NET Tutorials in London.

This was the first time I visited this event and it was a great honor that Jonas and me had been invited to do two workshops about behavior-driven development (BDD) and SpecFlow.

I enjoyed the event very much. I also visited a lot of interesting tracks, but maybe the best was to meet so many enthusiastic and “progressive” developers. It helped me a lot to charge my batteries - to keep on thinking about new ideas and improvements in the development process and technologies.

Chris has written a short summary on our sessions, I don’t want to repeat him. I was very pleased that we got so much interest and positive feedback after the sessions. It was a pleasure to work with Jonas and thanks also for TechTalk’s support.

Let me highlight only two things about the BDD session.

The best part in it (again) was the acceptance criteria writing task, where the attendees had to write 2-3 (the most important) acceptance criteria about the shopping cart feature of a book shop application. The goal of the exercise was to discuss good/bad practices and patterns in formulating scenarios. We thought that the best way to learn these was to learn them on the basis of personal experience. But as we can never guarantee that the attendees will really make those mistakes that we would like to highlight, this is always risky. But it worked again! We didn’t have to show any prepared solution (to be honest we didn’t prepare any), we could explain and discuss a lot of interesting things through the samples the attendees came up with. It was amazing!

In the beginning of the workshop where we wanted to collect possible hopes and fears about BDD to be addresses during the sessions, we had another task as well. Usually we do this with yellow stickers but this time we made a last minute adaption to our process – making the session “progressive”… Instead of stickers, now we used a twitter – everyone could post comments with the term #prognetbdd that we displayed on the projector and discussed. It was a fun!

SpecFlow: Pragmatic BDD for .NET

2009-10-19T20:04:00.001+02:00

Last year I worked a lot on finding better techniques for automated functional testing.

I was investigating topics like how to solve problems of automated web UI testing (especially ASP.NET), how to structure automated functional tests or how to ensure the preconditions of functional tests. We even created small tools to support this, but altogether it was not a big success.

Finally, we started to use cucumber and this was the first tool where I felt that it supported the concept we need. However, we had some issues with cucumber, too:

Using it with .NET projects (through IronRuby) was quite slow, and since the IronRuby documentation is not too detailed, we continually encountered ruby/.net conversion issues that were hard to debug and solve.
We had our continuous build infrastructure, but it was not prepared for running cucumber tests.
The development team did not have sufficient ruby knowledge.

Considering these problems, we finally decided to move forward: to implement a small .NET running environment for cucumber, where the step definitions are written in .NET. Since I’ve gathered quite a lot of experience with Visual Studio SDK, I came to the idea to implement this tool as a single-file generator that generates NUnit test fixture files from the cucumber file, which can be executed by NUnit runner, also on the build server, without extra configuration.

We resolved to implement the bindings as .NET methods, having [Given], [When] or [Then] attributes.

[Given("I have entered (.*) into the calculator")]       
public static void GivenIHaveEnteredANrIntoTheCalculator(int nr)       
{       
    //...       
}       

The tool contains a runtime part as well that finds the matching step definition method during the test executions and executes them. The first implementation used an Oslo parser, but finally we had to rewrite it to use ANTRL, because of performance issues.

The tool worked surprisingly well and we started to use it in a project where we had our cucumber tests already; we just had to rewrite the step definitions to .NET, since the cucumber files were compatible.

In the meanwhile, TechTalk decided to make an open-source project out of the in-house tool. But life is not easy, we have quickly realized that the names we have used so far (“NCucumber” or “Cucumber for .NET”) cannot really be used as a “trademark”. The cucumber team – somehow understandable – wanted to keep the name “cucumber” for products which are direct ports of cucumber. Although we are compatible in the cucumber file (gherkin) level, we don’t have an own execution environment (NUnit does it for us). So we have chosen the new name: SpecFlow.

The first version of SpecFlow is already available, so you can try it out (the source-code is also coming next week). To learn more, visit the website:

http://www.specflow.org

DevCamp09: Behavior Driven Development Applied in Unit Testing

2009-10-15T20:27:00.002+02:00

The presentation material (slides and demo source code) of my DevCamp09 presentation are on-line now and can be downloaded from the TechTalk website here.

The downloadable material also contains the framework that can be used to write BDD-style unit tests. This framework will most probably be joined to SpecFlow, the open-source BDD tool we have just launched (hence, the new namespace, TechTalk.SpecFlow.UnitTesting).

Session Abstract

Developer life was good in the golden age of the waterfall: big upfront designs left no surprises for later implementation, code documentation was a piece of cake as nobody else read it anyway, and testing was something developers just heard of remotely through a department called “QA”.

But things have changed for the worse with the rise of agile methodologies: designs are ruined by requirements coming in weekly and code that was already working perfectly suddenly breaks. Now, business wants to collaborate directly and is interested in details of the implementation, meaning that documentation can no longer be kept on the back burner. And QA is asking for a high coverage of unit tests out of the blue, causing big overheads to write and maintain them with all these ongoing changes.

However, a lot of concepts have been introduced that promise to ease the pain: IoC helps us reduce dependencies in the code, BDD helps us focus on the next functionality to deliver, TDD helps us structure code for testability and Fluent Interfaces are a way of building DSLs that help us communicate with business.

This talk will show how we put these concepts into practice in a project that followed an agile methodology. We’ll show how IoC helped us stay flexible, and how we structured our unit tests in a way that described the expected behavior and could also be understood by business. The source code of the supporting infrastructure we built will be made available to you after the talk as a download.

Screenshots from the demo

Execution result of a very simple BDD-style unit test

…and the unit test itself

Execution result of a more complex BDD-style unit test

…the unit test…

…and finally the implemented code

Doing it right

2009-04-20T19:17:00.001+02:00

I’m a software developer. From time to time, I come to the question why it is good to be a developer? Why is it good to write software?

I don’t have the ultimate answer yet, but I already have a few: making someone's work easier; to able to get out more value from the things we know; because it is a joy for me; because I can teach other developers to find this joy; or simply because it provides a reasonable good salary. (I always pick the one that fits best to my mood, but honestly I can also argue with each of them.)

I’m not working on software that saves lives directly, and there was a time when I was very sad about that. But then I have realized that the world is full of people that also don’t do that, still you can be very happy that they do what they are doing. (Now I cut the long list of examples, because I tend to be too sentimental.) The point is that you are happy with them because they do their job right.

Today I have found the website of the TED community. TED stands for Technology, Entertainment, Design, and it is a community (and a yearly conference) where people try to bring these concepts together and make something remarkable. It is innovative, but not necessarily technology-wise. And very-very inspiring. The best talks from the conferences are also available on-line, so you should better look at them. They are all relatively short (the allowed time for a speech is 18 minutes).

Why does this come here? Because sometimes you have to step back and look at things from a broader perspective in order to step further or gather new ideas. Here are some:

Matthew Childs: Hang in there! 9 life lessons from rock climbing

You will start listening to this as it would be about rock climbing, but you’ll soon realize that this is also about you (and for example software development). (04:48)

Scott McCloud: Understanding comics

As you’ve guessed this is also about more than comics. It is about vision and about finding ways to understand things (take it apart and put it together). About how technology and art can cooperate with each-other. (In this sense, it can be also helpful for software usability.) And last but not least to show how to make a memorable presentation. (17:08)

In McCloud’s talk, there is an interesting list of rules which I have to think of:

Learn from everyone
Follow no one
Watch for patterns
Work like hell

I have learned a lot today.

Scripting database including data in SQL Server 2008 Management Studio

2009-03-18T12:09:00.001+01:00

I was always missing a function in the management studio of the different SQL Server versions. This was to generate a script from an existing database, that also contains the INSERT statements for the existing data in the tables.

Today I have found out, that there is already such a feature in the Management Studio of SQL 2008, although it is not so easy to find it.

Actually it is very simple. You only have to use the “Generate Scripts” wizard, and on the options step, you have to enable the “Script Data” option somewhere at the end of that long list.

Enable WCF hosing in IIS 7

2009-01-18T15:24:00.001+01:00

As I was playing with WCF services on Windows 7, I had to set up IIS to host a WCF service. It turned out that it is not as straightforward as I expected. Here is a small summary about the steps that you have to do. (I have found a post of Pandurang Nayak that describes these steps, but that article is not available currently, only through Live Search cache).

Don’t forget to install IIS… :) Yes, it took me some time to realize that it is not installed by default.
When you install IIS (Contol Panel / Programs / Turn Windows features on or off), the sub-components selected by default are not enough. You need at least the Windows authentication, the IIS 6 Management Compatibility (for Visual Studio) and probably some more, but I gave up playing with the sub-components and finally selected all.
Don’t forget to restart your machine. Although you are not asked to do that, for me it did not work.
Still in the “Turn Windows features on or off” dialog, you have to enable sub components of .NET 3.5: WCF HTTP and Non-HTTP Activation. (Did you know that you can do such things here?)
As I have installed IIS after installing Visual Studio 2008, I had to register the ASP.NET components for IIS by executing C:\Windows\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis.exe –i
Finally you have to register WCF as well for IIS (to understand the .svc extension for example). You can do it by executing C:\Windows\Microsoft.NET\Framework\v3.0\Windows Communication Foundation\ServiceModelReg.exe –i

If you have followed these steps (plus maybe a few more restart) you will be able to host WCF services on IIS 7.

Visual Studio 2008 can help you to configure and debug the virtual directory for you, but for this feature you have to run Visual Studio as administrator.

But fortunately there is an informative error message if you forget to do so.

Windows 7 and developing WCF applications

2009-01-15T15:05:00.001+01:00

Yes, I know that this is weird, but I have installed Windows 7 to my development notebook to see how it survives. (Ok, I have a backup XP as well on the machine.)

Generally it works very well, I have not found any major problem with Windows 7, and it seems that the stability has been improved a lot since the pre-beta.

The only problem I have encountered so far is with WCF service applications. If I create a WCF service and host it on the ASP.NET Development server, then all of the WCF requests return an HTTP error:

The remote server returned an unexpected response: (400) Bad Request.

This can even be reproduced with the simplest WCF service that is created for example with the “WCF Service Application” project template.

It is also interesting that the service serves the WSDL request well, it just produces the error when you call a service method. With IIS (after I was able to set it up) the service worked fine.

I have no idea currently what can cause this behavior, but it can be reproduced on other Windows 7 machines as well.

Update: It's a known security-related issue with Cassini in Windows 7 beta build 7000 and will be fixed in next release. (see http://social.msdn.microsoft.com/Forums/en-US/wcf/thread/123f0d20-a018-48b0-ab92-4aae749f2d5e/)

Thanks to Johan for the update (http://weblogs.asp.net/jdanforth/archive/2009/03/14/bad-request-with-wcf-service-in-cassini-on-windows-7-beta.aspx).

OracleClient problem in .NET 2.0 SP2 (comes with VS2008 SP1)

2009-01-13T10:24:00.003+01:00

As mentioned in a lot of sources, Visual Studio 2008 SP1, in spite of its name, also contains service packs for the .NET core components. Namely:

.NET 2.0 SP2
.NET 3.0 SP2
.NET 3.5 SP1

Unfortunately there is not too much information available what these components contain, but it seems that they may seriously affect applications that were running perfectly before.

It seems for example that .NET 2.0 SP2 contains a change in the System.Data.OracleClient assembly that may break applications using an Oracle database.

The most visible result of the issue is that LOB handling fails with the exception:

Modifying a LOB requires that the connection be transacted.

Although this issue causes breaks in applications, there are even more dangerous effects of the bug. The transactions may automatically switch “Auto-commit” mode that can produce inconsistent and corrupt data in the database even without any visible error. More details here.

In December, Microsoft released an update to the .NET Framework service packs: KB959209

Funnily, Microsoft has not stopped confusing developers with the updates. Although the title of the update is “Microsoft .NET Framework 3.5 Family Update”, it contains updates for .NET 2.0 SP2, .NET 3.0 SP2 and .NET 3.5 SP1. You can install these updates individually. (To resolve the Oracle issue, it is enough to install the fix for .NET 2.0 SP2.)

Update: It seems that the update can be installed only to English .NET frameworks. Other language versions (e.g. German) can be also found at Microsoft Download Center (although it is not so easy). Here are the most common download links for German .NET frameworks:

WCF Security Guidance

2009-01-07T13:57:00.001+01:00

Probably it is well known, but I've only found it now:

patterns & practices: WCF Security Guidance Project

It is a complete book about WCF security on-line (you can download it or just browse through it). With its scenario-based approach and the practical samples and "how-to" chapters it seems to be very useful.

Visual Studio 2010 - Historical Debugger

2008-12-16T13:44:00.001+01:00

I've played a little bit with Visual Studio 2010, trying out the automated web UI tests, but I have found a feature that became quickly my favorite from the VS20010 feature list. This is the historical debugger.

The idea is easy: the IDE collects and stores information about the debugged application (important events, called methods, parameters, (handled) exceptions, etc.) that allows you later to jump back in time, and check what happened before you have actually run into a problem that you want to fix. You can walk through the execution steps and review when the things went wrong. (The experience is a little bit similar like the call stack window, but here you are not constrained to the callers and you can get much more information about the executed methods.)

Of course nothing is for free, the application will be slower with this historical debugger, but MS addressed this problem, and you can fine-tune the information to be collected (or even switch it off), and you can also limit the size of the collected data (it always keeps the last part of it, so when you break the execution the information that is relevant to the experienced problem will be most likely still available).

You can see a nice screencast about this new feature by Habib Heydarian and also some screenshots from Pavel Nikolov.

Entity Framework as an OR/M

2008-12-10T13:10:00.000+01:00

I know that Entity Framework (EF) has a lot of concepts behind the OR/M features, but I think most of the people who try it out just use it as an object-relational mapping tool. And it seems that also the tools around it (designer) are also driving you to this direction. And BTW this is the reason why MS could announce it a recommended replacement for LINQ to SQL.

I have also investigated EF from this point (I did a small application with it). Here are my feelings.

Note: I have used the currently released version of the Entity Framework, as I wanted to see, how it is useable now.

Generally I have been quite productive with EF and I have not encountered any problem that I could not solve. On the other hand, it did not really impress me. It does not really try to be smart, you have to do everything you want explicitly. This seems to be a defined concept that I can accept, but in some cases the default behaviors are not defensive enough, which caused some tricky errors in my application.

So I have collected some things where IMHO it makes sense to take extra care when writing an application with EF.

Designer

I think in a larger project the designer will be the first problematic point. As normally the model is modified by multiple developers, I already see the nasty merging issues that you can run into. But even for a single developer, it is very hard to have an overview of your domain model in a single screen, where only 6-8 entities can fit nicely.

I also encountered with funny issues with the designer that does not seem to support the code-first approach to extend your domain model. I wanted to create a new entity, so I selected the Add/Entity in the designer. Then I realized that I cannot map it to a table, until I add a table too - which you cannot do from the designer (at least I could not find out how). The new unmapped class, although it was not used in the application yet, broke all other use-cases too, so my application stopped working. So finally I created the table manually in the database and refreshed the model. It still did not work, and after 10 minutes I realized that the refresh also created a new entity with an "1" postfix and mapped the new table to it - but as the designer surface was full, the generated entity was out of the screen (and I did not scroll down). The only thing that I don't understand is why it is allowed to create a new entity from the designer then? And of course the same refers to the new properties created in the designer.

Lazy framework

From the OR/M functionality my biggest problem was the way how EF handles the unloaded relations. There is no lazy loading, I knew that. But I was very much surprised that if you access a property that is not loaded yet, you don't receive an exception! For references, it returns null, for collections an empty collection. This is a great source for hidden errors, so I decided that I'll consider a line of code like these, simply as a mistake:

Console.WriteLine(product.Category.Name);     
--      
DoSomething(product.Category)      
--      
foreach(var item in order.Items)      
    //...      

These lines properly should look like this:

if (!product.Category.IsLoaded)      
  product.CategoryReference.Load();      
Console.WriteLine(product.Category.Name);      
--      
if (!product.Category.IsLoaded)      
  product.CategoryReference.Load();      
DoSomething(product.Category)      
--      
if (!order.Items.IsLoaded)      
  order.Items.Load();      
foreach(var item in order.Items)      
  //...      

You say now that this is lazy loading. Ok, you are right, but if you want to be correct and you don't want to have lazy loading (because it is the evil that makes our application wrong), you still have to write:

if (!product.Category.IsLoaded)      
  throw new ProgrammersFaultEx(      
    "Don't you think that this should be loaded, mate?");      
Console.WriteLine(product.Category.Name);      
--      
if (!product.Category.IsLoaded)      
  throw new ProgrammersFaultEx(      
    "Please load the category, please, please, please!");      
DoSomething(product.Category)      
--      
if (!order.Items.IsLoaded)      
  throw new ProgrammersFaultEx(      
    "I told you already, that this wont work!");      
foreach(var item in order.Items)      
  //...      

You say now that it is too verbose. And you are actually right.

Transactions and query consistency

There is a little bit similar problem with the query consistency. The use case is that you write a method that lists some products.

void ListProducts()      
{      
    foreach(var p in db.Product.Where(p => p.Color == "blue"))      
        Console.WriteLine("{0} {1}", p.Name, p.Color);      
}

This method will work fine until you call it from an update use case, where you already updated a color of a product from blue to red. In this case you might see something like:

product1 blue     
product2 blue      
product3 red

Strange, isn't it? The problem is that the db.SaveChanges(); is only invoked at the end of the transaction, and therefore the database does not know about this change at the time when the method is called. The solution: you have to call the db.SaveChanges() earlier. Clear. But who should do this exactly? The method that updated the color? Just to make sure that methods like ListProducts() work well if they called later? Or should the ListProducts() call it? Just to make sure that it does not depend on pending changes? Or should someone else call it?

Well, of course this is not an easy question (which is good, if you see this as a reason for salary increase), but the funny thing here is that the OR/M knows that there is a pending product change and that you are about to list products! So why I should care?

Collection as a query

Sometimes you don't want to populate a collection, but you want to use it as a query for further transformations. If you write

order.Items.Where(oi => oi.Product.Color == "blue")

this will filter the loaded collection (which is an empty, when you forgot to call Load()). If you want to query the database, you have to write:

order.Items.CreateSourceQuery()      
    .Where(oi => oi.Product.Color == "blue")

But this will not contain the items, that are added/removed locally, so you have to call db.SaveChanges(), or check whether it is loaded already, so you end up with:

var collectionToQuery = order.Items.IsLoaded ?      
    order.Items : order.Items.CreateSourceQuery();      
collectionToQuery.Where(oi => oi.Product.Color == "blue")

but it does not compile, as order.Items is not queryable... :) So the winner is...

var collectionToQuery = order.Items.IsLoaded ?      
    order.Items.AsQueryable() : order.Items.CreateSourceQuery();      
collectionToQuery.Where(oi => oi.Product.Color == "blue")

And now you are finished.

Small annoying things

Of course I've found also some small annoying things. Without too much explanation here is a list:

Single() / SingleOrDefault() does not work in LINQ to Entities. You have to use First() / FirstOrDefault().
There is no operator in LINQ to Entities to express the SQL "LIKE". They suggest to use Contains(), but LIKE can do more!
You cannot map custom methods / properties, so you cannot encapsulate parts of the queries, and also you have no chance to solve things like the missing LIKE.
I have not found a way how to map a database field to an enum property. (You need to write a wrapper property).
There are not too many ways to control what is cached in the object context, you can Detach() objects individually or skip loading of a query result.
If you use the Include() method to add relations to the query load span, you have to specify the expressions to be loaded as string. In the time of lambdas, this is quite ugly IMHO. Also the syntax looks quite strange, if you want to include a collection and a reference in each item in the collection, as you have to write this as: orders.Include("Items.Product"), although Items is an EntityCollection<OrderItem> that does not have a "Product" property.
You cannot compare entities in queries, you have to compare the id fields (the id fields should be hidden by the OR/M as much as possible):
...Where(p => p.Customer == selectedCustomer)
causes

"Unable to create a constant value of type 'Closure type'. Only primitive types ('such as Int32, String, and Guid') are supported in this context."
and has to be written as:
...Where(p => p.Customer.Id == selectedCustomer.Id)