Frans Bouma's blog

ORM Profiler now .NET 4.5 / Async aware

FransBouma — Fri, 24 May 2013 11:29:28 GMT

We updated our data-access profiler ORM Profiler with a new .NET 4.5 specific interceptor dll. This interceptor, which is called SD.Tools.OrmProfiler.Interceptor.NET45.dll, can be found in the main installation folder of the ORM Profiler build we uploaded this week. If you're an ORM Profiler customer, please download the latest build from the customer area.

The .NET 4.5 interceptor intercepts calls to Async specific ADO.NET methods, and also contains all Entity Framework required classes, so there's no need for the EF4 specific interceptor if you're using .NET 4.5 and Entity Framework 5. Using this interceptor, any call to a *Async method will not fall back to the synchronous calls of System.Data.Common classes as the previous builds did, but will call into the Async methods of the used ADO.NET provider.

There's more async news coming soon, namely for LLBLGen Pro v4, stay tuned

LLBLGen Pro v4.0 feature highlights: Visual Studio integration

FransBouma — Mon, 13 May 2013 10:32:29 GMT

This is the first post of a series about the new features in LLBLGen Pro v4.0, which was released on April 6th, 2013. Starting with v4.0, the LLBLGen Pro designer is usable both as standalone (as it was before) and as an integrated designer in Visual Studio. I'll explain in this article what this integration is all about and what we had to do to make it work. If you want to know more about how to use this feature, please check out the manual page on this feature.

So how does this integration look like? Below is a screenshot from the manual:

It almost looks like the stand-alone designer, eh? Integrating in Visual Studio can be done in several ways: through e.g. a plug-in/add-in, an extension to the editor or a Visual Studio extension. Our integration is done through a Visual Studio extension (VSIX).

How the integration works, user perspective

When you start Visual Studio, the extension isn't visible: no menus, no tool windows, no tool bars. When you load an LLBLGen Pro project (a file with the extension '.llblgenproj') into Visual Studio, or create a new one inside Visual Studio using the normal item creation system, the designer is loaded inside Visual Studio: all panes are opened as tool windows and the project file itself is opened as a document editor inside the document well (central tab area) of Visual Studio. When this document editor, which is a single tab inside the document well, has the focus, the toolbar and menu are visible, and the project's contents is visible in the various tool windows (see screen shot above).

Opening another LLBLGen Pro project inside Visual Studio opens another document editor, but no extra tool windows. Instead, the contents of the tool windows is switched when you switch between document editors so they always show the project contents of the document editor which has focus. For a background on this, please see the section below called 'How we wrote the VSIX'.

Generating code will add the generated VS.NET projects to the loaded solution, if they're not already present there. This way of working as the advantage that you don't have to leave VS.NET to work with the designer, you can add the LLBLGen Pro project to your solution as a bonus don't have to manually add the generated code to your solution.

All functionality, except imports (see below) is the same as with the stand-alone designer, as the stand-alone designer is actually loaded inside Visual Studio and its parts are displayed within the IDE.

How we wrote the VSIX (or: how to lose your hair in 2 months)

Visual Studio is mainly an IDE around editing files: a file is opened with the associated editor of the file's extension in the document well and the editor associated with the file's extension can open 0 or more tool windows and do other things. You can of course open tool windows inside the document well which edit 'objects' and not 'files', however this isn't as simple as it sounds. And precisely that was a problem we had to solve: our designer is also an IDE with a central tab area but instead of opening files in that tab area, we open editors of objects.

To make Visual Studio work like our designer, we were forced to go what I'd call, 'the All-In' route: create custom project types for Visual Studio, rewrite our project explorer to build objects in the Solution Explorer in Visual Studio, open custom tool windows inside the document well of Visual Studio for each object edited. We quickly learned that creating an extension to Visual Studio is not a walk in the park: the code isn't that complex, it's the lack of serious documentation, examples and above all, the quirks of the IDE which made it a big a project to complete. Looking at the work for the All-in variant, we decided to go for the alternative: create a file-extension bound editor. Looking back this was a wise decision, as this extension too took us a long time to create, almost two months.

With v3.0, released in early 2010, we designed the designer to be modular so it wouldn't be to much work to make it usable from within another application, e.g. Visual Studio. The UI is broken up in several parts and all designer logic is structured in an event-oriented system with several controllers on top. The advantage of this is that whenever something happens inside the UI, one or more events are raised and observers to these events (or the events as result of these events) are picking them up and act accordingly. These observers are all located outside the UI. The main advantage of this is that even though the designer in stand-alone form looks like an MDI window with tool windows docked inside it, the control of these docked windows and what happens inside them is not within the MDI window.

The v4.0 designer is a successor of v3's so with the modular design at hand we decided to create a Visual Studio extension tied to the LLBLGen Pro project file file-extension '.llblgenproj'. In Visual Studio terms: a custom document editor. A Visual Studio extension is actually a Visual Studio package, packed in the form of an 'extension'. The extension contains of the following parts:

Each docked 'pane' in the designer is opened as a Visual Studio Tool window and docked inside the IDE
The document well of the designer is opened as a Visual Studio Document Editor in the document well.
The toolbar is created as a button bar inside Visual Studio
The menus are pivoted into a single menu in the main menu of Visual Studio
A factory which produces new document editors

To achieve this, we wrote an extension which loads the standalone designer into Visual Studio, and instead of telling it to open an MDI window and dock some panes into it, it simply tells the loaded designer to hand the pane windows and its own window to the extension object. The extension object then opens the Visual Studio tool windows and its main editor and there you are: integrated inside Visual Studio.

Simple? Well… not quite.

The fun starts with where Visual Studio extensions are located. There are two locations: inside the user's My Documents folder and inside the Visual Studio installation folder. Both locations are not ideal if you want your code to also be usable as stand-alone. The trick was to create a registry key at installation which is read by the Visual Studio extension which points to the folder where LLBLGen Pro was installed. I tried numerous things, including junctions but these all failed. The registry key is easy enough though.

The next problem was something we never had to run into before: what if multiple LLBLGen Pro projects are opened inside Visual Studio? Although opening multiple LLBLGen Pro projects will open multiple instances of the designer, without separating the AppDomains, we'd run into problems where statics were suddenly shared among instances. However, creating a new AppDomain per instance wouldn't work as that would require the editor panes we would like to show inside the designer to serializable, which isn't possible. Luckily we had grouped all statics inside objects which lived in a few singletons which act as Services inside our designer.

Whenever a designer instance was loaded into the extension, we assigned to it an ID and we'd request an object we dubbed integration controller from the designer, which would act like the controller of the designer and data exchange. Through this controller we also could tell the designer instance which document editor instance was active. The services/singletons then switched their state to the instance ID passed to the integration controller and all code of the designer and its sub-systems would think they were running inside the stand-alone version.

This state-switching also helped with the tool windows of Visual Studio: by default each Visual Studio tool window is created just once. You can create multiple instances if you must, but you'll soon learn that multiple instances of a tool window makes things less usable. So each time a document editor became active inside the IDE, the document editor would get called by the IDE and we simply made the extension to update the associated designer's state to the one associated with the document editor's ID and as well update the tool windows to the ones of the designer instance associated with the ID.

On to undo-redo. Our own designer has a full undo-redo system, based on our OSS library Algorithmia (available on github), and so does VS.NET. Of course these aren't compatible with each other however the designer's undo-redo should work in sync with the VS.NET undo-redo system. After looking into making things compatible for a few days, it occurred to me it doesn't matter: whenever a user does something through the document editor, the IDE will query your editor if the undo/redo state of the buttons should change. Our designer already kept track of that, so simply returning this state was enough. Whenever the user pressed the undo/redo buttons (or the keystrokes) inside the IDE, the document editor with focus got a call which was passed through to the integration controller to do the actual undo/redo on the model, similar to if the user would undo/redo in the stand-alone designer.

Undo-redo is a global system, however with multiple instances, there are multiple undo/redo stacks active at once, one per instance. Algorithmia's undo/redo system is able to work with that: it works with 'scopes'. This means you can have local undo/redo separated from the global undo/redo. We utilize this in the integration so each designer instance is a separate scope inside the undo/redo system, so undo/redo information isn't shared.

The last problem I'd like to address is installation. Visual Studio extensions, VSIXs, can be installed in two ways: you either dump all files in a folder in one of the two extension locations and let VS.NET know some extension installation took place or you wrap your code inside a .vsix file. The latter has as advantage that registration of your extension is done for you without a slow re-scan of the complete extension folders. It also takes care of locating the extensions folders. I first looked at the former alternative, as we use our own installer but the logic to locate the folders and install everything in both 2010 and 2012 was a pain, and everything was taken care of with a .vsix file: we simply run it silently during installation and it installs the extension in both 2010 and 2012 if present, in the right folder and registers the extension as well.

Was it all worth it?

Yes. Developing this extension was a true pain, I'd be lying if I called it something else, but as a tool / framework developer it's part of our job to write the painful code so our customers don't have to; that's why they pay for a license. In the end, I can only say I'm very pleased with the result and have heard only positive feedback from users, so in short: yes it was worth it.

As I said above, creating a Visual Studio extension isn't a walk in the park: using .NET for an extension is possible, the SDK looks OK, the starter project gives you a lot of code to start with, but after a short while you'll learn that below this thin layer of chrome, the actual metal is pretty rusty and shows its true age. The technical debt of the C/C++/COM based system is so big, I doubt it will ever get better, especially as there's little incentive to do so: the biggest 'customer' of VS.NET's integration system is Microsoft itself, and as we all know internal code is more rough, less documented than software that's delivered to customers outside the company.

If you're planning to develop your own Visual Studio extension, be aware of the fact you'll spend a lot of time cursing the IDE, it's long list of quirks and inconsistencies, missing docs and serious lack of examples how to do things: you'll run into many posts with similar questions as yours but with no or unusable answers. You're entitled to throw your hands into the air and scream the filthiest curse words in the direction of Redmond, but that won't get you near your goal. As an engineer, you have to think and act like one: write some prototype code and attach a debugger in another VS.NET instance to your test VS.NET instance to see what's happening, what's called when you do X.

It will be a struggle at first, but you'll get there: and you'll have to do it yourself, there's no-one out there to help you. But hey, that's the life of an engineer.

LLBLGen Pro v4.0 released!

FransBouma — Mon, 08 Apr 2013 09:48:08 GMT

I'm proud to announce we've released v4.0 of LLBLGen Pro! This release is the 13th full release we've done and it arrives in the year LLBLGen Pro turns 10. As always we've packed the new release with great new features and I hope we've added what you wanted us to add. You can find what's new here: http://www.llblgen.com/pages/whatsnew.aspx

Not only did we add new features, we also introduced something else: Lite.

Lite

LLBLGen Pro Lite is a free version of LLBLGen Pro. It packs all the features of the full version, however it comes with a limitation: an entity model is limited to 8 entities per model and it also come with no support. It's free as in beer and you can use the generated code in production.

Upgrades

V3.x users can upgrade with a 30% discount, by logging into the customer area.

LLBLGen Pro v4.0 BETA released!

FransBouma — Thu, 07 Mar 2013 16:35:59 GMT

We've released the first beta of LLBLGen Pro v4.0! We hope you all like the new features and additions we packed into this new release! To make sure stuff is truly tested in a lot of scenario's, the beta is available to all v3.x licensees.

How to get the beta
Log into the customer area, then select 'v3.5' from the Downloads menu and navigate on the downloads page to 'Betas'. No license is required, the beta license is included in the installer.

Installation
Run the installer as administrator. It will automatically install the VS.NET extension to use the designer inside VS.NET in VS.NET 2010 and VS.NET 2012 (pro and up), if available. Of course the designer is also usable as stand-alone, outside VS.NET.
The installer will start the VSIX (visual studio extension) installer at the end of the normal install. It might be the VSIX installer is still running when the normal installer ends. If you have VS.NET open when you install the beta, you'll have to close VS.NET and restart VS.NET to see the add-in.
You can install the beta side-by-side with older versions of LLBLGen Pro. V4's file format is compatible with v3, however a v4 file which is loaded into v3 and saved in v3 will lose v4 specific data.

Documentation
We included a word document with the details about this beta, which features are included and how to use them. This document, called LLBLGen Pro v4.0 beta readme.docx, is installed in the LLBLGen Pro v4 beta installation.

How to provide feedback
Please provide all feedback in the beta forum: http://www.llblgen.com/tinyforum/Threads.aspx?ForumID=52

What's new in V4.0
The list below is the global list of what's new in V4. The enclosed word document has the fine details of these new features and changes and also details of minor changes and new features not listed below.

Designer

Full Visual Studio integration (VS.NET 2010 / 2012)
Table Valued Functions support (EF/Linq to Sql/LLBLGen Pro RTF)
Additional interfaces/namespaces now use the rule system
Update / Delete rules for FK constraints (Cascade delete/update)
SQL Server 2012 sequence support (LLBLGen Pro RTF/NHibernate)
Action suggestions window
Custom type shortcuts can now have default length/precision
The designer is now a .NET 4.0 application, no more .net 4 runners.
Char <-> String system type converter
Re-scan of enum typeimport files without closing designer

Runtime Framework

Table Valued Functions support
Query result cache
Much faster entity fetches
Inlining DDD value types
SQL Server 2012 sequence support
Data Scopes (see https://github.com/SolutionsDesign/LLBLGenProExamples_4.0)
Xml Serializability of entire query API
True skip / offset
Fetch typed lists with QuerySpec
Runtime libraries are now built against .NET 3.5
QuerySpec and Linq assemblies are merged into ORMSupport Classes
OData support classes are now built against WCF Data Services 5.3 and support OData v3

I'm especially pleased with the VS.NET integration which turned out to be very nice, the TVF support, the much faster query fetches, the query result cache and the DataScopes. Please check out the example available at github (like above) how little code it takes to make a typical LoB application with the new DataScopes.

Re: You’re not your code

FransBouma — Fri, 25 Jan 2013 10:26:32 GMT

This post is a reply to Scott Hanselman’s post: "You’re not your code"

There are no two programmers in the world who like each others code, there's always criticism: "I would have done this differently" or worse "Whoa, dude, never heard of SOLID? Where are the tests for this?". IMHO, criticism is OK, as long as it's based on well founded arguments and they make the original programmer of the criticized code learn new things. I don't think Scott's post is about that.

I think Scott refers to this: simply bitching about someone else's code because you'd have implemented the same program much better, as you see yourself as a great developer is stupid.

I'd like to make an addition to that: if you think you're a great developer and express yourself in that fashion, and your professional code is not code one can expect from a professional (i.e. code which is maintainable, without (most of the) bugs, it's well documented etc.), you're a fraud and it's a little bit different: you tried to sell the idea that your hack, your hobby code is professional code.

However, not all code is professional code. Sometimes we simply hack something together to make something happen 1 time, or only for a single user (ourselves). For example internally here I use a change log system in where I log changes made to the various parts of the tools we produce. This is a very rudimentary piece of software, with a tremendously ugly UI, buggy event handlers and the databinding sometimes gets confused. This is because I wrote it in one afternoon, no tests, no design thoughts, just started hammering in code and when it compiled without errors I said "Done!".

But it works, it gets the job done of what I want to do: log the change made into a database so I can sync it with the online change log on our website so users can browse the change log online. Every time I run into a bug (which isn't often, as I know what to click and do) in this marvelous piece of code (ahum) I say to myself "it might be better if I fix this some day", but as the only time I use this software is when I have to do a push of a new build, this 'some day' is not today.

If this software would ever be released somewhere in source-code form, it will likely be bashed to pieces. If I would release it as source-code with the remark 'great system to log your changes and sync with online database', the bashing would IMHO be justified. But as it's a tool which is only for myself, and if I would release it as 'Dunno if this helps anyone, it works for me, have fun' and a bigwig would bash it to pieces because he's holier than me, that bashing IMHO wouldn't be justified: it's a hack, not professional software.

Bashing a hack released as a hack is stupid. Bashing a hack released as professional code isn't, as it exposes the hack as what it truly is: a hack.

Question of course is: when is a piece of code a hack / hobby code and when does it become professional code? IMHO the variety of answers to that question will tell you why the bashing of other people's code will continue no matter what and why there will never be two programmers in the world who like each others code without criticism.

Though we can do something. As it's difficult to determine when a piece of software is a hack if it's not described as such (not by one person but as a general qualification: if piece of software A is declared a hack, everyone will agree it is in fact crap), we as developers could keep that in mind when we use our bash-hammer on some piece of code we found online somewhere: what is truly gained from our criticism? Will the world be a better place? Will the software magically become better? Or is it just about our own ego that by declaring piece of software A a true example of crapcode, our own code must be much better?

The main reason we have to take that in mind is because we are our code. Well, not entirely, but we did write it. An outsider won't know whether we poured our soul into it and that we invested precious time in it. Bashing it to pieces because the fruits of our hard work is seen as a hack while it's not is going to hurt. If it doesn't hurt, you're not caring enough about the quality of your work, or they might have been right: your work was a hack after all, it didn't exceed the hobby code bar.

LLBLGen Pro and JSON serialization

FransBouma — Sun, 04 Nov 2012 15:29:01 GMT

I accidentally removed a reply from my previous blogpost, and as this blog-engine here at weblogs.asp.net is apparently falling apart, I can't re-add it as it thought it would be wise to disable comment controls on all posts, except new ones. So I'll post the reply here as a quote and reply on it.

'Steven' asks:

What would the future be for LLBLGen Pro to support JSON for serialization? Would it be worth the effort for a LLBLGenPro user to bother creating some code templates to produce additional JSON serializable classes?

Or just create some basic POCO classes which could be used for exchange of client/server data and use DTO to map these back to LLBGenPro ones? If I understand the work around, it is at the expense of losing xml serialization.

Well, as described in the previous post, to enable JSON serialization, you can do that with a couple of lines and some attribute assignments. However, indeed, the attributes might make the XML serialization not working, as described in the previous blogpost. This is the case if the service you're using serializes objects using the DataContract serializer: this serializer will give up to serialize the entity objects to XML as the entity objects implement IXmlSerializable and this is a no-go area for the DataContract serializer.

However, if your service doesn't use a DataContract serializer, or you serialize the objects manually to Xml using an xml serializer, you're fine.

When you want to switch to Xml serializing again, instead of JSON in WebApi, and you have decorated the entity classes with the data-contract attributes, you can switch off the DataContract serializer, by setting a global configuration setting:

var xml = GlobalConfiguration.Configuration.Formatters.XmlFormatter;
xml.UseXmlSerializer = true;

This will make the WebApi use the XmlSerializer, and run the normal IXmlSerializable interface implementation.

How to make ASP.NET WebApi serialize your LLBLGen Pro entities to JSON

FransBouma — Fri, 05 Oct 2012 13:01:00 GMT

LLBLGen Pro has several ways to serialize / deserialize your entity data in an LLBLGen Pro Entity object:

Binary, using superfast custom binary formatters, triggered by ISerializable methods.
XML, using XmlDocuments
XML, using XmlWriters, triggered by IXmlSerializable methods. Xml is configurable to various formats, from verbose (to rebuild the objects using the embedded type info) to compact ("Just the data, ma'am")

Once upon a time, all these fancy methods were the latest and greatest. Today however, they're not modern anymore, as a new format is taken over: JSON. JSON as the advantage that you can easily consume it in javascript on the client, no custom XML parser needed.

Microsoft recently released ASP.NET 4.0 MVC with a fancy new service system: WebAPI. WebAPI let's you create a service without the hassle that's required with a WCF service running inside IIS. Not only makes WebAPI it easy to write a service for your data needs on a client, it also makes it easy to serialize/deserialize this data as JSON.

This post is about how to make WebAPI work with LLBLGen Pro and serialize the data to JSON without the internals bleeding into the JSON data.

Note: by following these steps, the data is serialized as JSON only, the XML serializer won't work anymore (see below). By default, WebAPI serializes to XML, so you need these steps only if you want to serialize to JSON.

Step 1: decorate your entities, fields and navigators

LLBLGen Pro's designer has the ability to decorate generated classes and members of the generated classes with attributes, even based on rules you set. See a previous blogpost about this feature or the documentation for details.

Load your LLBLGen Pro project into the designer and select from the main menu Project -> Settings. A dialog opens. Navigate to General -> Conventions -> Code Generation -> Attributes.

At the top, 'Entity' is selected. Add below Attribute definition: DataContract

Press enter.

Select at the top 'NormalField'. Add below Attribute definition: DataMember

Press enter.

Repeat the action you did with NormalField also for NavigatorSingleValue and NavigatorCollection.

Click OK to close the settings dialog. You have now decorated all entities, the fields of the entities and the navigator properties with the required attributes.

Regenerate your code.

Step 2: Make WebAPI's JSON serialize do the right thing

Now, you might think that with the DataContract/DataMember attributes, we're going to use the DataContractJsonSerializer, but that's not the case. Not only is that serializer not recommended by many, including Rick Strahl, it also fails to do anything as it complains that the entity objects implement IXmlSerializable and it can't work with objects which implement IXmlSerializable and have DataContract attributes. These kind of errors show how dumb the serialization strategies (or lack thereof, if they had implement a proper strategy pattern for this it would all be OK) of Microsoft have been during the years: I have to have IXmlSerializable implemented on the classes for Xml serialization over normal WCF services to make sure change tracking data is preserved and also object graph references are left in-tact. The same error is given when we try to serialize the data to XML instead of JSON.

We're going to use the default JSON serializer of WebAPI, which is the Newtonsoft JSON.NET serializer. By default it doesn't do what we want, so we have to configure it. It's simple:

Open WebApiConfig.cs which is located in the App_Start folder. Now add the following lines to the Register method:

var json = config.Formatters.JsonFormatter;
json.SerializerSettings.PreserveReferencesHandling = Newtonsoft.Json.PreserveReferencesHandling.Objects;
json.SerializerSettings.ContractResolver = new DefaultContractResolver() 
    { 
        IgnoreSerializableInterface = true, 
        IgnoreSerializableAttribute = true 
    };

Step 3: There's no step 3

You're all setup now. A method like:

public IEnumerable<ProductEntity> GetAllProducts()
{
    using(var adapter = new DataAccessAdapter())
    {
        var metaData = new LinqMetaData(adapter);
        return metaData.Product.WithPath(p=>p.Prefetch(x=>x.Category)).ToList();     
    }
}

will now properly serialize the data (All product entities and their related category entities) to JSON, including reference tracking. Again, pulling the data as XML by specifying application/xml in the Accept header element of the request will make things fail as WebAPI chooses a DataContract serializer by default if it sees a [DataContract] attribute on the objects to serialize. It might be configurable to have it choose to use the normal XmlSerializer instead, but I'm no WebAPI expert so I can't give you a heads-up on that. If you know whether this is possible, please leave a comment below :)

Hope this helps!

Re: Everything's broken and nobody's upset

FransBouma — Tue, 18 Sep 2012 08:19:55 GMT

This is a re-worked answer to Scott Hanselman's great post 'Everything's broken and nobody's upset'. I replied it below the post, but I think it deserves a post here. Scott's post is about how we seemingly accept a long list of annoyances and issues with the software we use and how we apparently ignore the fact just how broken today's software seems to be. In my answer I tried to give an answer to why that is. This answer is below.

I think a lot of the problems with today's software are simply allowed to show up in public because the dev teams of these software products aren't given the time to fix them: Time to Market is critical, marketing already reserved a given launch date for the product, things are already in motion and only the bugs in the category 'it kills kittens and old ladies' are fixed at that point, the rest is 'moved to a later date'. But that later date never comes, because after the product is done, the team moves on to the newer version with new stuff, and fixing bugs is dreadful work, new stuff is exciting.

There's another thing about bugs which is also key in this issue: fixing them won't sell more licenses of vNext: people won't run to the store, yelling "OMG! They fixed bug 33422!!!1 I can't believe it!". They'll run to the store because new, shiny things have been added, like a completely new UI with ergonomic characteristics of which no-one really knows whether it's actually a step forward.

We, as software developers, all know software contains bugs, even though we did our best to fix them before RTM. But in the end, we have to admit that that label, 'RTM', is really just an arbitrary calendar event, not a landmark which says "0 bugs, It's ready!". This means that even though something went RTM, it doesn't mean it's bug free, it simply means: "It's good enough that it won't kill kittens nor old ladies".

A wise man, who unfortunately passed away way too soon, once said to me: "Your motivation and ability to fix issues and bugs is part of the quality you want to provide", which means: even though at first glance your software might look stunning out of the box, if that essential part of the quality of the software is missing, i.e. when a bug pops up it gets fixed, pronto, your software isn't of the quality you think it is. The insight given by this wise man opened my eyes all those years ago and I to this day still try to live up to it: stand for the quality you want to provide by giving customers the best support and software they could get.

If I look at today's software development landscape, I see a tremendous amount of people trying to write applications without the necessary skills and knowledge, in languages and platforms which have had a bad track-record for years, yet no-one seems to care, or at least too little. Last year I was in an open spaces session and some guy explained that they would place people who just started programming as 'trainee' at customers for 'free' and after a few months these trainees became 'junior' programmers and the client had to pay a fee per hour. I asked him what he thought of the term 'fraud', and he didn't understand what I meant. I tried to explain to him that if you think a person who has no training at all and you let him 'learn on the job' for a few months, is suddenly able to write any form of software, you're delusional. He didn't get what I wanted to say, and how it was a bad thing that someone with no skills was placed at a client as if the person knew what he was doing. Shockingly, some others in the room agreed with him.

But with the lack of highly trained professional developers growing and growing each day, more and more people who can tell the difference between a keyboard and a mouse are hired to do 'dev work', as the client doesn't know better and is already happy at least someone is there to do the work. I fear it only gets worse in the coming years. Frankly, I'm starting to get fed up with the bullshit pseudo-devs who seem to pop up more and more every day, who cry 'I'm just learning, don't be so rude!' when you tell them their work pretty much doesn't cut it, while at the same time they try to keep up the charade that they're highly skilled and experienced. You're either a trainee and therefore your work can lack in areas, or you're a professional and you have to take responsibility for the quality you say you'll provide.

Sadly this process of pseudo-devs which are seen as the true 'specialists', is in progress for some time now and will continue in the next decade I think. Let's hope they'll learn something about what 'quality' means with respect to software, that 'quality' is more than just the bits making up the software. But I'm not optimistic.

The Windows Store... why did I sign up with this mess again?

FransBouma — Wed, 12 Sep 2012 14:36:27 GMT

Yesterday, Microsoft revealed that the Windows Store is now open to all developers in a wide range of countries and locations. For the people who think "wtf is the 'Windows Store'?", it's the central place where Windows 8 users will be able to find, download and purchase applications (or as we now have to say to not look like a computer illiterate: <accent style="Kentucky">aaaaappss</accent>) for Windows 8.

As this is the store which is integrated into Windows 8, it's an interesting place for ISVs, as potential customers might very well look there first. This of course isn't true for all kinds of software, and developer tools in general aren't the kind of applications most users will download from the Windows store, but a presence there can't hurt.

Now, this Windows Store hosts two kinds of applications: 'Metro-style' applications and 'Desktop' applications. The 'Metro-style' applications are applications created for the new 'Metro' UI which is present on Windows 8 desktop and Windows RT (the single color/big font fingerpaint-oriented UI). 'Desktop' applications are the applications we all run and use on Windows today. Our software are desktop applications. The Windows Store hosts all Metro-style applications locally in the store and handles the payment for these applications. This means you upload your application (sorry, 'app') to the store, jump through a lot of hoops, Microsoft verifies that your application is not violating a tremendous long list of rules and after everything is OK, it's published and hopefully you get customers and thus earn money. Money which Microsoft will pay you on a regular basis after customers buy your application.

Desktop applications are not following this path however. Desktop applications aren't hosted by the Windows Store. Instead, the Windows Store more or less hosts a page with the application's information and where to get the goods. I.o.w.: it's nothing more than a product's Facebook page. Microsoft will simply redirect a visitor of the Windows Store to your website and the visitor will then use your site's system to purchase and download the application. This last bit of information is very important.

So, this morning I started with fresh energy to register our company 'Solutions Design bv' at the Windows Store and our two applications, LLBLGen Pro and ORM Profiler. First I went to the Windows Store dashboard page. If you don't have an account, you have to log in or sign up if you don't have a live account. I signed in with my live account. After that, it greeted me with a page where I had to fill in a code which was mailed to me. My local mail server polls every several minutes for email so I had to kick it to get it immediately.

I grabbed the code from the email and I was presented with a multi-step process to register myself as a company or as an individual. In red I was warned that this choice was permanent and not changeable. I chuckled: Microsoft apparently stores its data on paper, not in digital form. I chose 'company' and was presented with a lengthy form to fill out. On the form there were two strange remarks:

Per company there can just be 1 (one, uno, not zero, not two or more) registered developer, and only that developer is able to upload stuff to the store. I have no idea how this works with large companies, oh the overhead nightmares... "Sorry, but John, our registered developer with the Windows Store is on holiday for 3 months, backpacking through Australia, no, he's not reachable at this point. M'yeah, sorry bud. Hey, did you fill in those TPS reports yesterday?"
A separate Approver has to be specified, which has to be a different person than the registered developer. Apparently to Microsoft a company with just 1 person is not a company. Luckily we're with two people! *pfew*, dodged that one, otherwise I would be stuck forever: the choice I already made was not reversible!

After I had filled out the form and it was all well and good and accepted by the Microsoft lackey who had to write it all down in some paper notebook ("Hey, be warned! It's a permanent choice! Written down in ink, can't be changed!"), I was presented with the question how I wanted to pay for all this. "Pay for what?" I wondered. Must be the paper they were scribbling the information on, I concluded. After all, there's a financial crisis going on! How could I forget! Silly me.

"Ok fair enough".

The price was 75 Euros, not the end of the world. I could only pay by credit card, so it was accepted quickly. Or so I thought. You see, Microsoft has a different idea about CC payments. In the normal world, you type in your CC number, some date, a name and a security code and that's it. But Microsoft wants to verify this even more. They want to make a verification purchase of a very small amount and are doing that with a special code in the description. You then have to type in that code in a special form in the Windows Store dashboard and after that you're verified. Of course they'll refund the small amount they pull from your card.

Sounds simple, right? Well... no. The problem starts with the fact that I can't see the CC activity on some website: I have a bank issued CC card. I get the CC activity once a month on a piece of paper sent to me. The bank's online website doesn't show them. So it's possible I have to wait for this code till October 12th. One month.

"So what, I'm not going to use it anyway, Desktop applications don't use the payment system", I thought. "Haha, you're so naive, dear developer!" Microsoft won't allow you to publish any applications till this verification is done. So no application publishing for a month. Wouldn't it be nice if things were, you know, digital, so things got done instantly? But of course, that lackey who scribbled everything in the Big Windows Store Registration Book isn't that quick. Can't blame him though. He's just doing his job.

Now, after the payment was done, I was presented with a page which tells me Microsoft is going to use a third party company called 'Symantec', which will verify my identity again. The page explains to me that this could be done through email or phone and that they'll contact the Approver to verify my identity. "Phone?", I thought... that's a little drastic for a developer account to publish a single page of information about an external hosted software product, isn't it? On Facebook I just added a page, done. And paying you, Microsoft, took less information: you were happy to take my money before my identity was even 'verified' by this 3rd party's minions! "Double standards!", I roared. No-one cared. But it's the thought of getting it off your chest, you know.

Luckily for me, everyone at Symantec was asleep when I was registering so they went for the fallback option in case phone calls were not possible: my Approver received an email. Imagine you have to explain the idiot web of security theater I was caught in to someone else who then has to reply a random person over the internet that I indeed was who I said I was. As she's a true sweetheart, she gave me the benefit of the doubt and assured that for now, I was who I said I was.

Remember, this is for a desktop application, which is only a link to a website, some pictures and a piece of text. No file hosting, no payment processing, nothing, just a single page. Yeah, I also thought I was crazy. But we're not at the end of this quest yet.

I clicked around in the confusing menus of the Windows Store dashboard and found the 'Desktop' section. I get a helpful screen with a warning in red that it can't find any certified 'apps'. True, I'm just getting started, buddy. I see a link: "Check the Windows apps you submitted for certification". Well, I haven't submitted anything, but let's see where it brings me. Oh the thrill of adventure!

I click the link and I end up on this site: the hardware/desktop dashboard account registration. "Erm... but I just registered...", I mumbled to no-one in particular. Apparently for desktop registration / verification I have to register again, it tells me. But not only that, the desktop application has to be signed with a certificate. And not just some random el-cheapo certificate you can get at any mall's discount store. No, this certificate is special. It's precious. This certificate, the 'Microsoft Authenticode' Digital Certificate, is the only certificate that's acceptable, and jolly, it can be purchased from VeriSign for the price of only ... $99.-, but be quick, because this is a limited time offer! After that it's, I kid you not, $499.-. 500 dollars for a certificate to sign an executable. But, I do feel special, I got a special price. Only for me! I'm glowing. Not for long though.

Here I started to wonder, what the benefit of it all was. I now again had to pay money for a shiny certificate which will add 'Solutions Design bv' to our installer as the publisher instead of 'unknown', while our customers download the file from our website. Not only that, but this was all about a Desktop application, which wasn't hosted by Microsoft. They only link to it. And make no mistake. These prices aren't single payments. Every year these have to be renewed. Like a membership of an exclusive club: you're special and privileged, but only if you cough up the dough.

To give you an example how silly this all is: I added LLBLGen Pro and ORM Profiler to the Visual Studio Gallery some time ago. It's the same thing: it's a central place where one can find software which adds to / extends / works with Visual Studio. I could simply create the pages, add the information and they show up inside Visual Studio. No files are hosted at Microsoft, they're downloaded from our website. Exactly the same system.

As I have to wait for the CC transcripts to arrive anyway, I can't proceed with publishing in this new shiny store. After the verification is complete I have to wait for verification of my software by Microsoft. Even Desktop applications need to be verified using a long list of rules which are mainly focused on Metro-style applications. Even while they're not hosted by Microsoft. I wonder what they'll find. "Your application wasn't approved. It violates rule 14 X sub D: it provides more value than our own competing framework".

While I was writing this post, I tried to check something in the Windows Store Dashboard, to see whether I remembered it correctly. I was presented again with the question, after logging in with my live account, to enter the code that was just mailed to me. Not the previous code, a brand new one. Again I had to kick my mail server to pull the email to proceed. This was it. This 'experience' is so beyond miserable, I'm afraid I have to say goodbye for now to the 'Windows Store'. It's simply not worth my time.

Now, about live accounts. You might know this: live accounts are tied to everything you do with Microsoft. So if you have an MSDN subscription, e.g. the one which costs over $5000.-, it's tied to this same live account. But the fun thing is, you can login with your live account to the MSDN subscriptions with just the account id and password. No additional code is mailed to you. While it gives you access to all Microsoft software available, including your licenses.

Why the draconian security theater with this Windows Store, while all I want is to publish some desktop applications while on other Microsoft sites it's OK to simply sign in with your live account: no codes needed, no verification and no certificates? Microsoft, one thing you need with this store and that's: apps. Apps, apps, apps, apps, aaaaaaaaapps. Sorry, my bad, got carried away. I just can't stand the word 'app'. This store's shelves have to be filled to the brim with goods. But instead of being welcomed into the store with open arms, I have to fight an uphill battle with an endless list of rules and bullshit to earn the privilege to publish in this shiny store. As if I have to be thrilled to be one of the exclusive club called 'Windows Store Publishers'. As if Microsoft doesn't want it to succeed.

Craig Stuntz sent me a link to an old blog post of his regarding code signing and uploading to Microsoft's old mobile store from back in the WinMo5 days: http://blogs.teamb.com/craigstuntz/2006/10/11/28357/. Good read and good background info about how little things changed over the years.

I hope this helps Microsoft make things more clearer and smoother and also helps ISVs with their decision whether to go with the Windows Store scheme or ignore it. For now, I don't see the advantage of publishing there, especially not with the nonsense rules Microsoft cooked up. Perhaps it changes in the future, who knows.

LLBLGen Pro feature highlights: model views

FransBouma — Fri, 31 Aug 2012 10:38:11 GMT

(This post is part of a series of posts about features of the LLBLGen Pro system)

To be able to work with large(r) models, it's key you can view subsets of these models so you can have a better, more focused look at them. For example because you want to display how a subset of entities relate to one another in a different way than the list of entities. LLBLGen Pro offers this in the form of Model Views. Model Views are views on parts of the entity model of a project, and the subsets are displayed in a graphical way.

Additionally, one can add documentation to a Model View. As Model Views are displaying parts of the model in a graphical way, they're easier to explain to people who aren't familiar with entity models, e.g. the stakeholders you're interviewing for your project. The documentation can then be used to communicate specifics of the elements on the model view to the developers who have to write the actual code.

Below I've included an example. It's a model view on a subset of the entities of AdventureWorks. It displays several entities, their relationships (both relational and inheritance relationships) and also some specifics gathered from the interview with the stakeholder. As the information is inside the actual project the developer will work with, the information doesn't have to be converted back/from e.g .word documents or other intermediate formats, it's the same project. This makes sure there are less errors / misunderstandings.

(of course you can hide the docked documentation pane or dock it to another corner).

The Model View can contain entities which are placed in different groups. This makes it ideal to group entities together for close examination even though they're stored in different groups.

The Model View is a first-class citizen of the code-generator. This means you can write templates which consume Model Views and generate code accordingly. E.g. you can write a template which generates a service per Model View and exposes the entities in the Model View as a single entity graph, fetched through a method. (This template isn't included in the LLBLGen Pro package, but it's easy to write it up yourself with the built-in template editor).

Viewing an entity model in different ways is key to fully understand the entity model and Model Views help with that.

Beware when using .NET's named pipes in a windows forms application

FransBouma — Tue, 28 Aug 2012 17:08:47 GMT

Yesterday a user of our .net ORM Profiler tool reported that he couldn't get the snapshot recording from code feature working in a windows forms application. Snapshot recording in code means you start recording profile data from within the profiled application, and after you're done you save the snapshot as a file which you can open in the profiler UI. When using a console application it worked, but when a windows forms application was used, the snapshot was always empty: nothing was recorded.

Obviously, I wondered why that was, and debugged a little. Here's an example piece of code to record the snapshot. This piece of code works OK in a console application, but results in an empty snapshot in a windows forms application:

var snapshot = new Snapshot();
snapshot.Record();
using(var ctx = new ORMProfilerTestDataContext())
{
	var customers = ctx.Customers.Where(c => c.Country == "USA").ToList();
}
InterceptorCore.Flush();
snapshot.Stop();
string error=string.Empty;
if(!snapshot.IsEmpty)
{
	snapshot.SaveToFile(@"c:\temp\generatortest\test2\blaat.opsnapshot", out error);
}
if(!string.IsNullOrEmpty(error))
{
	Console.WriteLine("Save error: {0}", error);
}

(the Console.WriteLine doesn't do anything in a windows forms application, but you get the idea).

ORM Profiler uses named pipes: the interceptor (referenced and initialized in your application, the application to profile) sends data over the named pipe to a listener, which when receiving a piece of data begins reading it, asynchronically, and when properly read, it will signal observers that new data has arrived so they can store it in a repository. In this case, the snapshot will be the observer and will store the data in its own repository.

The reason the above code doesn't work in windows forms is because windows forms is a wrapper around Win32 and its WM_* message based system. Named pipes in .NET are wrappers around Windows named pipes which also work with WM_* messages. Even though we use BeginRead() on the named pipe (which spawns a thread to read the data from the named pipe), nothing is received by the named pipe in the windows forms application, because it doesn't handle the WM_* messages in its message queue till after the method is over, as the message pump of a windows forms application is handled by the only thread of the windows forms application, so it will handle WM_* messages when the application idles.

The fix is easy though: add Application.DoEvents(); right before snapshot.Stop(). Application.DoEvents() forces the windows forms application to process all WM_* messages in its message queue at that moment: all messages for the named pipe are then handled, the .NET code of the named pipe wrapper will react on that and the whole process will complete as if nothing happened.

It's not that simple to just say 'why didn't you use a worker thread to create the snapshot here?', because a thread doesn't get its own message pump: the messages would still be posted to the window's message pump. A hidden form would create its own message pump, so the additional thread should also create a window to get the WM_* messages of the named pipe posted to a different message pump than the one of the main window.

This WM_* messages pain is not something you want to be confronted with when using .NET and its libraries. Unfortunately, the way they're implemented, a lot of APIs are leaky abstractions, they bleed the characteristics of the OS objects they hide away through to the .NET code. Be aware of that fact when using them :)

LLBLGen Pro feature highlights: grouping model elements

FransBouma — Wed, 20 Jun 2012 10:59:02 GMT

(This post is part of a series of posts about features of the LLBLGen Pro system)

When working with an entity model which has more than a few entities, it's often convenient to be able to group entities together if they belong to a semantic sub-model. For example, if your entity model has several entities which are about 'security', it would be practical to group them together under the 'security' moniker. This way, you could easily find them back, yet they can be left inside the complete entity model altogether so their relationships with entities outside the group are kept.

In other situations your domain consists of semi-separate entity models which all target tables/views which are located in the same database. It then might be convenient to have a single project to manage the complete target database, yet have the entity models separate of each other and have them result in separate code bases.

LLBLGen Pro can do both for you. This blog post will illustrate both situations. The feature is called group usage and is controllable through the project settings. This setting is supported on all supported O/R mapper frameworks.

Situation one: grouping entities in a single model.

This situation is common for entity models which are dense, so many relationships exist between all sub-models: you can't split them up easily into separate models (nor do you likely want to), however it's convenient to have them grouped together into groups inside the entity model at the project level. A typical example for this is the AdventureWorks example database for SQL Server. This database, which is a single catalog, has for each sub-group a schema, however most of these schemas are tightly connected with each other: adding all schemas together will give a model with entities which indirectly are related to all other entities. LLBLGen Pro's default setting for group usage is AsVisualGroupingMechanism which is what this situation is all about: we group the elements for visual purposes, it has no real meaning for the model nor the code generated.

Let's reverse engineer AdventureWorks to an entity model. By default, LLBLGen Pro uses the target schema an element is in which is being reverse engineered, as the group it will be in. This is convenient if you already have categorized tables/views in schemas, like which is the case in AdventureWorks. Of course this can be switched off, or corrected on the fly.

When reverse engineering, we'll walk through a wizard which will guide us with the selection of the elements which relational model data should be retrieved, which we can later on use to reverse engineer to an entity model. The first step after specifying which database server connect to is to select these elements. below we can see the AdventureWorks catalog as well as the different schemas it contains. We'll include all of them.

After the wizard completes, we have all relational model data nicely in our catalog data, with schemas. So let's reverse engineer entities from the tables in these schemas. We select in the catalog explorer the schemas 'HumanResources', 'Person', 'Production', 'Purchasing' and 'Sales', then right-click one of them and from the context menu, we select Reverse engineer Tables to Entity Definitions.... This will bring up the dialog below. We check all checkboxes in one go by checking the checkbox at the top to mark them all to be added to the project.

As you can see LLBLGen Pro has already filled in the group name based on the schema name, as this is the default and we didn't change the setting. If you want, you can select multiple rows at once and set the group name to something else using the controls on the dialog. We're fine with the group names chosen so we'll simply click Add to Project. This gives the following result:

(I collapsed the other groups to keep the picture small ;)). As you can see, the entities are now grouped. Just to see how dense this model is, I've expanded the relationships of Employee:

As you can see, it has relationships with entities from three other groups than HumanResources. It's not doable to cut up this project into sub-models without duplicating the Employee entity in all those groups, so this model is better suited to be used as a single model resulting in a single code base, however it benefits greatly from having its entities grouped into separate groups at the project level, to make work done on the model easier.

Now let's look at another situation, namely where we work with a single database while we want to have multiple models and for each model a separate code base.

Situation two: grouping entities in separate models within the same project.

To get rid of the entities to see the second situation in action, simply undo the reverse engineering action in the project. We still have the AdventureWorks relational model data in the catalog. To switch LLBLGen Pro to see each group in the project as a separate project, open the Project Settings, navigate to General and set Group usage to AsSeparateProjects.

In the catalog explorer, select Person and Production, right-click them and select again Reverse engineer Tables to Entities.... Again check the checkbox at the top to mark all entities to be added and click Add to Project. We get two groups, as expected, however this time the groups are seen as separate projects. This means that the validation logic inside LLBLGen Pro will see it as an error if there's e.g. a relationship or an inheritance edge linking two groups together, as that would lead to a cyclic reference in the code bases.

To see this variant of the grouping feature, seeing the groups as separate projects, in action, we'll generate code from the project with the two groups we just created: select from the main menu: Project -> Generate Source-code... (or press F7 ;)). In the dialog popping up, select the target .NET framework you want to use, the template preset, fill in a destination folder and click Start Generator (normal). This will start the code generator process.

As expected the code generator has simply generated two code bases, one for Person and one for Production:

The group name is used inside the namespace for the different elements. This allows you to add both code bases to a single solution and use them together in a different project without problems. Below is a snippet from the code file of a generated entity class.

//...
using System.Xml.Serialization;
using AdventureWorks.Person;
using AdventureWorks.Person.HelperClasses;
using AdventureWorks.Person.FactoryClasses;
using AdventureWorks.Person.RelationClasses;
using SD.LLBLGen.Pro.ORMSupportClasses;

namespace AdventureWorks.Person.EntityClasses
{    
//...
	/// <summary>Entity class which represents the entity 'Address'.<br/><br/></summary>
	[Serializable]    
	public partial class AddressEntity : CommonEntityBase
//...

The advantage of this is that you can have two code bases and work with them separately, yet have a single target database and maintain everything in a single location. If you decide to move to a single code base, you can do so with a change of one setting. It's also useful if you want to keep the groups as separate models (and code bases) yet want to add relationships to elements from another group using a copy of the entity: you can simply reverse engineer the target table to a new entity into a different group, effectively making a copy of the entity. As there's a single target database, changes made to that database are reflected in both models which makes maintenance easier than when you'd have a separate project for each group, with its own relational model data.

Conclusion

LLBLGen Pro offers a flexible way to work with entities in sub-models and control how the sub-models end up in the generated code.

Azure, don't give me multiple VMs, give me one elastic VM

FransBouma — Fri, 08 Jun 2012 09:51:54 GMT

Yesterday, Microsoft revealed new major features for Windows Azure (see ScottGu's post). It all looks shiny and great, but after reading most of the material describing the new features, I still find the overall idea behind all of it flawed: why should I care on how much VMs my web app runs? Isn't that a problem to solve for the Windows Azure engineers / software? And what if I need the file system, why can't I simply get a virtual filesystem ?

To illustrate my point, let's use a real example: a product website with a customer system/database and next to it a support site with accompanying database. Both are written in .NET, using ASP.NET and use a SQL Server database each. The product website offers files to download by customers, very simple.

You have a couple of options to host these websites:

Buy a server, place it in a rack at an ISP and run the sites on that server
Use 'shared hosting' with an ISP, which means your sites' appdomains are running on the same machine, as well as the files stored, and the databases are hosted in the same server as the other shared databases.
Hire a VM, install your OS of choice at an ISP, and host the sites on that VM, basically the same as the first option, except you don't have a physical server
At some cloud-vendor, either host the sites 'shared' or in a VM. See above.

With all of those options, scalability is a problem, even the cloud-based ones, though not due to the same reasons:

The physical server solution has the obvious problem that if you need more power, you need to buy a bigger server or more servers which requires you to add replication and other overhead
Shared hosting solutions are almost always capped on memory usage / traffic and database size: if your sites get too big, you have to move out of the shared hosting environment and start over with one of the other solutions
The VM solution, be it a VM at an ISP or 'in the cloud' at e.g. Windows Azure or Amazon, in theory allows scaling out by simply instantiating more VMs, however that too introduces the same overhead problems as with the physical servers: suddenly more than 1 instance runs your sites.

If a cloud vendor offers its services in the form of VMs, you won't gain much over having a VM at some ISP: the main problems you have to work around are still there: when you spin up more than one VM, your application must be completely stateless at any moment, including the DB sub system, because what's in memory in instance 1 might not be in memory in instance 2.

This might sounds trivial but it's not. A lot of the websites out there started rather small: they were perfectly runnable on a single machine with normal memory and CPU power. After all, you don't need a big machine to run a website with even thousands of users a day. Moving these sites to a multi-VM environment will cause a problem: all the in-memory state they use, all the multi-page transitions they use while keeping state across the transition, they can't do that anymore like they did that on a single machine: state is something of the past, you have to store every byte of state in either a DB or in a viewstate or in a cookie somewhere so with the next request, all state information is available through the request, as nothing is kept in-memory.

Our example uses a bunch of files in a file system. Using multiple VMs will require that these files move to a cloud storage system which is mounted in each VM so we don't have to store the files on each VM. This might require different file paths, but this change should be minor. What's perhaps less minor is the maintenance procedure in place on the new type of cloud storage used: instead of ftp-ing into a VM, you might have to update the files using different ways / tools.

All in all this makes moving an existing website which was written for an environment that's based around a VM (namely .NET with its CLR) overly cumbersome and problematic: it forces you to refactor your website system to be able to be used 'in the cloud', which is caused by the limited way how e.g. Windows Azure offers its cloud services: in blocks of VMs.

Offer a scalable, flexible VM which extends with my needs

Instead, cloud vendors should offer simply one VM to me. On that VM I run the websites, store my DB and my files. As it's a virtual machine, how this machine is actually ran on physical hardware (e.g. partitioned), I don't care, as that's the problem for the cloud vendor to solve. If I need more resources, e.g. I have more traffic to my server, way more visitors per day, the VM stretches, like I bought a bigger box. This frees me from the problem which comes with multiple VMs: I don't have any refactoring to do at all: I can simply build my website as if it runs on my local hardware server, upload it to the VM offered by the cloud vendor, install it on the VM and I'm done.

"But that might require changes to windows!"
Yes, but Microsoft is Windows. Windows Azure is their service, they can make whatever change to what they offer to make it look like it's windows. Yet, they're stuck, like Amazon, in thinking in VMs, which forces developers to 'think ahead' and gamble whether they would need to migrate to a cloud with multiple VMs in the future or not. Which comes down to: gamble whether they should invest time in code / architecture which they might never need. (YAGNI anyone?)

So the VM we're talking about, is that a low-level VM which runs a guest OS, or is that VM a different kind of VM?

The flexible VM: .NET's CLR ?

My example websites are ASP.NET based, which means they run inside a .NET appdomain, on the .NET CLR, which is a VM. The only physical OS resource the sites need is the file system, however this too is accessed through .NET. In short: all the websites see is what .NET allows the websites to see, the world as the websites know it is what .NET shows them and lets them access. How the .NET appdomain is run physically, that's the concern of .NET, not mine. This begs the question why Windows Azure doesn't offer virtual appdomains? Or better: .NET environments which look like one machine but could be physically multiple machines.

In such an environment, no change has to be made to the websites to migrate them from a local machine or own server to the cloud to get proper scaling: the .NET VM will simply scale with the need: more memory needed, more CPU power needed, it stretches. What it offers to the application running inside the appdomain is simply increasing, but not fragmented: all resources are available to the application: this means that the problem of how to scale is back to where it should be: with the cloud vendor.

"Yeah, great, but what about the databases?"
The .NET application communicates with the database server through a .NET ADO.NET provider. Where the database is located is not a problem of the appdomain: the ADO.NET provider has to solve that. I.o.w.: we can host the databases in an environment which offers itself as a single resource and is accessible through one connection string without replication overhead on the outside, and use that environment inside the .NET VM as if it was a single DB.

But what about memory replication and other problems?

This environment isn't simple, at least not for the cloud vendor. But it is simple for the customer who wants to run his sites in that cloud: no work needed. No refactoring needed of existing code. Upload it, run it.

Perhaps I'm dreaming and what I described above isn't possible. Yet, I think if cloud vendors don't move into that direction, what they're offering isn't interesting: it doesn't solve a problem at all, it simply offers a way to instantiate more VMs with the guest OS of choice at the cost of me needing to refactor my website code so it can run in the straight jacket form factor dictated by the cloud vendor.

Let's not kid ourselves here: most of us developers will never build a website which needs a truck load of VMs to run it: almost all websites created by developers can run on just a few VMs at most. Yet, the most expensive change is right at the start: moving from one to two VMs. As soon as you have refactored your website code to run across multiple VMs, adding another one is just as easy as clicking a mouse button. But that first step, that's the problem here and as it's right there at the beginning of scaling the website, it's particularly strange that cloud vendors refuse to solve that problem and leave it to the developers to solve that.

Which makes migrating 'to the cloud' particularly expensive.

LLBLGen Pro feature highlights: automatic element name construction

FransBouma — Wed, 06 Jun 2012 13:27:00 GMT

(This post is part of a series of posts about features of the LLBLGen Pro system)

One of the things one might take for granted but which has a huge impact on the time spent in an entity modeling environment is the way the system creates names for elements out of the information provided, in short: automatic element name construction. Element names are created in both directions of modeling: database first and model first and the more names the system can create for you without you having to rename them, the better. LLBLGen Pro has a rich, fine grained system for creating element names out of the meta-data available, which I'll describe more in detail below. First the model element related element naming features are highlighted, in the section Automatic model element naming features and after that I'll go into more detail about the relational model element naming features LLBLGen Pro has to offer in the section Automatic relational model element naming features.

Automatic model element naming features

When working database first, the element names in the model, e.g. entity names, entity field names and so on, are in general determined from the relational model element (e.g. table, table field) they're mapped on, as the model elements are reverse engineered from these relational model elements. It doesn't take rocket science to automatically name an entity Customer if the entity was created after reverse engineering a table named Customer. It gets a little trickier when the entity which was created by reverse engineering a table called TBL_ORDER_LINES has to be named 'OrderLine' automatically. Automatic model element naming also takes into effect with model first development, where some settings are used to provide you with a default name, e.g. in the case of navigator name creation when you create a new relationship.

The features below are available to you in the Project Settings. Open Project Settings on a loaded project and navigate to Conventions -> Element Name Construction.

Strippers!
The above example 'TBL_ORDER_LINES' shows that some parts of the table name might not be needed for name creation, in this case the 'TBL_' prefix. Some 'brilliant' DBAs even add suffixes to table names, fragments you might not want to appear in the entity names. LLBLGen Pro offers you to define both prefix and suffix fragments to strip off of table, view, stored procedure, parameter, table field and view field names. In the example above, the fragment 'TBL_' is a good candidate for such a strip pattern. You can specify more than one pattern for e.g. the table prefix strip pattern, so even a really messy schema can still be used to produce clean names.

Underscores Be Gone
Another thing you might get rid of are underscores. After all, most naming schemes for entities and their classes use PasCal casing rules and don't allow for underscores to appear. LLBLGen Pro can automatically strip out underscores for you. It's an optional feature, so if you like the underscores, you're not forced to see them go: LLBLGen Pro will leave them alone when ordered to to so.

PasCal everywhere... or not, your call
LLBLGen Pro can automatically PasCal case names on word breaks. It determines word breaks in a couple of ways: a space marks a word break, an underscore marks a word break and a case difference marks a word break. It will remove spaces in all cases, and based on the underscore removal setting, keep or remove the underscores, and upper-case the first character of a word break fragment, and lower case the rest. Say, we keep the defaults, which is remove underscores and PasCal case always and strip the TBL_ fragment, we get with our example TBL_ORDER_LINES, after stripping TBL_ from the table name two word fragments: ORDER and LINES. The underscores are removed, the first character of each fragment is upper-cased, the rest lower-cased, so this results in OrderLines. Almost there!

Pluralization and Singularization
In general entity names are singular, like Customer or OrderLine so LLBLGen Pro offers a way to singularize the names. This will convert OrderLines, the result we got after the PasCal casing functionality, into OrderLine, exactly what we're after.

Show me the patterns!
There are other situations in which you want more flexibility. Say, you have an entity Customer and an entity Order and there's a foreign key constraint defined from the target of Order and the target of Customer. This foreign key constraint results in a 1:n relationship between the entities Customer and Order. A relationship has navigators mapped onto the relationship in both entities the relationship is between. For this particular relationship we'd like to have Customer as navigator in Order and Orders as navigator in Customer, so the relationship becomes Customer.Orders 1:n Order.Customer.

To control the naming of these navigators for the various relationship types, LLBLGen Pro defines a set of patterns which allow you, using macros, to define how the auto-created navigator names will look like. For example, if you rather have Customer.OrderCollection, you can do so, by changing the pattern from {$EndEntityName$P} to {$EndEntityName}Collection. The $P directive makes sure the name is pluralized, which is not what you want if you're going for <EntityName>Collection, hence it's removed.

When working model first, it's a given you'll create foreign key fields along the way when you define relationships. For example, you've defined two entities: Customer and Order, and they have their fields setup properly. Now you want to define a relationship between them. This will automatically create a foreign key field in the Order entity, which reflects the value of the PK field in Customer. (No worries if you hate the foreign key fields in your classes, on NHibernate and EF these can be hidden in the generated code if you want to). A specific pattern is available for you to direct LLBLGen Pro how to name this foreign key field. For example, if all your entities have Id as PK field, you might want to have a different name than Id as foreign key field. In our Customer - Order example, you might want to have CustomerId instead as foreign key name in Order. The pattern for foreign key fields gives you that freedom.

Abbreviations... make sense of OrdNr and friends
I already described word breaks in the PasCal casing paragraph, how they're used for the PasCal casing in the constructed name. Word breaks are used for another neat feature LLBLGen Pro has to offer: abbreviation support. Burt, your friendly DBA in the dungeons below the office has a hate-hate relationship with his keyboard: he can't stand it: typing is something he avoids like the plague. This has resulted in tables and fields which have names which are very short, but also very unreadable. Example: our TBL_ORDER_LINES example has a lovely field called ORD_NR. What you would like to see in your fancy new OrderLine entity mapped onto this table is a field called OrderNumber, not a field called OrdNr. What you also like is to not have to rename that field manually. There are better things to do with your time, after all.

LLBLGen Pro has you covered. All it takes is to define some abbreviation - full word pairs and during reverse engineering model elements from tables/views, LLBLGen Pro will take care of the rest. For the ORD_NR field, you need two values: ORD as abbreviation and Order as full word, and NR as abbreviation and Number as full word. LLBLGen Pro will now convert every word fragment found with the word breaks which matches an abbreviation to the given full word. They're case sensitive and can be found in the Project Settings: Navigate to Conventions -> Element Name Construction -> Abbreviations.

Automatic relational model element naming features

Not everyone works database first: it may very well be the case you start from scratch, or have to add additional tables to an existing database. For these situations, it's key you have the flexibility that you can control the created table names and table fields without any work: let the designer create these names based on the entity model you defined and a set of rules. LLBLGen Pro offers several features in this area, which are described in more detail below. These features are found in Project Settings: navigate to Conventions -> Model First Development.

Underscores, welcome back!
Not every database is case insensitive, and not every organization requires PasCal cased table/field names, some demand all lower or all uppercase names with underscores at word breaks. Say you create an entity model with an entity called OrderLine. You work with Oracle and your organization requires underscores at word breaks: a table created from OrderLine should be called ORDER_LINE. LLBLGen Pro allows you to do that: with a simple checkbox you can order LLBLGen Pro to insert an underscore at each word break for the type of database you're working with: case sensitive or case insensitive. Checking the checkbox Insert underscore at word break case insensitive dbs will let LLBLGen Pro create a table from the entity called Order_Line. Half-way there, as there are still lower case characters there and you need all caps. No worries, see below

Casing directives so everyone can sleep well at night
For case sensitive databases and case insensitive databases there is one setting for each of them which controls the casing of the name created from a model element (e.g. a table created from an entity definition using the auto-mapping feature). The settings can have the following values: AsProjectElement, AllUpperCase or AllLowerCase. AsProjectElement is the default, and it keeps the casing as-is. In our example, we need to get all upper case characters, so we select AllUpperCase for the setting for case sensitive databases. This will produce the name ORDER_LINE.

Sequence naming after a pattern
Some databases support sequences, and using model-first development it's key to have sequences, when needed, to be created automatically and if possible using a name which shows where they're used. Say you have an entity Order and you want to have the PK values be created by the database using a sequence. The database you're using supports sequences (e.g. Oracle) and as you want all numeric PK fields to be sequenced, you have enabled this by the setting Auto assign sequences to integer pks. When you're using LLBLGen Pro's auto-map feature, to create new tables and constraints from the model, it will create a new table, ORDER, based on your settings I previously discussed above, with a PK field ID and it also creates a sequence, SEQ_ORDER, which is auto-assigns to the ID field mapping. The name of the sequence is created by using a pattern, defined in the Model First Development setting Sequence pattern, which uses plain text and macros like with the other patterns previously discussed.

Grouping and schemas
When you start from scratch, and you're working model first, the tables created by LLBLGen Pro will be in a catalog and / or schema created by LLBLGen Pro as well. If you use LLBLGen Pro's grouping feature, which allows you to group entities and other model elements into groups in the project (described in a future blog post), you might want to have that group name reflected in the schema name the targets of the model elements are in. Say you have a model with a group CRM and a group HRM, both with entities unique for these groups, e.g. Employee in HRM, Customer in CRM. When auto-mapping this model to create tables, you might want to have the table created for Employee in the HRM schema but the table created for Customer in the CRM schema. LLBLGen Pro will do just that when you check the setting Set schema name after group name to true (default). This gives you total control over where what is placed in the database from your model.

But I want plural table names... and TBL_ prefixes!
For now we follow best practices which suggest singular table names and no prefixes/suffixes for names. Of course that won't keep everyone happy, so we're looking into making it possible to have that in a future version.

Conclusion

LLBLGen Pro offers a variety of options to let the modeling system do as much work for you as possible. Hopefully you enjoyed this little highlight post and that it has given you new insights in the smaller features available to you in LLBLGen Pro, ones you might not have thought off in the first place. Enjoy!

ORM Profiler v1.1 has been released!

FransBouma — Mon, 04 Jun 2012 09:43:24 GMT

We've released ORM Profiler v1.1, which has the following new features:

Real time profiling

A real time viewer (RTV) has been added, which gives insight in the activity as it is received by the client, in two views: a chronological connection overview and an activity graph overview. This RTV allows the user to directly record to a snapshot using record buttons, pause the view, mark a range to create a snapshot from that range, and view graphs about the # of connection open actions and # of commands per second. The RTV has a 'range' in which it keeps live data and auto-cleans data that's older than this range.

Screenshot of the activity graphs part of the real-time viewer:

Low-level activity tab

A new tab has been added to the Application tabs: the Low-level activity tab. This tab shows the main activity as it has been received over the named pipe. It can help to get insight in the chronological activity without the grouping over connections, so multiple connections at the same time per thread are easier to spot. Clicking a command will sync the rest of the application tabs, clicking a row will show the details below the splitter bar, as it is done with the other application tabs as well.

Default application name in interceptor

When an empty string or null is passed for application name to the Initialize method of the interceptor, the AppDomain's friendly name is used instead.

Copy call stack to clipboard

A call stack viewed in a grid in various parts of the UI is now copyable to the clipboard by clicking a button.

Enable/Disable interceptor from the config file

It's now possible to enable/disable the interceptor Initialization from the application's config file, using:

Code:

<appSettings>
	<add key="ORMProfilerEnabled" value="true"/>
</appSettings>

if value is true, the interceptor's Initialize method will proceed. If the value is false, the interceptor's Initialize method will not proceed and initialization won't be performed, meaning no interception will take place. If the setting is absent, or misconfigured, the Initialize method will proceed as normal and perform the initialization.

Stored procedure calls for select databases are now properly displayed as a call

For the databases: SQL Server, Oracle, DB2, Sybase ASA, Sybase ASE and Informix a stored procedure call is displayed as an execute/call statement and copy to clipboard works as-is.

I'm especially happy with the new real-time profiling feature in ORM Profiler, which is the flagship feature for this release: it offers a completely new way to use the profiler, namely directly during debugging: you can immediately see what's going on without the necessity of a snapshot. The activity graph feature combined with the auto-cleanup of older data, allows you to keep the profiler open for a long period of time and see any spike of activity on the profiled application.