Frans Bouma's blog

WCF Data Services and the IExpandProvider

FransBouma — Sun, 02 Jun 2013 10:22:58 GMT

Yesterday, a customer ran into a weird issue with our OData/WCF Data Services support in LLBLGen Pro: when an $expand directive was given which was more than one level deep, the service would only return the first level. $expand is OData's directive to eager load additional data into the data requested. So if I for example want to read the data of customer 'ALFKI' from the Northwind database and also that customer's orders, I'd issue this OData query:

http://localhost:3931/NorthwindService.svc/Customer('ALFKI')?$expand=Orders

This will return the data of ALFKI and the data of all Orders of that customer. This all worked fine. However, when I also would specify to load all OrderDetail entities for each order (so i.o.w.: a second level expand), it would not return these OrderDetail rows, but simply return the same data as the query above:

http://localhost:3931/NorthwindService.svc/Customer('ALFKI')?$expand=Orders/OrderDetails

This could be caused by a lot of things of course, but as it has worked in the past and it didn't work today, what did change? First I verified with ORM Profiler that the engine indeed executed three queries: one for Customer with a filter on CustomerID, one on Orders with the same predicate and one on OrderDetails with a filter on all OrderIds returned by the Orders query, a typical three node prefetch path query. Debugging the engine revealed that the expand directives were properly converted to a prefetch path, and the data was properly merged into a single graph of Customer – Orders – OrderDetails.

This frustrated me to no end, because my code did what it was told to do, it handed over the proper entity instance graph to the calling service code and apparently something went wrong there, but what and why was a mystery. Not only that, but who to ask what's wrong and if something was wrong on Microsoft's end, would they ever fix it? Writing the WCF Data Services support code for our O/R mapper framework was a frustrating experience as there's little documentation available how to add WCF Data Services support for your ORM/Data access technology (as everything about WCF Data Services is tightly designed around Entity Framework) so you have to piece together what you have to do from blog-posts, simple examples and decompiling (sorry) the Entity Framework WCF Data Services provider.

First rule of fixing issues when writing code to support a Microsoft framework is to look whether you're the only one having this problem. I was. Well, I did find a question on stackoverflow which asked what to do about the fact that IExpandProvider interface was deprecated. This refreshed my memory on it a bit. IExpandProvider is the interface which is to be implemented by the class which handles $expand fragments. That is, it was. Microsoft deprecated this interface some time ago but if your WCF Data Services service implemented the interface, it would still work as if nothing happened. For LLBLGen Pro v4 we moved our WCF Data Services code to the nuget version of the WCF Data Services code instead of the one shipped with .NET 4/4.5. Everything worked properly, so we signed it off. But we didn't have a test with two-level expands, so we missed an important aspect: it apparently seems the case the v5.x version of WCF Data Services doesn't return multiple levels of data fetched with $expand, if the service provides an IExpandProvider implementing object.

Reading the reply on the stackoverflow question by Vitek Karas, one of the WCF Data Services team members, I ended up on a blogpost which went on and on about expand wrappers, projection wrappers and what not, as an alternative to IExpandProvider implementing classes. It was unclear to me if I had to do that all myself or not: this is typical for what I had to wade through when writing the original WCF Data Services support code: there's no real documentation for this. As it looked rather daunting, and I have the feeling the guy who posted the question on stackoverflow thought the same, I had enough: I just want to provide top-quality code to my customers, but this endless stream of frustration made it impossible for me to do so.

I can understand why Microsoft has less eye for us, 3rd party ORM developers: we're a small group, we compete with their ORM framework which is there to keep people on .NET/MS platforms, Microsoft is still a product-oriented company and so other things might have higher priority. That doesn't make it less frustrating though.

After a good night sleep I woke up this morning with a daft thought in my mind: what if I simply told the service there was no IExpandProvider implemented? What would the service do? So I changed my WCF Data Services support code to simply return null if it was asked to return an IExpandProvider implementation and re-ran the tests. They worked! All levels of data were returned, and the queries were still efficiently done. But how? After all, there was no prefetch path handler code active anymore, and the prefetch path API is not the same as Entity Framework's (ours can do filters on included data, sort, exclude fields etc.).

If there's no IExpandProvider, the WCF Data Services creates a custom projection with the expand fragments as nested queries. I described them here more in detail. Our Linq provider recognizes these nested queries and performs a similar procedure as if it would do with prefetch paths: each nested query is executed as a separate query and the results are merged using hash maps in-memory with parent nodes and nested queries are filtered based on the data fetched in parent nodes. This means the engine will still run three queries on the Customer – Orders – OrderDetails expand path, though it will use the nested query pipeline in the linq provider (which can deal with nested custom projections as well as nested entity sets).

The fix was simple for my code: comment out some lines and everything was fine. But as I commented on Vitek's stackoverflow reply: what if the ORM has a proper eager-loading pipeline? It now has to rely on nested queries, something which isn't done properly in e.g. Linq to Sql (they're loaded in a SELECT N+1 fashion), unless a lot of work is done to make this load properly. You can't simply mark a working system 'obsolete', and expect frameworks supporting your service to say 'ok, no problem, we'll invest again some time and write some more code'.

There are two scenario's for fetching additional data: fetch additional entities with the entity loaded, or fetch a custom projection of a set of nested sets, because if you fetch an entity E, where in E do you store the custom projection of some related data? Why not keep IExpandProvider around for the scenario where you fetch entities and related entities, the normal eager load scenario? If one needs $expand to fetch a custom projection with nested sets, why not use the mechanism that's available now to replace IExpandProvider's logic? Just to support custom selects on entities to exclude fields for Entity Framework? Isn't it then better to implement an exclude/include facility in EF for that?

Anyway, it's solved now and I'm glad this scenario has a happy end as I didn't expect it would. I hope it helps some poor souls out there who are faced with the same problem and I also hope some people within Microsoft do realize that if you make it difficult for 3rd party frameworks to support your frameworks, they might stop doing so.

LLBLGen Pro v4.0 feature highlights: Resultset Caching

FransBouma — Wed, 29 May 2013 10:08:59 GMT

This is the second post of a series about the new features in LLBLGen Pro v4.0, which was released on April 6th, 2013. Today I'd like to highlight a new major feature of the LLBLGen Pro Runtime Framework, our own ORM framework shipped with LLBLGen Pro, namely Resultset Caching.

I have written about caching in ORMs before, though that particular way of caching was about materialized objects, which is something else than Resultset Caching. Resultset Caching is a mechanism which caches the actual resultset used during the execution of a select query for a given duration. This means that if a given Linq / QuerySpec query is specified to have its resultset cached for e.g. 10 seconds, every execution of the exact same query with the same parameters will pull the results from the cache for the 10 seconds after the query was first executed on the database.

Caching can lead to better performance in situations where a query is taking a relatively long time in the database and/or is executed a lot of times: by automatically re-using already read rows from the cache instead of from the database, a round trip + query execution is omitted. This is completely transparent and automatic: the only thing the developer has to do is to specify with a query whether it should cache its resultset and for how long.

The caching system is opt-in per query, so by default, all queries are executed on the database, every time, and no caching takes place. This is ideal because caching resultsets means that the resultset of a query which is defined to cache its results might pull its resultset from the cache instead of the database and therefore it might be the data is out-of-date with the data in the database: the cache data is 'stale' data, but for many situations this is ok, at least for a short period of time.

Resultset caching is done through cache objects which implement the caching of resultsets, either by delegating it to 3rd party servers like the .NET system cache or e.g. Memcached, or by doing the caching themselves like the built-in ResultsetCache class. Cache objects implement a common interface, IResultsetCache , and it's easy to implement a cache object class yourself. With the LLBLGen Pro Runtime Framework we ship the built-in ResultsetCache class and in the recently started LLBLGen Pro contrib library (github) we ship an implementation of the IResultsetCache interface for caching data in the .NET 4+ cache MemoryCache. As you can see with the MemoryCache supporting implementation, it's very easy to add a class which caches the data in your cache system of choice, e.g. Redis.

Registering a cache

Before caching can take place, at least one cache has to be registered with the CacheController. The CacheController is a static class which keeps track of the registered caches and is consulted by the query object whether a cached set of rows is available for that query.

Example of registering an instance of the ResultsetCache:

CacheController.RegisterCache(connectionString, new ResultsetCache());

After this line, all queries executed over a connection with the same connection string and which are specified to have their resultsets cached, are cached in the ResultsetCache instance registered. This means you can have multiple caches if you have multiple databases in use at the same time. You can also use one cache for all databases in use, by specifying an empty string as the connection string.

What makes a query unique? The Cache Key

Resultsets are cached in a cache under a key, a CacheKey instance. A CacheKey is created from a query object which provides the values for the key.

A CacheKey uses:

The SQL statement
All parameter values before execution
Flags for in-memory distinct/limiting/paging
Values for rows to skip/take in in-memory paging scenarios

A hashcode is calculated from these values and an Equals implementation makes sure two CacheKey instances are only considered equal if all of these values are indeed equal. This makes sure that two queries which differ only in e.g. parameter values will not return each other's resultsets from the cache. Two pieces of code which are fetching data differently (e.g. a normal entity collection fetch and a projection using all entity fields onto a custom class) but result in the same SQL query and parameters will re-use the same set from the cache, if both queries are defined to be cacheable.

Specifying a query to have its resultset cached

LLBLGen Pro has three ways to specify a query: Linq, QuerySpec and the Low-level API. We primarily added caching specification code to Linq and QuerySpec, as those two query APIs are the recommended way to formulate queries. I'll show how to specify caching directives in Linq and QuerySpec below.

QuerySpec

// typed list fetch 
var tl = new OrderCustomerTypedList();
var q = tl.GetQuerySpecQuery(new QueryFactory())
                 .CacheResultset(10);  // cache for 10 seconds
new DataAccessAdapter().FetchAsDataTable(q, tl);

// typed view fetch
var qf = new QueryFactory();
var q = qf.Create()
            .Select(InvoicesFields.CustomerId, InvoicesFields.CustomerName, InvoicesFields.OrderId)
            .Where(InvoicesFields.Country.StartsWith("U"))
            .CacheResultset(10);        // cache for 10 seconds
var tv = new InvoicesTypedView();
new DataAccessAdapter().FetchAsDataTable(q, tv);

// dynamic list / custom projection
var qf = new QueryFactory();
var q = qf.Employee
        .From(QueryTarget.InnerJoin(qf.Order)
                   .On(EmployeeFields.EmployeeId == OrderFields.EmployeeId))
        .OrderBy(EmployeeFields.EmployeeId.Ascending()).Offset(1).Distinct()
        .Select(() => new
            {
                EmployeeId = EmployeeFields.EmployeeId.ToValue<int>(),
                Notes = EmployeeFields.Notes.ToValue<string>()
            })
        .CacheResultset(5);    // cache for 5 seconds
var results = new DataAccessAdapter().FetchQuery(q);

// custom projection with nested set
var qf = new QueryFactory();
var q = qf.Create()
        .Select(() => new
          {
                Key = CustomerFields.Country.ToValue<string>(),
                CustomersInCountry = qf.Customer.TargetAs("C")
                                         .CorrelatedOver(CustomerFields.Country.Source("C") ==
                                                         CustomerFields.Country)
                                         .ToResultset()
          })
        .GroupBy(CustomerFields.Country)
        .CacheResultset(10);     // cache all for 10 seconds
var results = new DataAccessAdapter().FetchQuery(q);

// entity fetch
var qf = new QueryFactory();
var q = qf.Customer
            .Where(CustomerFields.Country == "Germany")
                 .WithPath(CustomerEntity.PrefetchPathOrders
                               .WithSubPath(OrderEntity.PrefetchPathEmployee),
                                      CustomerEntity.PrefetchPathEmployeeCollectionViaOrder)
            .CacheResultset(10);    // cache all for 10 seconds
var customers = new DataAccessAdapter().FetchQuery(q);

Linq

// custom projection, with nested set
var metaData = new LinqMetaData(new DataAccessAdapter());
var q = (from c in metaData.Customer
          group c by c.Country into g
          select new
          {
             g.Key,
             CustomersInCountry = (
                  from c2 in metaData.Customer
                  where g.Key == c2.Country
                  select c2)
          })
         .CacheResultset(10);    // cache all for 10 seconds

// entity fetch with prefetch path
LinqMetaData metaData = new LinqMetaData(adapter);
var q = (from c in metaData.Customer
          where c.Country == "Germany"
          select c).WithPath<CustomerEntity>(cpath => cpath
                 .Prefetch<OrderEntity>(c => c.Orders)
                     .SubPath(opath => opath
                         .Prefetch(o => o.OrderDetails)
                         .Prefetch<EmployeeEntity>(o => o.Employee).Exclude(e => e.Photo, e => e.Notes)))
          .CacheResultset(10);    // cache all for 10 seconds

As you can see, it's just an operator to the query, and from then on its resultset is cached in the cache object registered for the connection string used at runtime. If there's already a resultset, that one is used; if there's no cached resultset found, the one read from the database is cached for the specified amount of time so the next execution of the same query will re-use that resultset.

an easy way to tweak your data-access code on a per-query basis.

ORM Profiler now .NET 4.5 / Async aware

FransBouma — Fri, 24 May 2013 11:29:28 GMT

We updated our data-access profiler ORM Profiler with a new .NET 4.5 specific interceptor dll. This interceptor, which is called SD.Tools.OrmProfiler.Interceptor.NET45.dll, can be found in the main installation folder of the ORM Profiler build we uploaded this week. If you're an ORM Profiler customer, please download the latest build from the customer area.

The .NET 4.5 interceptor intercepts calls to Async specific ADO.NET methods, and also contains all Entity Framework required classes, so there's no need for the EF4 specific interceptor if you're using .NET 4.5 and Entity Framework 5. Using this interceptor, any call to a *Async method will not fall back to the synchronous calls of System.Data.Common classes as the previous builds did, but will call into the Async methods of the used ADO.NET provider.

There's more async news coming soon, namely for LLBLGen Pro v4, stay tuned

LLBLGen Pro v4.0 feature highlights: Visual Studio integration

FransBouma — Mon, 13 May 2013 10:32:29 GMT

This is the first post of a series about the new features in LLBLGen Pro v4.0, which was released on April 6th, 2013. Starting with v4.0, the LLBLGen Pro designer is usable both as standalone (as it was before) and as an integrated designer in Visual Studio. I'll explain in this article what this integration is all about and what we had to do to make it work. If you want to know more about how to use this feature, please check out the manual page on this feature.

So how does this integration look like? Below is a screenshot from the manual:

It almost looks like the stand-alone designer, eh? Integrating in Visual Studio can be done in several ways: through e.g. a plug-in/add-in, an extension to the editor or a Visual Studio extension. Our integration is done through a Visual Studio extension (VSIX).

How the integration works, user perspective

When you start Visual Studio, the extension isn't visible: no menus, no tool windows, no tool bars. When you load an LLBLGen Pro project (a file with the extension '.llblgenproj') into Visual Studio, or create a new one inside Visual Studio using the normal item creation system, the designer is loaded inside Visual Studio: all panes are opened as tool windows and the project file itself is opened as a document editor inside the document well (central tab area) of Visual Studio. When this document editor, which is a single tab inside the document well, has the focus, the toolbar and menu are visible, and the project's contents is visible in the various tool windows (see screen shot above).

Opening another LLBLGen Pro project inside Visual Studio opens another document editor, but no extra tool windows. Instead, the contents of the tool windows is switched when you switch between document editors so they always show the project contents of the document editor which has focus. For a background on this, please see the section below called 'How we wrote the VSIX'.

Generating code will add the generated VS.NET projects to the loaded solution, if they're not already present there. This way of working as the advantage that you don't have to leave VS.NET to work with the designer, you can add the LLBLGen Pro project to your solution as a bonus don't have to manually add the generated code to your solution.

All functionality, except imports (see below) is the same as with the stand-alone designer, as the stand-alone designer is actually loaded inside Visual Studio and its parts are displayed within the IDE.

How we wrote the VSIX (or: how to lose your hair in 2 months)

Visual Studio is mainly an IDE around editing files: a file is opened with the associated editor of the file's extension in the document well and the editor associated with the file's extension can open 0 or more tool windows and do other things. You can of course open tool windows inside the document well which edit 'objects' and not 'files', however this isn't as simple as it sounds. And precisely that was a problem we had to solve: our designer is also an IDE with a central tab area but instead of opening files in that tab area, we open editors of objects.

To make Visual Studio work like our designer, we were forced to go what I'd call, 'the All-In' route: create custom project types for Visual Studio, rewrite our project explorer to build objects in the Solution Explorer in Visual Studio, open custom tool windows inside the document well of Visual Studio for each object edited. We quickly learned that creating an extension to Visual Studio is not a walk in the park: the code isn't that complex, it's the lack of serious documentation, examples and above all, the quirks of the IDE which made it a big a project to complete. Looking at the work for the All-in variant, we decided to go for the alternative: create a file-extension bound editor. Looking back this was a wise decision, as this extension too took us a long time to create, almost two months.

With v3.0, released in early 2010, we designed the designer to be modular so it wouldn't be to much work to make it usable from within another application, e.g. Visual Studio. The UI is broken up in several parts and all designer logic is structured in an event-oriented system with several controllers on top. The advantage of this is that whenever something happens inside the UI, one or more events are raised and observers to these events (or the events as result of these events) are picking them up and act accordingly. These observers are all located outside the UI. The main advantage of this is that even though the designer in stand-alone form looks like an MDI window with tool windows docked inside it, the control of these docked windows and what happens inside them is not within the MDI window.

The v4.0 designer is a successor of v3's so with the modular design at hand we decided to create a Visual Studio extension tied to the LLBLGen Pro project file file-extension '.llblgenproj'. In Visual Studio terms: a custom document editor. A Visual Studio extension is actually a Visual Studio package, packed in the form of an 'extension'. The extension contains of the following parts:

Each docked 'pane' in the designer is opened as a Visual Studio Tool window and docked inside the IDE
The document well of the designer is opened as a Visual Studio Document Editor in the document well.
The toolbar is created as a button bar inside Visual Studio
The menus are pivoted into a single menu in the main menu of Visual Studio
A factory which produces new document editors

To achieve this, we wrote an extension which loads the standalone designer into Visual Studio, and instead of telling it to open an MDI window and dock some panes into it, it simply tells the loaded designer to hand the pane windows and its own window to the extension object. The extension object then opens the Visual Studio tool windows and its main editor and there you are: integrated inside Visual Studio.

Simple? Well… not quite.

The fun starts with where Visual Studio extensions are located. There are two locations: inside the user's My Documents folder and inside the Visual Studio installation folder. Both locations are not ideal if you want your code to also be usable as stand-alone. The trick was to create a registry key at installation which is read by the Visual Studio extension which points to the folder where LLBLGen Pro was installed. I tried numerous things, including junctions but these all failed. The registry key is easy enough though.

The next problem was something we never had to run into before: what if multiple LLBLGen Pro projects are opened inside Visual Studio? Although opening multiple LLBLGen Pro projects will open multiple instances of the designer, without separating the AppDomains, we'd run into problems where statics were suddenly shared among instances. However, creating a new AppDomain per instance wouldn't work as that would require the editor panes we would like to show inside the designer to serializable, which isn't possible. Luckily we had grouped all statics inside objects which lived in a few singletons which act as Services inside our designer.

Whenever a designer instance was loaded into the extension, we assigned to it an ID and we'd request an object we dubbed integration controller from the designer, which would act like the controller of the designer and data exchange. Through this controller we also could tell the designer instance which document editor instance was active. The services/singletons then switched their state to the instance ID passed to the integration controller and all code of the designer and its sub-systems would think they were running inside the stand-alone version.

This state-switching also helped with the tool windows of Visual Studio: by default each Visual Studio tool window is created just once. You can create multiple instances if you must, but you'll soon learn that multiple instances of a tool window makes things less usable. So each time a document editor became active inside the IDE, the document editor would get called by the IDE and we simply made the extension to update the associated designer's state to the one associated with the document editor's ID and as well update the tool windows to the ones of the designer instance associated with the ID.

On to undo-redo. Our own designer has a full undo-redo system, based on our OSS library Algorithmia (available on github), and so does VS.NET. Of course these aren't compatible with each other however the designer's undo-redo should work in sync with the VS.NET undo-redo system. After looking into making things compatible for a few days, it occurred to me it doesn't matter: whenever a user does something through the document editor, the IDE will query your editor if the undo/redo state of the buttons should change. Our designer already kept track of that, so simply returning this state was enough. Whenever the user pressed the undo/redo buttons (or the keystrokes) inside the IDE, the document editor with focus got a call which was passed through to the integration controller to do the actual undo/redo on the model, similar to if the user would undo/redo in the stand-alone designer.

Undo-redo is a global system, however with multiple instances, there are multiple undo/redo stacks active at once, one per instance. Algorithmia's undo/redo system is able to work with that: it works with 'scopes'. This means you can have local undo/redo separated from the global undo/redo. We utilize this in the integration so each designer instance is a separate scope inside the undo/redo system, so undo/redo information isn't shared.

The last problem I'd like to address is installation. Visual Studio extensions, VSIXs, can be installed in two ways: you either dump all files in a folder in one of the two extension locations and let VS.NET know some extension installation took place or you wrap your code inside a .vsix file. The latter has as advantage that registration of your extension is done for you without a slow re-scan of the complete extension folders. It also takes care of locating the extensions folders. I first looked at the former alternative, as we use our own installer but the logic to locate the folders and install everything in both 2010 and 2012 was a pain, and everything was taken care of with a .vsix file: we simply run it silently during installation and it installs the extension in both 2010 and 2012 if present, in the right folder and registers the extension as well.

Was it all worth it?

Yes. Developing this extension was a true pain, I'd be lying if I called it something else, but as a tool / framework developer it's part of our job to write the painful code so our customers don't have to; that's why they pay for a license. In the end, I can only say I'm very pleased with the result and have heard only positive feedback from users, so in short: yes it was worth it.

As I said above, creating a Visual Studio extension isn't a walk in the park: using .NET for an extension is possible, the SDK looks OK, the starter project gives you a lot of code to start with, but after a short while you'll learn that below this thin layer of chrome, the actual metal is pretty rusty and shows its true age. The technical debt of the C/C++/COM based system is so big, I doubt it will ever get better, especially as there's little incentive to do so: the biggest 'customer' of VS.NET's integration system is Microsoft itself, and as we all know internal code is more rough, less documented than software that's delivered to customers outside the company.

If you're planning to develop your own Visual Studio extension, be aware of the fact you'll spend a lot of time cursing the IDE, it's long list of quirks and inconsistencies, missing docs and serious lack of examples how to do things: you'll run into many posts with similar questions as yours but with no or unusable answers. You're entitled to throw your hands into the air and scream the filthiest curse words in the direction of Redmond, but that won't get you near your goal. As an engineer, you have to think and act like one: write some prototype code and attach a debugger in another VS.NET instance to your test VS.NET instance to see what's happening, what's called when you do X.

It will be a struggle at first, but you'll get there: and you'll have to do it yourself, there's no-one out there to help you. But hey, that's the life of an engineer.

LLBLGen Pro v4.0 released!

FransBouma — Mon, 08 Apr 2013 09:48:08 GMT

I'm proud to announce we've released v4.0 of LLBLGen Pro! This release is the 13th full release we've done and it arrives in the year LLBLGen Pro turns 10. As always we've packed the new release with great new features and I hope we've added what you wanted us to add. You can find what's new here: http://www.llblgen.com/pages/whatsnew.aspx

Not only did we add new features, we also introduced something else: Lite.

Lite

LLBLGen Pro Lite is a free version of LLBLGen Pro. It packs all the features of the full version, however it comes with a limitation: an entity model is limited to 8 entities per model and it also come with no support. It's free as in beer and you can use the generated code in production.

Upgrades

V3.x users can upgrade with a 30% discount, by logging into the customer area.

LLBLGen Pro v4.0 BETA released!

FransBouma — Thu, 07 Mar 2013 16:35:59 GMT

We've released the first beta of LLBLGen Pro v4.0! We hope you all like the new features and additions we packed into this new release! To make sure stuff is truly tested in a lot of scenario's, the beta is available to all v3.x licensees.

How to get the beta
Log into the customer area, then select 'v3.5' from the Downloads menu and navigate on the downloads page to 'Betas'. No license is required, the beta license is included in the installer.

Installation
Run the installer as administrator. It will automatically install the VS.NET extension to use the designer inside VS.NET in VS.NET 2010 and VS.NET 2012 (pro and up), if available. Of course the designer is also usable as stand-alone, outside VS.NET.
The installer will start the VSIX (visual studio extension) installer at the end of the normal install. It might be the VSIX installer is still running when the normal installer ends. If you have VS.NET open when you install the beta, you'll have to close VS.NET and restart VS.NET to see the add-in.
You can install the beta side-by-side with older versions of LLBLGen Pro. V4's file format is compatible with v3, however a v4 file which is loaded into v3 and saved in v3 will lose v4 specific data.

Documentation
We included a word document with the details about this beta, which features are included and how to use them. This document, called LLBLGen Pro v4.0 beta readme.docx, is installed in the LLBLGen Pro v4 beta installation.

How to provide feedback
Please provide all feedback in the beta forum: http://www.llblgen.com/tinyforum/Threads.aspx?ForumID=52

What's new in V4.0
The list below is the global list of what's new in V4. The enclosed word document has the fine details of these new features and changes and also details of minor changes and new features not listed below.

Designer

Full Visual Studio integration (VS.NET 2010 / 2012)
Table Valued Functions support (EF/Linq to Sql/LLBLGen Pro RTF)
Additional interfaces/namespaces now use the rule system
Update / Delete rules for FK constraints (Cascade delete/update)
SQL Server 2012 sequence support (LLBLGen Pro RTF/NHibernate)
Action suggestions window
Custom type shortcuts can now have default length/precision
The designer is now a .NET 4.0 application, no more .net 4 runners.
Char <-> String system type converter
Re-scan of enum typeimport files without closing designer

Runtime Framework

Table Valued Functions support
Query result cache
Much faster entity fetches
Inlining DDD value types
SQL Server 2012 sequence support
Data Scopes (see https://github.com/SolutionsDesign/LLBLGenProExamples_4.0)
Xml Serializability of entire query API
True skip / offset
Fetch typed lists with QuerySpec
Runtime libraries are now built against .NET 3.5
QuerySpec and Linq assemblies are merged into ORMSupport Classes
OData support classes are now built against WCF Data Services 5.3 and support OData v3

I'm especially pleased with the VS.NET integration which turned out to be very nice, the TVF support, the much faster query fetches, the query result cache and the DataScopes. Please check out the example available at github (like above) how little code it takes to make a typical LoB application with the new DataScopes.

Re: You’re not your code

FransBouma — Fri, 25 Jan 2013 10:26:32 GMT

This post is a reply to Scott Hanselman’s post: "You’re not your code"

There are no two programmers in the world who like each others code, there's always criticism: "I would have done this differently" or worse "Whoa, dude, never heard of SOLID? Where are the tests for this?". IMHO, criticism is OK, as long as it's based on well founded arguments and they make the original programmer of the criticized code learn new things. I don't think Scott's post is about that.

I think Scott refers to this: simply bitching about someone else's code because you'd have implemented the same program much better, as you see yourself as a great developer is stupid.

I'd like to make an addition to that: if you think you're a great developer and express yourself in that fashion, and your professional code is not code one can expect from a professional (i.e. code which is maintainable, without (most of the) bugs, it's well documented etc.), you're a fraud and it's a little bit different: you tried to sell the idea that your hack, your hobby code is professional code.

However, not all code is professional code. Sometimes we simply hack something together to make something happen 1 time, or only for a single user (ourselves). For example internally here I use a change log system in where I log changes made to the various parts of the tools we produce. This is a very rudimentary piece of software, with a tremendously ugly UI, buggy event handlers and the databinding sometimes gets confused. This is because I wrote it in one afternoon, no tests, no design thoughts, just started hammering in code and when it compiled without errors I said "Done!".

But it works, it gets the job done of what I want to do: log the change made into a database so I can sync it with the online change log on our website so users can browse the change log online. Every time I run into a bug (which isn't often, as I know what to click and do) in this marvelous piece of code (ahum) I say to myself "it might be better if I fix this some day", but as the only time I use this software is when I have to do a push of a new build, this 'some day' is not today.

If this software would ever be released somewhere in source-code form, it will likely be bashed to pieces. If I would release it as source-code with the remark 'great system to log your changes and sync with online database', the bashing would IMHO be justified. But as it's a tool which is only for myself, and if I would release it as 'Dunno if this helps anyone, it works for me, have fun' and a bigwig would bash it to pieces because he's holier than me, that bashing IMHO wouldn't be justified: it's a hack, not professional software.

Bashing a hack released as a hack is stupid. Bashing a hack released as professional code isn't, as it exposes the hack as what it truly is: a hack.

Question of course is: when is a piece of code a hack / hobby code and when does it become professional code? IMHO the variety of answers to that question will tell you why the bashing of other people's code will continue no matter what and why there will never be two programmers in the world who like each others code without criticism.

Though we can do something. As it's difficult to determine when a piece of software is a hack if it's not described as such (not by one person but as a general qualification: if piece of software A is declared a hack, everyone will agree it is in fact crap), we as developers could keep that in mind when we use our bash-hammer on some piece of code we found online somewhere: what is truly gained from our criticism? Will the world be a better place? Will the software magically become better? Or is it just about our own ego that by declaring piece of software A a true example of crapcode, our own code must be much better?

The main reason we have to take that in mind is because we are our code. Well, not entirely, but we did write it. An outsider won't know whether we poured our soul into it and that we invested precious time in it. Bashing it to pieces because the fruits of our hard work is seen as a hack while it's not is going to hurt. If it doesn't hurt, you're not caring enough about the quality of your work, or they might have been right: your work was a hack after all, it didn't exceed the hobby code bar.

LLBLGen Pro and JSON serialization

FransBouma — Sun, 04 Nov 2012 15:29:01 GMT

I accidentally removed a reply from my previous blogpost, and as this blog-engine here at weblogs.asp.net is apparently falling apart, I can't re-add it as it thought it would be wise to disable comment controls on all posts, except new ones. So I'll post the reply here as a quote and reply on it.

'Steven' asks:

What would the future be for LLBLGen Pro to support JSON for serialization? Would it be worth the effort for a LLBLGenPro user to bother creating some code templates to produce additional JSON serializable classes?

Or just create some basic POCO classes which could be used for exchange of client/server data and use DTO to map these back to LLBGenPro ones? If I understand the work around, it is at the expense of losing xml serialization.

Well, as described in the previous post, to enable JSON serialization, you can do that with a couple of lines and some attribute assignments. However, indeed, the attributes might make the XML serialization not working, as described in the previous blogpost. This is the case if the service you're using serializes objects using the DataContract serializer: this serializer will give up to serialize the entity objects to XML as the entity objects implement IXmlSerializable and this is a no-go area for the DataContract serializer.

However, if your service doesn't use a DataContract serializer, or you serialize the objects manually to Xml using an xml serializer, you're fine.

When you want to switch to Xml serializing again, instead of JSON in WebApi, and you have decorated the entity classes with the data-contract attributes, you can switch off the DataContract serializer, by setting a global configuration setting:

var xml = GlobalConfiguration.Configuration.Formatters.XmlFormatter;
xml.UseXmlSerializer = true;

This will make the WebApi use the XmlSerializer, and run the normal IXmlSerializable interface implementation.

How to make ASP.NET WebApi serialize your LLBLGen Pro entities to JSON

FransBouma — Fri, 05 Oct 2012 13:01:00 GMT

LLBLGen Pro has several ways to serialize / deserialize your entity data in an LLBLGen Pro Entity object:

Binary, using superfast custom binary formatters, triggered by ISerializable methods.
XML, using XmlDocuments
XML, using XmlWriters, triggered by IXmlSerializable methods. Xml is configurable to various formats, from verbose (to rebuild the objects using the embedded type info) to compact ("Just the data, ma'am")

Once upon a time, all these fancy methods were the latest and greatest. Today however, they're not modern anymore, as a new format is taken over: JSON. JSON as the advantage that you can easily consume it in javascript on the client, no custom XML parser needed.

Microsoft recently released ASP.NET 4.0 MVC with a fancy new service system: WebAPI. WebAPI let's you create a service without the hassle that's required with a WCF service running inside IIS. Not only makes WebAPI it easy to write a service for your data needs on a client, it also makes it easy to serialize/deserialize this data as JSON.

This post is about how to make WebAPI work with LLBLGen Pro and serialize the data to JSON without the internals bleeding into the JSON data.

Note: by following these steps, the data is serialized as JSON only, the XML serializer won't work anymore (see below). By default, WebAPI serializes to XML, so you need these steps only if you want to serialize to JSON.

Step 1: decorate your entities, fields and navigators

LLBLGen Pro's designer has the ability to decorate generated classes and members of the generated classes with attributes, even based on rules you set. See a previous blogpost about this feature or the documentation for details.

Load your LLBLGen Pro project into the designer and select from the main menu Project -> Settings. A dialog opens. Navigate to General -> Conventions -> Code Generation -> Attributes.

At the top, 'Entity' is selected. Add below Attribute definition: DataContract

Press enter.

Select at the top 'NormalField'. Add below Attribute definition: DataMember

Press enter.

Repeat the action you did with NormalField also for NavigatorSingleValue and NavigatorCollection.

Click OK to close the settings dialog. You have now decorated all entities, the fields of the entities and the navigator properties with the required attributes.

Regenerate your code.

Step 2: Make WebAPI's JSON serialize do the right thing

Now, you might think that with the DataContract/DataMember attributes, we're going to use the DataContractJsonSerializer, but that's not the case. Not only is that serializer not recommended by many, including Rick Strahl, it also fails to do anything as it complains that the entity objects implement IXmlSerializable and it can't work with objects which implement IXmlSerializable and have DataContract attributes. These kind of errors show how dumb the serialization strategies (or lack thereof, if they had implement a proper strategy pattern for this it would all be OK) of Microsoft have been during the years: I have to have IXmlSerializable implemented on the classes for Xml serialization over normal WCF services to make sure change tracking data is preserved and also object graph references are left in-tact. The same error is given when we try to serialize the data to XML instead of JSON.

We're going to use the default JSON serializer of WebAPI, which is the Newtonsoft JSON.NET serializer. By default it doesn't do what we want, so we have to configure it. It's simple:

Open WebApiConfig.cs which is located in the App_Start folder. Now add the following lines to the Register method:

var json = config.Formatters.JsonFormatter;
json.SerializerSettings.PreserveReferencesHandling = Newtonsoft.Json.PreserveReferencesHandling.Objects;
json.SerializerSettings.ContractResolver = new DefaultContractResolver() 
    { 
        IgnoreSerializableInterface = true, 
        IgnoreSerializableAttribute = true 
    };

Step 3: There's no step 3

You're all setup now. A method like:

public IEnumerable<ProductEntity> GetAllProducts()
{
    using(var adapter = new DataAccessAdapter())
    {
        var metaData = new LinqMetaData(adapter);
        return metaData.Product.WithPath(p=>p.Prefetch(x=>x.Category)).ToList();     
    }
}

will now properly serialize the data (All product entities and their related category entities) to JSON, including reference tracking. Again, pulling the data as XML by specifying application/xml in the Accept header element of the request will make things fail as WebAPI chooses a DataContract serializer by default if it sees a [DataContract] attribute on the objects to serialize. It might be configurable to have it choose to use the normal XmlSerializer instead, but I'm no WebAPI expert so I can't give you a heads-up on that. If you know whether this is possible, please leave a comment below :)

Hope this helps!

Re: Everything's broken and nobody's upset

FransBouma — Tue, 18 Sep 2012 08:19:55 GMT

This is a re-worked answer to Scott Hanselman's great post 'Everything's broken and nobody's upset'. I replied it below the post, but I think it deserves a post here. Scott's post is about how we seemingly accept a long list of annoyances and issues with the software we use and how we apparently ignore the fact just how broken today's software seems to be. In my answer I tried to give an answer to why that is. This answer is below.

I think a lot of the problems with today's software are simply allowed to show up in public because the dev teams of these software products aren't given the time to fix them: Time to Market is critical, marketing already reserved a given launch date for the product, things are already in motion and only the bugs in the category 'it kills kittens and old ladies' are fixed at that point, the rest is 'moved to a later date'. But that later date never comes, because after the product is done, the team moves on to the newer version with new stuff, and fixing bugs is dreadful work, new stuff is exciting.

There's another thing about bugs which is also key in this issue: fixing them won't sell more licenses of vNext: people won't run to the store, yelling "OMG! They fixed bug 33422!!!1 I can't believe it!". They'll run to the store because new, shiny things have been added, like a completely new UI with ergonomic characteristics of which no-one really knows whether it's actually a step forward.

We, as software developers, all know software contains bugs, even though we did our best to fix them before RTM. But in the end, we have to admit that that label, 'RTM', is really just an arbitrary calendar event, not a landmark which says "0 bugs, It's ready!". This means that even though something went RTM, it doesn't mean it's bug free, it simply means: "It's good enough that it won't kill kittens nor old ladies".

A wise man, who unfortunately passed away way too soon, once said to me: "Your motivation and ability to fix issues and bugs is part of the quality you want to provide", which means: even though at first glance your software might look stunning out of the box, if that essential part of the quality of the software is missing, i.e. when a bug pops up it gets fixed, pronto, your software isn't of the quality you think it is. The insight given by this wise man opened my eyes all those years ago and I to this day still try to live up to it: stand for the quality you want to provide by giving customers the best support and software they could get.

If I look at today's software development landscape, I see a tremendous amount of people trying to write applications without the necessary skills and knowledge, in languages and platforms which have had a bad track-record for years, yet no-one seems to care, or at least too little. Last year I was in an open spaces session and some guy explained that they would place people who just started programming as 'trainee' at customers for 'free' and after a few months these trainees became 'junior' programmers and the client had to pay a fee per hour. I asked him what he thought of the term 'fraud', and he didn't understand what I meant. I tried to explain to him that if you think a person who has no training at all and you let him 'learn on the job' for a few months, is suddenly able to write any form of software, you're delusional. He didn't get what I wanted to say, and how it was a bad thing that someone with no skills was placed at a client as if the person knew what he was doing. Shockingly, some others in the room agreed with him.

But with the lack of highly trained professional developers growing and growing each day, more and more people who can tell the difference between a keyboard and a mouse are hired to do 'dev work', as the client doesn't know better and is already happy at least someone is there to do the work. I fear it only gets worse in the coming years. Frankly, I'm starting to get fed up with the bullshit pseudo-devs who seem to pop up more and more every day, who cry 'I'm just learning, don't be so rude!' when you tell them their work pretty much doesn't cut it, while at the same time they try to keep up the charade that they're highly skilled and experienced. You're either a trainee and therefore your work can lack in areas, or you're a professional and you have to take responsibility for the quality you say you'll provide.

Sadly this process of pseudo-devs which are seen as the true 'specialists', is in progress for some time now and will continue in the next decade I think. Let's hope they'll learn something about what 'quality' means with respect to software, that 'quality' is more than just the bits making up the software. But I'm not optimistic.

The Windows Store... why did I sign up with this mess again?

FransBouma — Wed, 12 Sep 2012 14:36:27 GMT

Yesterday, Microsoft revealed that the Windows Store is now open to all developers in a wide range of countries and locations. For the people who think "wtf is the 'Windows Store'?", it's the central place where Windows 8 users will be able to find, download and purchase applications (or as we now have to say to not look like a computer illiterate: <accent style="Kentucky">aaaaappss</accent>) for Windows 8.

As this is the store which is integrated into Windows 8, it's an interesting place for ISVs, as potential customers might very well look there first. This of course isn't true for all kinds of software, and developer tools in general aren't the kind of applications most users will download from the Windows store, but a presence there can't hurt.

Now, this Windows Store hosts two kinds of applications: 'Metro-style' applications and 'Desktop' applications. The 'Metro-style' applications are applications created for the new 'Metro' UI which is present on Windows 8 desktop and Windows RT (the single color/big font fingerpaint-oriented UI). 'Desktop' applications are the applications we all run and use on Windows today. Our software are desktop applications. The Windows Store hosts all Metro-style applications locally in the store and handles the payment for these applications. This means you upload your application (sorry, 'app') to the store, jump through a lot of hoops, Microsoft verifies that your application is not violating a tremendous long list of rules and after everything is OK, it's published and hopefully you get customers and thus earn money. Money which Microsoft will pay you on a regular basis after customers buy your application.

Desktop applications are not following this path however. Desktop applications aren't hosted by the Windows Store. Instead, the Windows Store more or less hosts a page with the application's information and where to get the goods. I.o.w.: it's nothing more than a product's Facebook page. Microsoft will simply redirect a visitor of the Windows Store to your website and the visitor will then use your site's system to purchase and download the application. This last bit of information is very important.

So, this morning I started with fresh energy to register our company 'Solutions Design bv' at the Windows Store and our two applications, LLBLGen Pro and ORM Profiler. First I went to the Windows Store dashboard page. If you don't have an account, you have to log in or sign up if you don't have a live account. I signed in with my live account. After that, it greeted me with a page where I had to fill in a code which was mailed to me. My local mail server polls every several minutes for email so I had to kick it to get it immediately.

I grabbed the code from the email and I was presented with a multi-step process to register myself as a company or as an individual. In red I was warned that this choice was permanent and not changeable. I chuckled: Microsoft apparently stores its data on paper, not in digital form. I chose 'company' and was presented with a lengthy form to fill out. On the form there were two strange remarks:

Per company there can just be 1 (one, uno, not zero, not two or more) registered developer, and only that developer is able to upload stuff to the store. I have no idea how this works with large companies, oh the overhead nightmares... "Sorry, but John, our registered developer with the Windows Store is on holiday for 3 months, backpacking through Australia, no, he's not reachable at this point. M'yeah, sorry bud. Hey, did you fill in those TPS reports yesterday?"
A separate Approver has to be specified, which has to be a different person than the registered developer. Apparently to Microsoft a company with just 1 person is not a company. Luckily we're with two people! *pfew*, dodged that one, otherwise I would be stuck forever: the choice I already made was not reversible!

After I had filled out the form and it was all well and good and accepted by the Microsoft lackey who had to write it all down in some paper notebook ("Hey, be warned! It's a permanent choice! Written down in ink, can't be changed!"), I was presented with the question how I wanted to pay for all this. "Pay for what?" I wondered. Must be the paper they were scribbling the information on, I concluded. After all, there's a financial crisis going on! How could I forget! Silly me.

"Ok fair enough".

The price was 75 Euros, not the end of the world. I could only pay by credit card, so it was accepted quickly. Or so I thought. You see, Microsoft has a different idea about CC payments. In the normal world, you type in your CC number, some date, a name and a security code and that's it. But Microsoft wants to verify this even more. They want to make a verification purchase of a very small amount and are doing that with a special code in the description. You then have to type in that code in a special form in the Windows Store dashboard and after that you're verified. Of course they'll refund the small amount they pull from your card.

Sounds simple, right? Well... no. The problem starts with the fact that I can't see the CC activity on some website: I have a bank issued CC card. I get the CC activity once a month on a piece of paper sent to me. The bank's online website doesn't show them. So it's possible I have to wait for this code till October 12th. One month.

"So what, I'm not going to use it anyway, Desktop applications don't use the payment system", I thought. "Haha, you're so naive, dear developer!" Microsoft won't allow you to publish any applications till this verification is done. So no application publishing for a month. Wouldn't it be nice if things were, you know, digital, so things got done instantly? But of course, that lackey who scribbled everything in the Big Windows Store Registration Book isn't that quick. Can't blame him though. He's just doing his job.

Now, after the payment was done, I was presented with a page which tells me Microsoft is going to use a third party company called 'Symantec', which will verify my identity again. The page explains to me that this could be done through email or phone and that they'll contact the Approver to verify my identity. "Phone?", I thought... that's a little drastic for a developer account to publish a single page of information about an external hosted software product, isn't it? On Facebook I just added a page, done. And paying you, Microsoft, took less information: you were happy to take my money before my identity was even 'verified' by this 3rd party's minions! "Double standards!", I roared. No-one cared. But it's the thought of getting it off your chest, you know.

Luckily for me, everyone at Symantec was asleep when I was registering so they went for the fallback option in case phone calls were not possible: my Approver received an email. Imagine you have to explain the idiot web of security theater I was caught in to someone else who then has to reply a random person over the internet that I indeed was who I said I was. As she's a true sweetheart, she gave me the benefit of the doubt and assured that for now, I was who I said I was.

Remember, this is for a desktop application, which is only a link to a website, some pictures and a piece of text. No file hosting, no payment processing, nothing, just a single page. Yeah, I also thought I was crazy. But we're not at the end of this quest yet.

I clicked around in the confusing menus of the Windows Store dashboard and found the 'Desktop' section. I get a helpful screen with a warning in red that it can't find any certified 'apps'. True, I'm just getting started, buddy. I see a link: "Check the Windows apps you submitted for certification". Well, I haven't submitted anything, but let's see where it brings me. Oh the thrill of adventure!

I click the link and I end up on this site: the hardware/desktop dashboard account registration. "Erm... but I just registered...", I mumbled to no-one in particular. Apparently for desktop registration / verification I have to register again, it tells me. But not only that, the desktop application has to be signed with a certificate. And not just some random el-cheapo certificate you can get at any mall's discount store. No, this certificate is special. It's precious. This certificate, the 'Microsoft Authenticode' Digital Certificate, is the only certificate that's acceptable, and jolly, it can be purchased from VeriSign for the price of only ... $99.-, but be quick, because this is a limited time offer! After that it's, I kid you not, $499.-. 500 dollars for a certificate to sign an executable. But, I do feel special, I got a special price. Only for me! I'm glowing. Not for long though.

Here I started to wonder, what the benefit of it all was. I now again had to pay money for a shiny certificate which will add 'Solutions Design bv' to our installer as the publisher instead of 'unknown', while our customers download the file from our website. Not only that, but this was all about a Desktop application, which wasn't hosted by Microsoft. They only link to it. And make no mistake. These prices aren't single payments. Every year these have to be renewed. Like a membership of an exclusive club: you're special and privileged, but only if you cough up the dough.

To give you an example how silly this all is: I added LLBLGen Pro and ORM Profiler to the Visual Studio Gallery some time ago. It's the same thing: it's a central place where one can find software which adds to / extends / works with Visual Studio. I could simply create the pages, add the information and they show up inside Visual Studio. No files are hosted at Microsoft, they're downloaded from our website. Exactly the same system.

As I have to wait for the CC transcripts to arrive anyway, I can't proceed with publishing in this new shiny store. After the verification is complete I have to wait for verification of my software by Microsoft. Even Desktop applications need to be verified using a long list of rules which are mainly focused on Metro-style applications. Even while they're not hosted by Microsoft. I wonder what they'll find. "Your application wasn't approved. It violates rule 14 X sub D: it provides more value than our own competing framework".

While I was writing this post, I tried to check something in the Windows Store Dashboard, to see whether I remembered it correctly. I was presented again with the question, after logging in with my live account, to enter the code that was just mailed to me. Not the previous code, a brand new one. Again I had to kick my mail server to pull the email to proceed. This was it. This 'experience' is so beyond miserable, I'm afraid I have to say goodbye for now to the 'Windows Store'. It's simply not worth my time.

Now, about live accounts. You might know this: live accounts are tied to everything you do with Microsoft. So if you have an MSDN subscription, e.g. the one which costs over $5000.-, it's tied to this same live account. But the fun thing is, you can login with your live account to the MSDN subscriptions with just the account id and password. No additional code is mailed to you. While it gives you access to all Microsoft software available, including your licenses.

Why the draconian security theater with this Windows Store, while all I want is to publish some desktop applications while on other Microsoft sites it's OK to simply sign in with your live account: no codes needed, no verification and no certificates? Microsoft, one thing you need with this store and that's: apps. Apps, apps, apps, apps, aaaaaaaaapps. Sorry, my bad, got carried away. I just can't stand the word 'app'. This store's shelves have to be filled to the brim with goods. But instead of being welcomed into the store with open arms, I have to fight an uphill battle with an endless list of rules and bullshit to earn the privilege to publish in this shiny store. As if I have to be thrilled to be one of the exclusive club called 'Windows Store Publishers'. As if Microsoft doesn't want it to succeed.

Craig Stuntz sent me a link to an old blog post of his regarding code signing and uploading to Microsoft's old mobile store from back in the WinMo5 days: http://blogs.teamb.com/craigstuntz/2006/10/11/28357/. Good read and good background info about how little things changed over the years.

I hope this helps Microsoft make things more clearer and smoother and also helps ISVs with their decision whether to go with the Windows Store scheme or ignore it. For now, I don't see the advantage of publishing there, especially not with the nonsense rules Microsoft cooked up. Perhaps it changes in the future, who knows.

LLBLGen Pro feature highlights: model views

FransBouma — Fri, 31 Aug 2012 10:38:11 GMT

(This post is part of a series of posts about features of the LLBLGen Pro system)

To be able to work with large(r) models, it's key you can view subsets of these models so you can have a better, more focused look at them. For example because you want to display how a subset of entities relate to one another in a different way than the list of entities. LLBLGen Pro offers this in the form of Model Views. Model Views are views on parts of the entity model of a project, and the subsets are displayed in a graphical way.

Additionally, one can add documentation to a Model View. As Model Views are displaying parts of the model in a graphical way, they're easier to explain to people who aren't familiar with entity models, e.g. the stakeholders you're interviewing for your project. The documentation can then be used to communicate specifics of the elements on the model view to the developers who have to write the actual code.

Below I've included an example. It's a model view on a subset of the entities of AdventureWorks. It displays several entities, their relationships (both relational and inheritance relationships) and also some specifics gathered from the interview with the stakeholder. As the information is inside the actual project the developer will work with, the information doesn't have to be converted back/from e.g .word documents or other intermediate formats, it's the same project. This makes sure there are less errors / misunderstandings.

(of course you can hide the docked documentation pane or dock it to another corner).

The Model View can contain entities which are placed in different groups. This makes it ideal to group entities together for close examination even though they're stored in different groups.

The Model View is a first-class citizen of the code-generator. This means you can write templates which consume Model Views and generate code accordingly. E.g. you can write a template which generates a service per Model View and exposes the entities in the Model View as a single entity graph, fetched through a method. (This template isn't included in the LLBLGen Pro package, but it's easy to write it up yourself with the built-in template editor).

Viewing an entity model in different ways is key to fully understand the entity model and Model Views help with that.

Beware when using .NET's named pipes in a windows forms application

FransBouma — Tue, 28 Aug 2012 17:08:47 GMT

Yesterday a user of our .net ORM Profiler tool reported that he couldn't get the snapshot recording from code feature working in a windows forms application. Snapshot recording in code means you start recording profile data from within the profiled application, and after you're done you save the snapshot as a file which you can open in the profiler UI. When using a console application it worked, but when a windows forms application was used, the snapshot was always empty: nothing was recorded.

Obviously, I wondered why that was, and debugged a little. Here's an example piece of code to record the snapshot. This piece of code works OK in a console application, but results in an empty snapshot in a windows forms application:

var snapshot = new Snapshot();
snapshot.Record();
using(var ctx = new ORMProfilerTestDataContext())
{
	var customers = ctx.Customers.Where(c => c.Country == "USA").ToList();
}
InterceptorCore.Flush();
snapshot.Stop();
string error=string.Empty;
if(!snapshot.IsEmpty)
{
	snapshot.SaveToFile(@"c:\temp\generatortest\test2\blaat.opsnapshot", out error);
}
if(!string.IsNullOrEmpty(error))
{
	Console.WriteLine("Save error: {0}", error);
}

(the Console.WriteLine doesn't do anything in a windows forms application, but you get the idea).

ORM Profiler uses named pipes: the interceptor (referenced and initialized in your application, the application to profile) sends data over the named pipe to a listener, which when receiving a piece of data begins reading it, asynchronically, and when properly read, it will signal observers that new data has arrived so they can store it in a repository. In this case, the snapshot will be the observer and will store the data in its own repository.

The reason the above code doesn't work in windows forms is because windows forms is a wrapper around Win32 and its WM_* message based system. Named pipes in .NET are wrappers around Windows named pipes which also work with WM_* messages. Even though we use BeginRead() on the named pipe (which spawns a thread to read the data from the named pipe), nothing is received by the named pipe in the windows forms application, because it doesn't handle the WM_* messages in its message queue till after the method is over, as the message pump of a windows forms application is handled by the only thread of the windows forms application, so it will handle WM_* messages when the application idles.

The fix is easy though: add Application.DoEvents(); right before snapshot.Stop(). Application.DoEvents() forces the windows forms application to process all WM_* messages in its message queue at that moment: all messages for the named pipe are then handled, the .NET code of the named pipe wrapper will react on that and the whole process will complete as if nothing happened.

It's not that simple to just say 'why didn't you use a worker thread to create the snapshot here?', because a thread doesn't get its own message pump: the messages would still be posted to the window's message pump. A hidden form would create its own message pump, so the additional thread should also create a window to get the WM_* messages of the named pipe posted to a different message pump than the one of the main window.

This WM_* messages pain is not something you want to be confronted with when using .NET and its libraries. Unfortunately, the way they're implemented, a lot of APIs are leaky abstractions, they bleed the characteristics of the OS objects they hide away through to the .NET code. Be aware of that fact when using them :)

LLBLGen Pro feature highlights: grouping model elements

FransBouma — Wed, 20 Jun 2012 10:59:02 GMT

(This post is part of a series of posts about features of the LLBLGen Pro system)

When working with an entity model which has more than a few entities, it's often convenient to be able to group entities together if they belong to a semantic sub-model. For example, if your entity model has several entities which are about 'security', it would be practical to group them together under the 'security' moniker. This way, you could easily find them back, yet they can be left inside the complete entity model altogether so their relationships with entities outside the group are kept.

In other situations your domain consists of semi-separate entity models which all target tables/views which are located in the same database. It then might be convenient to have a single project to manage the complete target database, yet have the entity models separate of each other and have them result in separate code bases.

LLBLGen Pro can do both for you. This blog post will illustrate both situations. The feature is called group usage and is controllable through the project settings. This setting is supported on all supported O/R mapper frameworks.

Situation one: grouping entities in a single model.

This situation is common for entity models which are dense, so many relationships exist between all sub-models: you can't split them up easily into separate models (nor do you likely want to), however it's convenient to have them grouped together into groups inside the entity model at the project level. A typical example for this is the AdventureWorks example database for SQL Server. This database, which is a single catalog, has for each sub-group a schema, however most of these schemas are tightly connected with each other: adding all schemas together will give a model with entities which indirectly are related to all other entities. LLBLGen Pro's default setting for group usage is AsVisualGroupingMechanism which is what this situation is all about: we group the elements for visual purposes, it has no real meaning for the model nor the code generated.

Let's reverse engineer AdventureWorks to an entity model. By default, LLBLGen Pro uses the target schema an element is in which is being reverse engineered, as the group it will be in. This is convenient if you already have categorized tables/views in schemas, like which is the case in AdventureWorks. Of course this can be switched off, or corrected on the fly.

When reverse engineering, we'll walk through a wizard which will guide us with the selection of the elements which relational model data should be retrieved, which we can later on use to reverse engineer to an entity model. The first step after specifying which database server connect to is to select these elements. below we can see the AdventureWorks catalog as well as the different schemas it contains. We'll include all of them.

After the wizard completes, we have all relational model data nicely in our catalog data, with schemas. So let's reverse engineer entities from the tables in these schemas. We select in the catalog explorer the schemas 'HumanResources', 'Person', 'Production', 'Purchasing' and 'Sales', then right-click one of them and from the context menu, we select Reverse engineer Tables to Entity Definitions.... This will bring up the dialog below. We check all checkboxes in one go by checking the checkbox at the top to mark them all to be added to the project.

As you can see LLBLGen Pro has already filled in the group name based on the schema name, as this is the default and we didn't change the setting. If you want, you can select multiple rows at once and set the group name to something else using the controls on the dialog. We're fine with the group names chosen so we'll simply click Add to Project. This gives the following result:

(I collapsed the other groups to keep the picture small ;)). As you can see, the entities are now grouped. Just to see how dense this model is, I've expanded the relationships of Employee:

As you can see, it has relationships with entities from three other groups than HumanResources. It's not doable to cut up this project into sub-models without duplicating the Employee entity in all those groups, so this model is better suited to be used as a single model resulting in a single code base, however it benefits greatly from having its entities grouped into separate groups at the project level, to make work done on the model easier.

Now let's look at another situation, namely where we work with a single database while we want to have multiple models and for each model a separate code base.

Situation two: grouping entities in separate models within the same project.

To get rid of the entities to see the second situation in action, simply undo the reverse engineering action in the project. We still have the AdventureWorks relational model data in the catalog. To switch LLBLGen Pro to see each group in the project as a separate project, open the Project Settings, navigate to General and set Group usage to AsSeparateProjects.

In the catalog explorer, select Person and Production, right-click them and select again Reverse engineer Tables to Entities.... Again check the checkbox at the top to mark all entities to be added and click Add to Project. We get two groups, as expected, however this time the groups are seen as separate projects. This means that the validation logic inside LLBLGen Pro will see it as an error if there's e.g. a relationship or an inheritance edge linking two groups together, as that would lead to a cyclic reference in the code bases.

To see this variant of the grouping feature, seeing the groups as separate projects, in action, we'll generate code from the project with the two groups we just created: select from the main menu: Project -> Generate Source-code... (or press F7 ;)). In the dialog popping up, select the target .NET framework you want to use, the template preset, fill in a destination folder and click Start Generator (normal). This will start the code generator process.

As expected the code generator has simply generated two code bases, one for Person and one for Production:

The group name is used inside the namespace for the different elements. This allows you to add both code bases to a single solution and use them together in a different project without problems. Below is a snippet from the code file of a generated entity class.

//...
using System.Xml.Serialization;
using AdventureWorks.Person;
using AdventureWorks.Person.HelperClasses;
using AdventureWorks.Person.FactoryClasses;
using AdventureWorks.Person.RelationClasses;
using SD.LLBLGen.Pro.ORMSupportClasses;

namespace AdventureWorks.Person.EntityClasses
{    
//...
	/// <summary>Entity class which represents the entity 'Address'.<br/><br/></summary>
	[Serializable]    
	public partial class AddressEntity : CommonEntityBase
//...

The advantage of this is that you can have two code bases and work with them separately, yet have a single target database and maintain everything in a single location. If you decide to move to a single code base, you can do so with a change of one setting. It's also useful if you want to keep the groups as separate models (and code bases) yet want to add relationships to elements from another group using a copy of the entity: you can simply reverse engineer the target table to a new entity into a different group, effectively making a copy of the entity. As there's a single target database, changes made to that database are reflected in both models which makes maintenance easier than when you'd have a separate project for each group, with its own relational model data.

Conclusion

LLBLGen Pro offers a flexible way to work with entities in sub-models and control how the sub-models end up in the generated code.

Azure, don't give me multiple VMs, give me one elastic VM

FransBouma — Fri, 08 Jun 2012 09:51:54 GMT

Yesterday, Microsoft revealed new major features for Windows Azure (see ScottGu's post). It all looks shiny and great, but after reading most of the material describing the new features, I still find the overall idea behind all of it flawed: why should I care on how much VMs my web app runs? Isn't that a problem to solve for the Windows Azure engineers / software? And what if I need the file system, why can't I simply get a virtual filesystem ?

To illustrate my point, let's use a real example: a product website with a customer system/database and next to it a support site with accompanying database. Both are written in .NET, using ASP.NET and use a SQL Server database each. The product website offers files to download by customers, very simple.

You have a couple of options to host these websites:

Buy a server, place it in a rack at an ISP and run the sites on that server
Use 'shared hosting' with an ISP, which means your sites' appdomains are running on the same machine, as well as the files stored, and the databases are hosted in the same server as the other shared databases.
Hire a VM, install your OS of choice at an ISP, and host the sites on that VM, basically the same as the first option, except you don't have a physical server
At some cloud-vendor, either host the sites 'shared' or in a VM. See above.

With all of those options, scalability is a problem, even the cloud-based ones, though not due to the same reasons:

The physical server solution has the obvious problem that if you need more power, you need to buy a bigger server or more servers which requires you to add replication and other overhead
Shared hosting solutions are almost always capped on memory usage / traffic and database size: if your sites get too big, you have to move out of the shared hosting environment and start over with one of the other solutions
The VM solution, be it a VM at an ISP or 'in the cloud' at e.g. Windows Azure or Amazon, in theory allows scaling out by simply instantiating more VMs, however that too introduces the same overhead problems as with the physical servers: suddenly more than 1 instance runs your sites.

If a cloud vendor offers its services in the form of VMs, you won't gain much over having a VM at some ISP: the main problems you have to work around are still there: when you spin up more than one VM, your application must be completely stateless at any moment, including the DB sub system, because what's in memory in instance 1 might not be in memory in instance 2.

This might sounds trivial but it's not. A lot of the websites out there started rather small: they were perfectly runnable on a single machine with normal memory and CPU power. After all, you don't need a big machine to run a website with even thousands of users a day. Moving these sites to a multi-VM environment will cause a problem: all the in-memory state they use, all the multi-page transitions they use while keeping state across the transition, they can't do that anymore like they did that on a single machine: state is something of the past, you have to store every byte of state in either a DB or in a viewstate or in a cookie somewhere so with the next request, all state information is available through the request, as nothing is kept in-memory.

Our example uses a bunch of files in a file system. Using multiple VMs will require that these files move to a cloud storage system which is mounted in each VM so we don't have to store the files on each VM. This might require different file paths, but this change should be minor. What's perhaps less minor is the maintenance procedure in place on the new type of cloud storage used: instead of ftp-ing into a VM, you might have to update the files using different ways / tools.

All in all this makes moving an existing website which was written for an environment that's based around a VM (namely .NET with its CLR) overly cumbersome and problematic: it forces you to refactor your website system to be able to be used 'in the cloud', which is caused by the limited way how e.g. Windows Azure offers its cloud services: in blocks of VMs.

Offer a scalable, flexible VM which extends with my needs

Instead, cloud vendors should offer simply one VM to me. On that VM I run the websites, store my DB and my files. As it's a virtual machine, how this machine is actually ran on physical hardware (e.g. partitioned), I don't care, as that's the problem for the cloud vendor to solve. If I need more resources, e.g. I have more traffic to my server, way more visitors per day, the VM stretches, like I bought a bigger box. This frees me from the problem which comes with multiple VMs: I don't have any refactoring to do at all: I can simply build my website as if it runs on my local hardware server, upload it to the VM offered by the cloud vendor, install it on the VM and I'm done.

"But that might require changes to windows!"
Yes, but Microsoft is Windows. Windows Azure is their service, they can make whatever change to what they offer to make it look like it's windows. Yet, they're stuck, like Amazon, in thinking in VMs, which forces developers to 'think ahead' and gamble whether they would need to migrate to a cloud with multiple VMs in the future or not. Which comes down to: gamble whether they should invest time in code / architecture which they might never need. (YAGNI anyone?)

So the VM we're talking about, is that a low-level VM which runs a guest OS, or is that VM a different kind of VM?

The flexible VM: .NET's CLR ?

My example websites are ASP.NET based, which means they run inside a .NET appdomain, on the .NET CLR, which is a VM. The only physical OS resource the sites need is the file system, however this too is accessed through .NET. In short: all the websites see is what .NET allows the websites to see, the world as the websites know it is what .NET shows them and lets them access. How the .NET appdomain is run physically, that's the concern of .NET, not mine. This begs the question why Windows Azure doesn't offer virtual appdomains? Or better: .NET environments which look like one machine but could be physically multiple machines.

In such an environment, no change has to be made to the websites to migrate them from a local machine or own server to the cloud to get proper scaling: the .NET VM will simply scale with the need: more memory needed, more CPU power needed, it stretches. What it offers to the application running inside the appdomain is simply increasing, but not fragmented: all resources are available to the application: this means that the problem of how to scale is back to where it should be: with the cloud vendor.

"Yeah, great, but what about the databases?"
The .NET application communicates with the database server through a .NET ADO.NET provider. Where the database is located is not a problem of the appdomain: the ADO.NET provider has to solve that. I.o.w.: we can host the databases in an environment which offers itself as a single resource and is accessible through one connection string without replication overhead on the outside, and use that environment inside the .NET VM as if it was a single DB.

But what about memory replication and other problems?

This environment isn't simple, at least not for the cloud vendor. But it is simple for the customer who wants to run his sites in that cloud: no work needed. No refactoring needed of existing code. Upload it, run it.

Perhaps I'm dreaming and what I described above isn't possible. Yet, I think if cloud vendors don't move into that direction, what they're offering isn't interesting: it doesn't solve a problem at all, it simply offers a way to instantiate more VMs with the guest OS of choice at the cost of me needing to refactor my website code so it can run in the straight jacket form factor dictated by the cloud vendor.

Let's not kid ourselves here: most of us developers will never build a website which needs a truck load of VMs to run it: almost all websites created by developers can run on just a few VMs at most. Yet, the most expensive change is right at the start: moving from one to two VMs. As soon as you have refactored your website code to run across multiple VMs, adding another one is just as easy as clicking a mouse button. But that first step, that's the problem here and as it's right there at the beginning of scaling the website, it's particularly strange that cloud vendors refuse to solve that problem and leave it to the developers to solve that.

Which makes migrating 'to the cloud' particularly expensive.