Dmitri Nesteruk

Simpler MVC localization

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Wed, 11 Nov 2009 10:31:00 +0300

I’ve been working on a typical implementation of Asp.Net MVC localization. However, unlike the textbook examples, I wanted to have just the language, not the culture, feature in my paths. Specifically, intead of writing http://mydomain.com/en-US/ I wanted to be able to write just http://mydomain.com/en/. Unfortunately, you cannot set the CurrentCulture variable to a ‘neurtal’ culture. Consequently, what I ended up doing is keeping a dictionary of cultures and applying the default one whenever needed:

public class InternationalizationAttribute : ActionFilterAttribute { private static Dictionary<string, string> cultureMappings = new Dictionary<string, string> { {"en", "US"}, {"sv", "SE"}, {"ru", "RU"} }; public override void OnActionExecuting(ActionExecutingContext filterContext) { string language = (string)filterContext.RouteData.Values["language"] ?? "en"; string culture = cultureMappings[language]; try { Thread.CurrentThread.CurrentCulture = CultureInfo.GetCultureInfo(language + "-" + culture); Thread.CurrentThread.CurrentUICulture = CultureInfo.GetCultureInfo(language + "-" + culture); } catch { } } }

I don’t know, though, maybe there’s a better way of doing this.

Improving business orientation

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Thu, 05 Nov 2009 23:20:00 +0300

Throughought my development career, I always thought of IT as self-serving. The idea of software actually servicing a business didn’t escape me, but it was never as pertinent as the pursuit of software for the art of it instead of its usefulness. I guess I’m old enough now to realize that this is a key issue with software altogether – that we often like it for the wrong reasons.

So what am I going to do about it? A couple of things.

First, I will stop writing technology articles of the self-serving variety. I think I’ll write more on how IT helps business, rather than on how IT does something fantastic internally. Don’t get me wrong, technical articles will be there too.

Also, I will focus more on domain-specific issues, rather than abstract examples. Specifically, I want to write more about business intelligence (BI), which should be useful not just to programmers, but also to people who are interested in technology applications in business.

I will also start thinking more about ways in which all our technical wizardry can be used to improve our own business. This ‘eating our dogfood’ approach is precisely what technology companies should be doing more of. I’m pretty sure one can get economical benefits by applying your own technical expertise in running a business.

Wish me luck.

Use ASP.NET to collect visitor stats from blogs you do not own (but can post in)

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Tue, 03 Nov 2009 22:35:00 +0300

This is a quickie post explaining how to use ASP.NET in order to collect visitor statistics on a page which does not allow JavaScript. The idea is simple – you place a 1×1px transparent GIF onto the image, with the GIF’s address being on your server.

The GIF actually points to a typical ASPX page, i.e., you write something like the following:

<img src="http://mysite/stats/default.aspx"/>

The page default.aspx is a basic WebForms page. Its statistical tracking is done in the Page_Load() method, the contents of the ASPX file being completely irrelevant for our purposes.

Three things happen in Page_Load(). First, since we don’t want our service abused, we check that the page that sent the request is ‘legit’. If you don’t do this, people will figure out that you have a service, and might even try to DDoS you just for the fun of it. I also check that the IP isn’t mine, so that my own visits to the page do not count.

private static readonly string[] permittedURLs = new[] { "http://site.com/page/" }; protected void Page_Load(object sender, EventArgs e) { if (Request.UrlReferrer == null || Array.IndexOf(permittedURLs, Request.UrlReferrer.AbsoluteUri) == -1 || Request.UserHostAddress == "1.2.3.4") // my IP address return; }

If any of the checks fail, we return from the page, yielding the ASPX content instead of the image. If they succeed, we add an entry to the database. The example below uses LINQ2SQL in order to add an entry concerning the time the page was accessed, the IP of whoever accessed it, and the page it was accessed from.

using (var dc = new VisitsDataContext()) { Visit v = new Visit(); v.DateAndTime = DateTime.Now; v.IPAddress = Request.UserHostAddress; v.ReferrerURL = Request.UrlReferrer != null ? Request.UrlReferrer.AbsoluteUri : null; dc.Visits.InsertOnSubmit(v); dc.SubmitChanges(); }

For safety, I check referrer for null, which helps with debugging. Now, with that done, I load up a 1-pixel GIF (which I keep in the same folder) and serve it. Then, I end the response so that the ASPX doesn’t get sent after the image data.

using (Bitmap bmp = (Bitmap)Bitmap.FromFile(Server.MapPath("OnePixel.gif"))) { Response.ContentType = "image/gif"; bmp.Save(Response.OutputStream, ImageFormat.Gif); } Response.End();

Using the code shown, you can easily and discreetly collect usage statistics from any web-resource which you do not own but can post images on.

Outsourcing vs Custom Development

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Wed, 28 Oct 2009 19:53:00 +0300

I don’t like the term ‘outsourcing’. In fact, if you visit our company site, you’ll find no mention of ‘outsourcing’, ‘offshore’ and similar terms. Maybe I’m being unfair here, but there seem to be some sort of negative connotation associated with the term, and I prefer the term ‘custom development’ (or just ‘custom dev’) instead. The way I see it,

Outsourcing = hiring developers because they are cheap

Custom development = hiring developers because they are brilliant

The not-so-subtle difference has to do with the way people view software development generally, applying inappropriate measurement techniques when assessing the efficacy of software development. See, unlike mechanical engineering, where a part’s production time is fixed and predictable, software engineering is creative and non-linear, which implies that conventional economic models of ‘parts per hour’ are all but useless.

Now, it’s not just that. The quality of manufactured parts typically doesn’t vary much. With quality control, person A is roughly as good as person B in terms of making some part or other. The difference of 2× is not significant. In software development, the difference can be 100x, but while quality differs vastly, wages do not. This makes the idea of ‘cheap developers’ somewhat silly, because developers by and large distribute themselves evenly on the wage scale whatever the wages are.

Now, if projects succeeded regularly, you might argue that I’m wrong and it doesn’t matter. But projects fail. As in EPIC FAIL. Projects mess up due to a zillion reasons, making a truly shocking statistic for an industry that is supposed to be precise and ultra-efficient (hey, we have measurement tools and all that, right?). Somehow, I’m thinking, the idea of having brilliant developers (managers, qa engineers, etc.) actually fits into the larger idea of actually making the project on time and budget. Which is what this business is all about, last time I checked.

The moral of the story is this: custom development companies like mine will never be cheap, but will make every effort to product good software. Which is why we’re in this business altogether.

Looking for projects on freelance sites

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Mon, 26 Oct 2009 18:37:00 +0300

I’ve recently been investigating various freelance sites, such as Rent-a-Coder, looking for juicy contracts to pick up and, to be honest, I’m amazed by people’s mentality. For example, I saw a contract that I really liked – it was in line with our expertise, the client seemed serious, but… they’re looking to pay $1500-5000 on what is basically about $100k of development effort.

Now, this got me thinking: obviously we won’t be bidding for such a contract, but other people will. Can they really offer such an amount of development effort? Even if we assume that rates on freelance sites are a lot smaller than ours, I’m thinking that people still need to make a profit, right? Either that, or everyone out there is out to con unsuspecting buyers of their money. Because it looks like you’ve got a buyer who wants a BMW but only has enough for a Lada. Do they get a BMW? I don’t think so.

The economics of freelance sites are bewildering.

TypograFix 1.6 Released

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Tue, 20 Oct 2009 20:48:00 +0300

Well, you’ve guessed it, TypograFix 1.6 is out!!! There’s been lots of additions which, I’m happy to say, were driven by need rather than a desire to just code something. Here is a summary of some of the things that have been implemented recently.

Freeform images like . These images are not only generated inside the program by Direct2D, but they can also be automatically uploaded to a web server of your choice (I’m using a good old-fashioned Asp.Net web service here), so posting text rendered as images on any online resource is suddenly very easy.

On a small sidenote, this functionality is one of a very small number of features that I’ve made using C++ instead of C#. Deployment got a lot tougher, but the advantages are well worth it.
Graphs. Yes, I’m only just getting started with graphs here, but already I’ve got some successes, weaving a DSL into my editor and managing to produce pretty graph structures, like the one seen on the right. The only thing I have failed to make (stupid me) is any support for subpixel hinting. Bizarrely, those same methods that have worked for me in the process of generating WPF images have totally stopped working when using the Automated Graph Layout. Which is just sad, really.
I’ve managed to add syntax highlighting and discriminator tags for a few languages, most notably Boo. Syntax highlighting is still a pain for many reasons (none of them being my fault), but at least I’m getting to a state where this feature is universally usable.
Autoinsertion options save a lot of time. Basically, the F1—F12 keys can be customized to emit strings when pressed. For example, my F2 keys emits a close-of-article mark,  ■, in HTML. Very useful.
There are some all-round UI improvements. To improve update speeds as well as mitigate potential problems, I’ve implemented caching for images, and have also added a feature where text transformation can be temporarily suspended so that 10-sec delays do not throw you off if you’re doing something important.
Last but not least, I’ve added support for M:tG symbols. It may not seem like much, but now I have a very efficient shortcut for typing RGBWU and getting rendered on the page. It’s one of those little niceties when you write your own editor.

That’s all for now. As always, after any serious release, I’ll be off to test all this stuff and iron out the bugs. Rest assured, version 1.7 is already in the works.

VS2010, please, can I have a faster editor?

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Tue, 20 Oct 2009 02:02:00 +0300

I’d like to share a personal experience, if I may. I’ve been working with Visual Studio for absolute ages, since about Version 5 (which I think was called 97, but I can’t be sure). And while VS by itself was and is becoming more and more powerful with every release, I have to point out a (well-known) flaw that weighs heavily against its continuous use.

Visual Studio is slow.

Now, now, don’t argue. It’s unfortunate, but it’s true: beleive me, I’ve been running VS from RAM and it’s still slow. What do I mean by ‘slow’, then? Well, the fact that when I work in SharpDevelop, things just fly. I type quickly, nothing lags, compilation is a breeze. But in VS, I feel like there’s a demon standing beside me, waiting to slap me the moment I get to any serious level of productivity.

Now, the problem in VS2010 has been exacerbated, of course. 2008 is already pretty slow, especially with all the bells and whistles, but 2010 uses WPF for its UI and – I don’t know, maybe it’s just me – but my peception from the betas is that the extra goodness of all those nice shiny buttons is negated by the fact that I can’t get my code quickly.

One can open a philosophical can of worms here about what productivity means, but my intinct tells me that the undamental goal of software development is to write good code, quickly. And for that you need

A text editor
A compiler
A good debugger

If it weren’t for debugging, you can bet I’d be using other IDEs most of the time because what concerns me is how fast my ideas turn to code. But regardless of it all, I am of course anxious to see VS2010 released and in action. I’m betting that potential clients would also want to see it, and not 2008, on our machines in future, and for us 2010 will be a staplemark of progress as much as anything else.

We’ll just have to wait and see.

Html encoding bugs in AntiXss

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Sat, 17 Oct 2009 02:13:00 +0300

I’m annoyed. I’ve been playing with various AntiXss tools, including the one made by Microsoft. They all work, and they do what is expected, but they expose a fairly nasty bug. This bug is not in the Xss tools themselves, but rather in the HttpUtility’s HtmlEncode() method.

You see, it’s a bad, buggy method. It can encode an < to become an <, but that’s about all it can do. It cannot encode something like a • (•), or a — (—) or a ξ (ξ). Why? Because it doesn’t know how.

Because of this, the AntiXss methods are next to useless. What I effectively have to do is go over the resulting code after the transformation has taken place, doing ridiculous stuff like x = x.Replace("•", "•"). This is a real annoyance because I cannot be expected to be able to cover every HTML entity under the sun.

This goes to show a simple fact: the idea of doing AntiXss by Html-decoding everything and only encoding what you want is fundamentally wrong. There has got to be a better way out there somewhere.

A tip for managing complex CSS

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Mon, 12 Oct 2009 13:23:00 +0300

Managing a custom CSS is a nightmare, but making changes to it is even more so – especially if you need to make the same change in lots of places. My solution – applying T4 to CSS, thereby creating a Site.css style only as a byproduct of a Site.tt template:

<#@ output extension=".css" #> <# string fontFamily = "Calibri, Arial, Helvetica, Sans-Serif"; int fontSize = 12; // px #> body, p, td, li, a, input { font-family: <#= fontFamily #>; } body { font-size: <#= fontSize #>px; background-color: Black; color: White; } h1, h2 { font-family: <#= fontFamily #>; font-weight: normal }

Another problem solved.

Internal DSL in C# via reflection

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Fri, 09 Oct 2009 22:02:00 +0300

There are many approaches to implementing user-entered DSLs. In my application, I wanted a user to enter a DSL and show its result immediately (pseudo real-time) on the screen. I wanted natural text input for the creation of directed graphs with simple statements. For example, I wanted the statement edge a b to create something like this:

There are many ways of doing this, of course, but I went for something simple: using reflection to transform strings into executable code. What I mean is that when someone writes edge a b I really want them to call edge("a", "b") in a certain class.

public void edge(string source, string target) { currentEdge = graph.AddEdge(source, target); }

With this in mind, I had to implement a simplistic parser that would find the method for me (without paying the cost of reflection). A basic string.Split() got me the constituent parts and I managed to find the right method:

public void AddInstruction(string line) { if (string.IsNullOrEmpty(line)) return; foreach (Match m in Regex.Matches(line, "([\"'])(?:\\\\\\1|.)*?\\1")) line = line.Replace(m.Value, m.Value.Replace(' ', '`')); string[] parts = line.Split(' ').Select(p => p.Replace('`', ' ').Unquote()).ToArray(); if (parts.Length < 1) return; // having acquired the parts, see if there's a matching method MethodInfo mi = null; if (cachedMethodInfo.ContainsKey(new KeyValuePair<string,int>(parts[0], parts.Length - 1))) mi = cachedMethodInfo[new KeyValuePair<string, int>(parts[0], parts.Length - 1)]; else { var methods = GetType().GetMethods().Where(m => m.Name.ToLower() == parts[0] && m.GetParameters().Length == (parts.Length - 1)); if (methods.Any()) { mi = methods.First(); cachedMethodInfo.Add(new KeyValuePair<string, int>(mi.Name, parts.Length - 1), mi); } } if (mi != null) { var pars = mi.GetParameters(); object[] ps = new object[0]; if (pars.Length == parts.Length - 1) { // try building a parameter array ps = new object[pars.Length]; for (int i = 0; i < pars.Length; i++) { var par = pars[i]; var source = parts[i + 1]; ps[i] = ConvertString(source, par.ParameterType); } } // parameters ready - call it try { mi.Invoke(this, ps); } catch (Exception ex) { } } }

You’ll notice that the MethodInfo for a particular method is cached: I keep a Dictionary<KeyValuePair<string,int>,MethodInfo> with the key being a pair of the method name and number of parameters.

Does it work? It sure does, and the evaluations run in real-time: the graph gets re-rendered (and this is GDI+ rendering, mind you) after each keystroke. I am still polishing the details, but here’s an illustration of something more complex:

As I said, this extension to my app is still in the works, but already I feel like I’ve added value to the application. All withoug concerning myself with F#, DLR, MPS and other exotic ways of implementing DSLs. All it took is some C# debugging, and here we are!

Correlation of pay and benefits

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Tue, 06 Oct 2009 22:11:00 +0300

One of the problems I experience when planning projects (well, not a problem exactly) is this idea of applying equal rates to unequal programmers. Sure, all men and women are created equal but, as it happens, programmers are not: some are indeed better than others. This leads to a certain small issue, since customers typically think of ‘juniors’, ‘seniors’ and ‘leads’, whereas I, being on a more humanist (or, as Spolsky says, ‘anthropological’) side, think of developers as, ahem, people.

So why am I talking about this problem? Because I want to market services of people emphasizing that they are not good, but brilliant. I mean, what’s the point of having a really kick-ass developer if their time is worth the same as someone who is just good? That’s right – there is no real point.

This applies to the issue of career planning: how can a person strive to ‘go beyond’ if we are unable to support it financially? I mean sure, people can raise their own profile by participating in user groups, conferences, etc., but it’s all a bit meaningless if they can’t afford a house to live in or a car to drive.

Okay, so here I go: I want to reward good developers with money. The better a developer is, the more money their talent should earn the company, and the more money I want them to earn. I’ve read all the postings about money being far from #1 on anyone’s priorities but, honestly, I don’t know who writes those things, and who does this research (oh, wait, I used to do it).

Seriously, lots of companies think that infrastructural investments are the key. If you walk into a Swedish-owned company in Russia, you’ll likely see very comfortable office planning, large screens, in other words, good support for creature comforts. Yes, those are important, but some people think that’s all that is important.

Coming back to the issue of how to reward great people. I suppose one way is to pay them out of your profits, but that really is cheesy, because it’s customer appreciation that is important. I would love people to bring in more revenue, but developers enjoy it when customers, not their boss, tells them that their software is effing great. In short, two simple necessary conditions form the synergy that makes development a viable career:

People telling you that your code is good
Progressive correlation between how good your code is and how much you earn

The above conditions are a dream. I’ve never worked in a company where the above holds. Hopefully, it’s the kind of company I’ll manage to create.

Sending a Bitmap to a web service

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Fri, 02 Oct 2009 19:55:00 +0300

It turns out that System.Drawing.Bitmap cannot be serialized accross the wire, which means that you need to convert it to a byte[] on one end, and re-serialize on another. This is, of course, done using MemoryStream, but it’s very hard to get the encoding right on the sending end, so the best bet for the web service is to use the Bitmap format:

using (var ms = new MemoryStream()) { bmp2.Save(ms, ImageFormat.Bmp); using (var c = new UploadServiceSoapClient()) { string result = c.UploadImage(ms.ToArray(), "secret_key"); Clipboard.SetText(result); MessageBox.Show(result); } }

What the above implies is that a fairly large chunk of data (width × height × 4) is sent accross the wire. But, on the other hand, you can write a more predictable image-saving function:

using (var ms = new MemoryStream(bitmapData)) using (var bmp = new Bitmap(ms)) { // make a filename and save Guid g = Guid.NewGuid(); string filename = string.Format(@"c:\temp\{0}.jpg", g); SaveImageAsJpeg(bmp, filename); return string.Format(@"http://tempuri.org/{0}.jpg", g); }

Anyways, I don’t know if there’s a more efficient way of doing this but, at least it works. Oh, and the encode-and-save function should make for a nice example in defensive programming:

private static void SaveImageAsJpeg(Image bmp, string filename) { if (File.Exists(filename)) File.Delete(filename); var codecs = ImageCodecInfo.GetImageEncoders().Where(c => c.MimeType == "image/jpeg"); if (codecs.Any()) { EncoderParameters ep = new EncoderParameters(1); ep.Param[0] = new EncoderParameter(System.Drawing.Imaging.Encoder.Quality, 100L); bmp.Save(filename, codecs.First(), ep); } else bmp.Save(filename, ImageFormat.Jpeg); }

The road ahead

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Tue, 29 Sep 2009 17:58:00 +0300

I have spent one year working on TypograFix, adding little bits to the program every day, making it better and better. But now, I’m tempted to completely change the rules of the game by rewriting the application, employing a completely different strategy. Why? Because I’ve encountered a (fairly stupid) problem that I feel unable to solve with the current way of doing things.

Status Quo

Right now, TypograFix is a very powerful Html postprocessor, which turns ad-hoc HTML into something that can actually be published somewhere. I have published over 100 articles online using the system, and have fine-tuned it to the extent where it meets 99.99% of all my needs, with the extra 0.01% covered by hand-editing HTML or having to dig into the source code and changing things.

So far, the program does what it is supposed to do, but it has a few flaws:

Inefficient design
Even though I have coded the program with the best practices in mind, the transformation script itself is one massive switch statement that goes through dozens of cases. The transformation function itself forms a kind of anti-pattern as far as programming goes, because its maintenance by someone other than me would be a nightmare.
Lack of unit tests
When I started writing the program, I didn’t apply TDD. I know better now but, fact is, the transformer messes up sometimes. I have no integration tests, either, so I can’t say whether a new feature breaks old ones or not. I just hope that it doesn’t but, in reality, I’m not sure.
Lack of consistency
Instead of going for the best language and framework available, I went with C# and WPF. This choice is dubious, mainly because I don’t really need WPF for such a simple UI. In fact, this application is emphatically not the type that one needs the power of WPF for. Also, C# is inefficient for graphics, which led me to add C++ code to the application at a later stage just so I could get faster rendering with Direct2D and support for OpenMP and the like.
No support for uniform format
Turning HTML into HTML is just one of the many things that I need to do on a daily basis. But I seem to have failed to account for the possiblity of, say, publishing a book from the same materials.

The last point is particularly important because – guess what – I’m going to write a book! And, seeing how I have no intention whatsoever to use DocBook (don’t ask me why), I feel that I need to have a tool that is capable of producing both Html and, say, PDF or Word from a single source. Sounds neat, huh?

So What?

I’ve decided to make another version of TypograFix. This time, I intend to target not just the web (that’s easy, as practice indicates), but also the print medium. After all, producing nicely-made PDF documents is already a pretty good incentive, and seeing how Word isn’t up to the task (will it ever support the kern feature?), it might make sense to make something proprietary.

So what I’m thinking of is a structured document editor. The idea isn’t exactly new, but good execution is something of a rarity. Here’s what I’m thinking:

Client-server architecture
Fast, idiot-proof client (either RIA or ‘thick’)
Really good tech choices—especially languages
A thought-out architecture

Let the fun begin!

Didactic inefficiencies

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Sun, 27 Sep 2009 21:28:00 +0300

Even though I gain most of the information through the internet, I beleive that there is a fundamental problem with presenting information this way. Sure, a large proportion of content maps perfectly to text & image mediums, but I for one beleive that whenever you are describing some process or other, you should present it in a way that stays coherent as you move on.

Here’s a case in point. I was reading an article regarding the implementation of the Unit of Work pattern with NHibernate. At some point, I realized that the code has lost its internal consistency such that things simply would not compile. This is bad because, from my perspective, it keeps me second-guessing about what the author meant at a particular instance.

So here’s the deal: on the one hand, you want to present the solution in small pieces, but on the other hand, you need to offer people the possibility of viewing the whole thing. So far, this hasn’t been done well by anyone, apart from the boring ‘download source code’ links which aren’t really helpful because by the time you unzip the stuff and open it in Visual Studio, the train of thought is lost.

I beleive that software can and should overcome this problem. We already have teaching systems like Captivate that help us make things more interactive. If only we can somehow make it work with a program like Visual Studio, well, I think it would be really cool.

Meanwhile, I’ll be thinking of how to improve TypograFix to present information better.

More fun with C++ and DirectWrite

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Fri, 25 Sep 2009 23:02:00 +0300

It may appear somewhat crazy for a .Net developer, but I’ve spent the whole of today practicing TDD with C++. My goal – writing a parser that can interpret my own custom notation for OpenType and turn it into rendered text via DirectWrite.

The darn thing took a whole day of my life!

But, hard work and perseverence have paid off. TypograFix is now capable of understanding OT markup. What do I mean? Okay, here’s an example. Suppose I write the following:

[30,i,sw]A[/i,sw]nother [sc,24]Day In[/sc] [Constantia,i]Paradise

On my machine, here’s what I actually get:

Essentially, what I’ve made is a markup-driven rendering system that can create perfect CT/OT text. This fits nicely with my original plan of adding image generation – now I have two rendering methods, one for automatically replacing headings (already using it in places!) and this, the so-called ‘freeform’ text. So far, I have added support for

Font size & face
Italics & swash
Small capitals
Ligatures

What’s next, then? In terms of TypograFix, I’m very happy with what I have already, but I will be working on adding support for directed graphs (and no, not using GraphViz), and maybe support for mind maps. If you have access to the private distribution of TypograFix, you have every reason to be excited!

I might also resume work on DevTalk, though the complete lack of any actionwhatsoever from other project members is downright demotivating.

Block-level formatting in DirectWrite

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Thu, 24 Sep 2009 09:55:00 +0300

Having figured out how to render basic text using DirectWrite, I decided to go further and try to figure out how I can apply block-level formatting to paragraphs. Amazingly, it ended up being very easy. Here’s how it goes.

If you consider IDWriteTextFormat to be the thing that applies basic formatting (font family, size, bold/italic/underline) to a block of text, then IDWriteTextLayout is its older, much more featured cousin. This interface lets you apply block-level formatting. Specifically, what it lets you do is apply OpenType features. Let’s take a look at how it works by first creating an instance of this type:

if (SUCCEEDED(hr)) { hr = pDWriteFactory->CreateTextLayout(markup, wcslen(markup), pTextFormat, width, height, &pTextLayout); }

You’ll notice that TextFormat participates in the creation of TextLayout, forming a ‘baseline’ for subsequent manipulation. Now that we have the layout, we can also create a Typography object that will let us define the total set of OT features that we want to apply:

if (SUCCEEDED(hr)) { pDWriteFactory->CreateTypography(&pTypography); }

Having a Typography object in play, we can start setting its properties, adding the OT features we want. The syntax is presented below. It does look a bit weird (especially for someone used to C#), but it works:

if (SUCCEEDED(hr)) { if (smallCaps) { DWRITE_FONT_FEATURE feature; feature.nameTag = DWRITE_FONT_FEATURE_TAG::DWRITE_FONT_FEATURE_TAG_SMALL_CAPITALS; feature.parameter = 1; pTypography->AddFontFeature(feature); } }

Now that I have defined small-caps formatting, I need to apply it to a range of characters. The range has a starting point and a character count. In the following code block, I apply it to the whole paragraph:

if (SUCCEEDED(hr)) { DWRITE_TEXT_RANGE range; range.startPosition = 0; range.length = wcslen(markup); pTextLayout->SetTypography(pTypography, range); }

Now, the moment of truth: we need to slightly modify the drawing statement to draw the text layout we have defined:

pRT->DrawTextLayout(D2D1::Point2F(), pTextLayout, pBlackBrush);

That’s it! I bet you want to see an example! Here it is.

Reviewing My First Asp.Net MVC Project

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Sun, 20 Sep 2009 14:52:00 +0300

I am on my way to finishing a first ‘proper’ Asp.Net MVC project (don’t worry, it’s tiny) and I thought I’d philosophize a little bit on what went right and what went wrong.

First, some context. What I wrote is basically a small group management web app – something along the line of Google Groups, except that it’s a bit simpler and uses MVC. I stack was C#, Linq2Sql (don’t ask) and Sql Server 2008 as the back-end.

Syntax

If I had to start it all again, I’d pick the Spark view engine instead of the default WebForms one. There’s something utterly ridiculous in writing SomeLongName sln = (SomeLongName[])ViewState["SomeLongIdentifier"], and Spark is the perfect tool to avoid this. Also, the unnecessary proliferation of <%= if (Some.Constant) { %> ... <% } %> clauses is downright annoying, and affects readability of code. True, Spark doesn’t ‘do away’ with such closes (I mean the curly braces and the # prefix), but at least it makes it way more neat.

Linq2Sql

I have no problems with Linq2Sql. At all. For a small site, it’s what the doctor ordered, and given the small load on the web app itself, I’ve never noticed any lag. Well, you have to bear in mind that even high-load sites such as StackOverflow use Linq2Sql, so it’s not the worst tool in the world. Personally, I consider it pretty decent, though I’m using NHibernate on things like DevTalk.

jQuery

I didn’t know any jQuery to start with, but learning it was a breeze, and the fun stuff available as additions (e.g., jQuery UI or the time picker) make it totally awesome to use when you need really good controls, fast.

Styling and Layout

I more or less messed up with respect to layout. I went with using tables instead of divs for the page structure, which made things tricky on a number of levels. I mean, the site works, but it’s not easy to work with its code, because layouting is done by the table, and you have to play by the table’s rules.

Also, I went through two re-styling phases, trying to get the fonts and colours to a state where they could look stylish yet project an aura of seriousness and professionalism.

Content

Made lots of mistakes regarding content. Dead/broken links, links that weren’t clear enough, content that wasn’t placed in the right spot. The only thing I did well is the administration section – but that’s the part that ordinary users do not see, so I was probably the only person to appreciate it.

One real screw-up is that I used <h1> tags for general-purpose content rather than the site name. I can already see that it has affected the site SEO-wise, and that I must revise this policy. Of course, to do this, lots of links need to be changed. I just hope that I can manage it with the ‘Replace In Files’ feature.

Use of AOP

For some reason, instead of using filters I tried using PostSharp for things like authentication. It failed, and failed so miserably that I was forced to wipe out most of my aspects, pick up a book and basically ‘do it right’. Well, that’s another lesson learned.

Error handling

One small success is that I managed to wire up Codeplex.Diagnostics to the database to provide some really good feedback about error conditions. I’ve been very impressed with the framework, and now I use it with other web projects.

Using Direct2D and DirectWrite in .Net/C# Development

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Mon, 14 Sep 2009 22:00:00 +0300

One of my personal goals is to celebrate several successes every day – even if these successes are fairly small. Today, I succeeded in using the Direct2D and DirectWrite libraries from .Net! It took several hours, of course, but I managed to my .Net code to lock a bitmap and manipulate it in unmanaged C++ code, actually writing some text to the bitmap.

Here’s how it goes. First, you need a System.Drawing.Bitmap created on the .Net side. You lock its bits, then send it over to the unmanaged (‘native’) function:

private static Size RegenerateFreeformImage(ConversionOptions options, string markup, int index) { const int w = 512, h = 256; using (Bitmap bmp = new Bitmap(w, h)) { BitmapData data = bmp.LockBits(new Rectangle(0, 0, w, h), ImageLockMode.ReadWrite, bmp.PixelFormat); NativeMethods.RenderMarkup(data.Scan0, markup, data.Width, data.Height, data.Stride, options.FreeformFontFamily, options.FreeformFontSize); bmp.UnlockBits(data); string filename = Path.Combine(Path.GetTempPath(), options.FreeformPrefix + index + ".jpg"); using (Bitmap bmp2 = FlowBasedRenderer.Crop(bmp)) SaveImageAsJpeg(bmp2, filename); } return new Size(w, h); }

The C++ code picks up on this with the following (rather basic) signature:

MYAPI void RenderMarkup(BYTE* dst, LPCWSTR markup, int width, int height, int stride, LPCWSTR fontFamily, float fontSize) { ⋮ }

Inside, all operations happen on a separate bitmap object that we create, together with a render target.

if (SUCCEEDED(hr)) { hr = pWICFactory->CreateBitmap(width, height, GUID_WICPixelFormat32bppBGR, WICBitmapCacheOnLoad, &pWICBitmap); } if (SUCCEEDED(hr)) { hr = pD2DFactory->CreateWicBitmapRenderTarget( pWICBitmap, D2D1::RenderTargetProperties(), &pRT); }

Before rendering text to the render target, some preparatory steps are needed:

if (SUCCEEDED(hr)) { hr = pDWriteFactory->CreateTextFormat( fontFamily, NULL, DWRITE_FONT_WEIGHT_NORMAL, DWRITE_FONT_STYLE_NORMAL, DWRITE_FONT_STRETCH_NORMAL, fontSize, L"", &pTextFormat); } if (SUCCEEDED(hr)) { pTextFormat->SetTextAlignment(DWRITE_TEXT_ALIGNMENT_LEADING); pTextFormat->SetParagraphAlignment(DWRITE_PARAGRAPH_ALIGNMENT_NEAR); } if (SUCCEEDED(hr)) { hr = pRT->CreateSolidColorBrush( D2D1::ColorF(D2D1::ColorF::Black), &pBlackBrush ); }

Finally, we render text to the bitmap. What’s show below is the simplest case – it’s worth bearing in mind that DirectWrite can do really amazing OpenType wizardry that’s a bit too complex to show here.

if (SUCCEEDED(hr)) { pRT->BeginDraw(); pRT->Clear(D2D1::ColorF(D2D1::ColorF::White)); D2D1_SIZE_F rtSize = pRT->GetSize(); pRT->DrawText( markup, wcslen(markup), pTextFormat, D2D1::RectF(0, 0, rtSize.width, rtSize.height), pBlackBrush); hr = pRT->EndDraw(); }

Finally, here’s something that’s amazingly easy to do: we just copy the pixels from the bitmap we rendered to to the Bitmap we passed from .Net. Does it ever get more simple?

WICRect r; r.X = r.Y = 0; r.Width = width; r.Height = height; hr = pWICBitmap->CopyPixels(&r, stride, sizeof(Pixel) * width * height, dst);

Of course, this addition was written for my TypograFix pet project. Meanwhile, let me leave you with the fruits of my labour. The image below was generated using Direct2D & DirectWrite.

Ahh, can life possibly get any better?

Community and Project News

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Fri, 11 Sep 2009 00:45:00 +0300

Some more stuff about user groups: don’t know why I’m writing in English, but still… first, Ineta will now (probably) happen once a week. That’s right – the idea is to meet every Friday at 6, give or take. I’m not sure what variety will be there, but the forthcoming two sessions will cover ClickOnce and SharePoint respectively. I suspect there will be more Micro Framework-related stuff to come.

The second bit of fun is that we are planning a meeting on P/Invoke for the Spbalt.net group. We’ve got a new, cosy location (it’s got a TV instead of a projector, which is cute) that we plan to use. Should be fun.

Some news about the projects I’m working on:

We’re almost ready to release the smart client we’re building (sort of). I think an early release is in everyone’s best interest. This project will target a huge segment of the Russian development community, so we’re a bit paranoid about quality. But I reckon we can suffer at least one non-epic fail without hitting the dirt. Ooh, I’m excited.
The other project, which is a web platform for developers, is slowly taking shape at http://devtalk.net. This project nominally has about 20ish developers, so in theory you’d expect it to progress quickly. In reality, I see it going live in 2010.

Lastly: I’ve been off the grid for a while presentation-wise, but after the Spbalt.net talk I’m thinking of presenting on Ineta, seeing how it’s spinning up to be an active community again. I doubt I can do a tech-savvy intro into technology X, but some advanced stuff, like F# or Linq2Events might be kind of fun to do.

Unwelcome Advice

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Wed, 09 Sep 2009 22:04:00 +0300

One unpleasant side effect of representing some sort of business rather than yourself as an individual is that, suddently, a lot more people begin to lecture you (or make suggestions) regarding how software business should work. It would seem that, as if by magic, many developers have just turned into specialists regarding recruiting, project management, acquisitions, etc. and have all felt a strange, inexplicable compulsion to clobber me with advice that consultants would typically give in an attempt to win some business for themselves.

Take a break, you guys!

Seriously, I don’t mind people telling me what works and what doesn’t – but only if it a) comes from experience; and b) this experience is relevant to what I am doing. Failing that, everyone should just get a cup of coffee because – guess what – we’ve done software projects before, we’ve managed, nobody got hurt, so we’re not exactly kids who need to be educated about how the real world works. Beleive me, we know!

Pseudo-expertise of the “Fowler wrote that…” form is an annoying, pernicious trend that many people are substituting for real experience and real answers. If you ever feel compelled to educate some ‘poor misguided enterpreneur’ about his ‘erroneous ways’ in the software world, why not go into business yourself and do it the right way? This way, at least, you’ll do a lot more good – both to yourself and to others.

Like Asp.Net MVC or Silverlight? We need you!

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Mon, 07 Sep 2009 15:32:00 +0300

We’re building an online community for Russian-speaking developers. We need, ahem, developers in Asp.Net MVC, Silverlight, AJAX/JQuery; we also need testers, SEO people and ideologists. There’s already more than 20 members on the team. You can find the project here: http://code.google.com/p/runet. Contact me or leave a comment below if you want to participate.

TypograFix and Local Image Generation

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Sun, 30 Aug 2009 16:29:00 +0300

A while back, I made a decision to add support for image generation in TypograFix. Basically, the idea is to get headings and other image elements with generated text made automatically by the program. An example can be seen here. So what did I learn?

First, 64-bit development with Visual Studio turned out to be absolute hell. To begin with, all the code I wrote on 32-bit Windows failed to work. It took me several hours to figure out why Visual Studio was using a 32-bit compiler with a 64-bit linker. I even had to open the Add/Remove Features window to check that 64-bit support was, in fact, installed. Also – stupidly enough – VS’s internal settings messed up not just the Microsoft C++ compiler, but also Intel’s one, much to my dismay.

The second piece of complete nonsense was that basic use of DllImport does not work reliably in 64-bit. You end up having to do dumpbin /exports and specifying the full mangled name of the function. Which is simply ridiculous, as there seems to be no reliable alternative to doing this.

But then of course everything failed due to 64-bit pointer arithmetic. I didn’t have time to check, so I stuck with slow System.Drawing.Bitmap-based manipulations. They worked, and I managed to make an article (a translation of something I published on CP) with image-generated headings and comments replaced by neat inserts with Google-like rounded corners. Which is good.

Oh, as for generating files, I found a really simple solution. When working locally, I put them in the temporary folder. Then, when I save the file, I also save all the images, all ready for FTP and whatnot. Yep, I do set Jpeg quality to 100% because who cares about bandwidth?

TypograFix will no longer be updated on ClickOnce. I have invested far too much time into the project to offer it for free. If you can help, you’ll get a copy.

How to unit-test asynchronous operations?

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Sun, 23 Aug 2009 16:57:00 +0300

I recently had to write unit tests for asynchronous operations – specifically, files being downloaded off the internet. But how do you write a test which checks the result of an async operation? You can’t just subscribe to some XxxCompleted event and do Asserts there because your test runner would have exited the test method by then.

The solution? Block the test method:

[TestFixture] public class MyTests { private ManualResetEvent waitHandle; [Test] public void TestAsyncPageDownloading() { waitHandle = new ManualResetEvent(false); ⋮ wdp.GetWebDataCompleted += wdp_GetWebDataCompleted; wdp.GetWebDataAsync(new Uri("http://nesteruk.org/blog"), new object()); waitHandle.WaitOne(); // blocks unit tests until asserts run } void wdp_GetWebDataCompleted(object sender, GetWebDataCompletedEventArgs e) { StreamReader sr = new StreamReader(e.Stream); string s = sr.ReadToEnd(); Console.WriteLine("Here's what we got:"); Console.WriteLine(s); // the following assert will be called by the test runner Assert.Contains(s, "Dmitri", "My webpage should have my name."); waitHandle.Set(); // unblocks unit test } }

Maybe there’s a better way to do this, though?

Making DSLs in F#

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Mon, 17 Aug 2009 08:54:00 +0300

Download EstimationDSL - 1.52 KB

Introduction

Source Code

Problem Statement

End Users
Continous Process Improvement

Choosing a Language
First DSL Statement
Handling Lists
Referencing with Strings
Greater Fluency
Generating the Project
Conclusion

Introduction

If you’re like me, you are already fed up with people throwing the term ‘DSL’ around without showing a good example of how it’s done and where they are used – not to mention giving a decent, human-readable description of DSLs that doesn’t allude to extraneous concepts (I’m talking about things like Oslo or MPS, mainly).

Okay, what on Earth is a DSL? A DSL is a way of defining domain-specific (you could say industry-specific, but it can be much narrower) logic using English instead of a programming language. The obvious benefit is that non-technical people can edit the DSL, without worrying about curly braces, semicolons, that sort of thing.

In this article, I’m going to show how to use the F# programming language to make a simple DSL that helps with software estimation. Just so you know, it’s a real-life example, with a more complex version of the DSL being used at our company. All right, let’s hit it!

Just to say upfront, this article doesn’t present book-perfect F# code, the reason being that it doesn’t really matter since F# code is our end-product. I can think of a dozen ways to improve the F# code presented, but, as I said, that’s not the purpose of this exercise.

Source Code

The attached code is a single .fs file, since I would probably get flogged for posting a VS2010 solution. I hope you know what to do with it. To run it, you need to have Project 2007 installed on your machine – otherwise it won’t work. Good luck!

Problem Statement

When someone wants software written, they typically contact a software development company with something called an RFP, short for “Request for Proposal”. Depending on the level of detail of this RFP, the code shop can either do a detailed estimate or give a ballpark figure. Barring cases of clients wanting dedicated teams or having vague requirements, the estimate the code shop makes is a fixed-price project timeline. Yeah, I know it doesn’t sound particularly agile to people, but in situations where the prospective client has done lots of up-front design, it actually makes sense.

Anyway, someone has to do the estimate, i.e. partition the project into tasks, give them duration, assign resources (= people) to tasks, define milestones, et cetera. This sort of estimate can be done in a program such as Microsoft Project – a good choice in our case, since automating Office from an F# app is easy.

So why a DSL then? Well to be honest, you can probably do an estimate at the 10^th of the time it takes to draw up the GANTT chart with all the reorderings and point-and-click mechanics of Project. Not only that, but typically you need to apply some sort of logic (i.e. use your brain, ugh!) to make sure the project plan is nice and balanced (I mean resource utilization and such). Having a DSL means you can optimize and autogenerate the plan. With a DSL, you can stick as much business logic into your planning procedure as you want – for example, if your company has a process database (this is CMMI Level 4-ish, btw), you could try validating the plan against empirical data.

Are you sold on the DSL idea for estimates? If not, here’s another boon: integration. You can integrate Project with other systems to get more value from existing data. For example, if you run Dynamics CRM, you can tweak resource pricing for a particular client in order to put better developers on the features they consider important. It all sounds very pompous and BI-ish, but that’s life of a code shop.

End Users

In most code shops, barring a few exceptions, estimates are done by project managers (PMs). These guys are sometimes techies, sometimes not, so you can’t expect them to know programming. But you can certainly expect them to be able to work with a DSL and then press some magic key to generate a project plan or otherwise involve their DSL scribblings in an estimation BI scenario.

Continous Process Improvement

At the risk of sounding cheeky, if you are constantly working on your (say) estimation DSL, improving it and tailoring it, it’s an excellent opportunity for improving your business processes. Think about it – as a code shop, you can use idle dev resources to improve the efficiency of your business. Is this great, or what? I think it is, anyway.

Choosing a Language

With the problem out of the way, let’s think of a solution. You can certainly make a free-form DSL language and write a parser, but it’s kind of tedious. A simpler solution is to use a programming language which looks so English (no cultural bias here – feel free to use Japanese or any other language) that the end user won’t know the difference. Of course, some language syntax will creep into the DSL, but its level varies.

Popular languages for DSLs include Boo (which Ayende is trying to make popular), Ruby and F#. Boo is extremely powerful, and for cases where you need metaprogramming support (not our scenario), it’s fantastic. Ruby I know next to nothing about, so no comment. Now, F# is a popular language, and a first-class citizen of .Net infrastructure (as of VS2010 especially). So we’re going to look at making an infrastructure in which PMs can easily write project estimates.

Let me issue a small disclaimer: being a language geared towards immutability, F# looks a bit weird when used with highly mutable concepts, such as a project which can accumulate tasks or milestones. This weirdness can be easily compensated by writing the DSL data structures in C# and then using them from F#. However, for this example, I’ll use F# excusively.

First DSL Statement

I’ll try to keep this simple. Let’s start by defining a project with a name and start date:

project "Write F# DSL Article" starts_on "16/8/2009"

The above is a completely legal F# statement. It’s basically a function call to a function called project which takes 3 parameters. The first parameter is a project name – no, you can’t avoid the quotes here unless you want to parse stuff yourself. The second parameter is a dummy – a constant whose value is unimportant; its only purpose here is to make the spec readable and BDD-esque. You’re welcome to expand the starts_on keyword into two separate parts, but I prefer not to overdo it, especially when there’s a risk of keywords creeping in. The third parameter is the project starting date as a string.

Beleive it or not, the DSL uses OOP constructs in order to manage the constructed project. For example, we have a Project class which is our DSL representation of a project. It’s shown below. Skipping ahead a bit, make sure your types don’t collide with other assemblies’ types. After all, MS Project assemblies might just have a Project type too.

type Project() =
  [<DefaultValue>] val mutable Name : string
  [<DefaultValue>] val mutable Resources : Resource list
  [<DefaultValue>] val mutable StartDate : DateTime
  [<DefaultValue>] val mutable Groups : Group list

I did warn about the strange syntax, did I not? The above is F#’s way of making public fields. You’ll notice also that I use list types instead of the System.Collections.Generic ones. It doesn’t really matter what you use for the DSL so long as it works.

Our DSL will support just one project which will be at ‘global scope’, so to speak:

let mutable my_project = new Project()

The naming convention is a bit ad hoc here save for the fact that, when we get to the end of the spec, we need a statement to actually do something and my_project is a nice identifier name for that action. But now, we can finally show the project statement from earlier.

let project name startskey start =
  my_project <- new Project()
  my_project.Name <- name
  my_project.Resources <- []
  my_project.Groups <- []
  my_project.StartDate <- DateTime.Parse(start)

There. I have probably revealed 90% of what DSL construction is like. You can close this article and go off exploring right now, since you already know how it all works. In the rest of this article, I’ll be showing a few F# implementation details.

Handling Lists

Work in projects is done by resources (not very gratifying, is it?). A resource is a particular person (“John”) with a particular job title (“Junior DBA”) and a particular hourly rate ($65). A project keeps references to resources via a Resource list (see, F# is human-readable). Let’s look at the definition of Resource:

type Resource() =
  [<DefaultValue>] val mutable Name : string
  [<DefaultValue>] val mutable Position : string
  [<DefaultValue>] val mutable Rate : int

I’m abusing F# once again, but at least the structure is easy to work with. Now, a resource definition is also part of our DSL, and might look as follows:

resource "Dmitri" isa "Project Manager" with_rate 140

The above statement employs the same trickery as project except that it acts on an already-existing global variable my_project:

let resource name isakey position ratekey rate =
  let r = new Resource()
  r.Name <- name
  r.Position <- position
  r.Rate <- rate
  my_project.Resources <- r :: my_project.Resources

Resources and all other lists we use end up being listed in reverse order. It’s not a problem though – we reverse them when the time comes. If you don’t like it, use List<T> instead.

Referencing with Strings

The next concept of our DSL I want to introduct is a group of tasks. A group of tasks is typically done by one person to maintain, ahem, cognitive cohesion. We define a group as follows:

group "Project Coordination" done_by "Dmitri"

To put things in context, let’s look at the Group class:

type Group() =
  [<DefaultValue>] val mutable Name : string
  [<DefaultValue>] val mutable Person : Resource
  [<DefaultValue>] val mutable Tasks : Task list

A group references a particular resource, which our DSL specifies only as a string. Problem? I don’t think so:

let group name donebytoken resource =
  let g = new Group()
  g.Name <- name
  g.Person <- my_project.Resources |> List.find(fun f -> f.Name = resource)
  my_project.Groups <- g :: my_project.Groups

Notice how, unlike with LINQ, we don’t have to call Single() after searching for the right resource.

Greater Fluency

Last but not least, we define tasks. Now, isn’t it great when you can say, for example, the following:

task "PayPal Integration" takes 2 weeks

In fact, you can. This type of fluency is achieved by judiciously defining timespan constants so that their values are meaningful. For example,

let hours = 1
let hour = 1
let days = 2
let day = 2
let weeks = 3
let week = 3
let months = 4
let month = 4

The values do not matter so long as they are distinct. Now we can define a task…

type Task() =
  [<DefaultValue>] val mutable Name : string
  [<DefaultValue>] val mutable Duration : string

… and add it to the project:

let task name takestoken count timeunit =
  let t = new Task()
  t.Name <- name
  let dummy = 1 + count
  match timeunit with
  | 1 -> t.Duration <- String.Format("{0}h", count)
  | 2 -> t.Duration <- String.Format("{0}d", count)
  | 3 -> t.Duration <- String.Format("{0}wk", count)
  | 4 -> t.Duration <- String.Format("{0}mon", count)
  | _ -> raise(ArgumentException("only spans of hour(s), day(s), week(s) and month(s) are supported"))
  let g = List.hd my_project.Groups
  g.Tasks <- t :: g.Tasks

Notice that for each timespan I slightly change the way the duration is phrased so that Project is capable of eating the spec. The dummy expression above tells F# that count is an integer – I could have defined it explicitly, of course, but I’m just too lazy. Oh, by the way, notice how easy it is for us to find the current (i.e., last) task. Because we’re using F#, this task is actually first in the list, so we can just call List.hd.

Generating the Project

We’ve got everything and are ready to generate the project. The following (somewhat cheesy) command does it:

prepare my_project

Now, I’m about to show you the whole prepare definition, which uses the Project API to make the, umm, project. Notice how succint F# is:

let prepare (proj:Project) =
  let app = new ApplicationClass()
  app.Visible <- true
  let p = app.Projects.Add()
  p.Name <- proj.Name
  proj.Resources |> List.iter(fun r ->
    let r' = p.Resources.Add()
    r'.Name <- r.Position // position, not name :)
    let tables = r'.CostRateTables
    let table = tables.[1]
    table.PayRates.[1].StandardRate <- r.Rate
    table.PayRates.[1].OvertimeRate <- (r.Rate + (r.Rate >>> 1)))
  // make root task with project name
  let root = p.Tasks.Add()
  root.Name <- proj.Name
  // add groups
  proj.Groups |> List.rev |> List.iter(fun g -> 
    let t = p.Tasks.Add()
    t.Name <- g.Name
    t.OutlineLevel <- 2s
    // who is responsible for this group?
    t.ResourceNames <- g.Person.Position
    // add tasks
    let tasksInOrder = g.Tasks |> List.rev
    tasksInOrder |> List.iter(fun t' ->
        let t'' = p.Tasks.Add(t'.Name)
        t''.Duration <- t'.Duration
        t''.OutlineLevel <- 3s
        // make task follow previous
        let idx = tasksInOrder |> List.findIndex(fun f -> f.Equals(t'))
        if (idx > 0) then 
          t''.Predecessors <- Convert.ToString(t''.Index - 1)
      )
    )

Yep, we finally reverse those backward lists with List.rev – probably not the fastest operation in the world, but I don’t care. All that matters is that the script runs and gives us the results we want – resource definitions, group names and tasks which are grouped and linked. What more could a PM ask for? (Quite a bit actually, but that’s another story.)

A complete project definition can therefore look like this:

project "F# DSL Article" starts "01/01/2009"
resource "Dmitri" isa "Writer" with_rate 140
resource "Computer" isa "Dumb Machine" with_rate 0
group "DSL Popularization" done_by "Dmitri"
task "Create basic estimation DSL" takes 1 day
task "Write article" takes 1 day
task "Post article and wait for comments" takes 1 week
group "Infrastructure Support" done_by "Computer"
task "Provide VS2010 and MS Project" takes 1 day
task "Download and deploy TypograFix" takes 1 day
task "Sit idly while owner waits for comments" takes 1 week
prepare my_project

Conclusion

This article shows that making a DSL in F# is really simple. Of course, the thing about DSLs is they are domain-specific, so for the domain you choose you might encounter a lot more challenges. Have fun!

Windows 2008 R2 out on MSDN!

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Fri, 14 Aug 2009 23:48:00 +0300

People talk about the release of Windows 7 but, IMHO, today’s release of Windows 2008 R2 on MSDN is more important. Why? Because I beleive developers should have a Server version of Windows installed on their machines. After all, you learn whatever version of Windows you use, and the Server variety is a lot more complex – so you’ll be picking up important new skills that help with enterprise development.

I am downloading the DVD image as I write this. Care to guess what software I’m going to install?

Late-night musings on Russian Outsourcing

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Tue, 11 Aug 2009 00:51:00 +0300

It’s no secret that the wheels of software outsourcing here in Russia have either ground to a halt or have significantly slowed down. It’s not exactly surprising, seeing how most profitable, well-funded code shops were developing software either for financial institutions or for companies whose well-being would be affected (sometimes rather dramatically) by the demise or restructuring of said insitutions. Such as start-ups, for example.

So what’s happening exactly? Well, those outsourcers who didn’t get bankrupted (and some ODCs did get closed down, such as Saxo Bank’s, for instance) have certainly lost their share, have made job cuts and are now sitting it out while waiting for business to return. On the other hand, new companies (such as mine) are springing up all the time, eager to retake the business from the big players as soon as it arises. And why not? The crisis has certainly leveled the playing field, and people’s strategies for attracting new business have changed as a result.

My approach to all of this is two-pronged. On the other hand, we have to be ready to pounce on practically any order that comes out, even if the rates aren’t in our favor (I sometimes wonder why Russian outsourcing exists at all, when freelance sites teem with $15/hour coders), even if we don’t particularly fancy a particular client. Now’s not the time to be picky. Which is why the 2^nd avenue of escape from this crisis is taking on internal (i.e., Russian) customers. This is even more challenging, and leads to two problems: the first being that you can’t just attract Russian business without connections (especially government contracts), and the second being that you probably can’t charge the same rates as you charge your Western partners unless you net someone like Gazprom (which is challenging due to part 1 of this argument).

There must be a select few companies on this planet who make really great software and who charge really decent rates for it. And I bet they’re keeping their secrets very well, let everyone in jump on their bandwagon.

Beware of C-style macros in Boo

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Thu, 09 Jul 2009 12:50:00 +0300

It is strange that the term macro is used to denote the chief metaprogramming facility in .Net. After all, macros in C++ are a lot weaker – they cannot manipulate the AST, in fact, they cannot even manipulate code internally or do compile-time evaluation.

Indeed we, as developers, can use the macro facilities in languages such as Boo in order to produce C-style macros. For example, here’s how one would instrument a Boo repetition macro in C style:

macro RepeatCStyle(c as int): return [| for i in range($c): $(RepeatCStyle.Body) |]

This macro simply serves as a syntactic shortcut, which means that if you call it with

RepeatXxx 3: print "Go"

you’ll basically get the following C# code:

int ___temp14 = 0; while (___temp14 < 3) { int i = ___temp14; ___temp14++; Console.WriteLine("Go"); }

You could have used an ordinary helper function in this case. The beauty of macros is in compile-time evaluation and effective inlining of code. Consider the following macro:

macro RepeatInline(c as int): body = RepeatInline.Body yieldAll body for i in range(c)

This macro is more interesting because it effectively returns three copies of the same statement as an IEnumerable. The end-result is the following C# equivalent:

Console.WriteLine("Go"); Console.WriteLine("Go"); Console.WriteLine("Go");

This is certainly much closer to the macros’ purpose as a metaprogramming tool.

TypograFix Improvements

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Tue, 07 Jul 2009 13:36:00 +0300

My pet project, TypograFix, is about a year old (I could be wrong), and it seems that most issues regarding text transforms have been ironed out precisely to the extent where I can produce HTML that is perfect or near-perfect. Now that the bulk of the work is over, I’m thinking of adding some of the more exotic (and difficult to do) features, including:

Server-side image generation — This is a very vague objective but, at the very least, I should have the possibility to generate, e.g., a histogram or a nice heading just by typing text. It’s completely feasible, though of course this feature requires IsolatedStorage on the client side, and image hosting whenever the article is posted. This is annoying, because we would need to specify the URL or the target folder (i.e., in the app itself you’d have a field called Image Url which would contain something like http://nesteruk.org/images/, to which the filenames are appended.) One other potential feature is the generation of graphs and such.
Better code highlighting — seeing how one cannot use CSS on most publis resources, most syntax highlighting is done in very strange ways, using <blockquote>, <code> and <font> tags. Current customizations inside code blocks are already very rich, though, with support for colours, italic comments, and the like. But I think that more can be done: after all, we’re allowed to use the <img> tag in code, so perhaps we can somehow make text more readable? In actual fact, the current use of the <code> tag is dubious, because the tag offers no value when formatting. A new formatting script for code – if I ever bother to write one – will probably be written in F#.
Stylesheet management — this is important because, although I’ve added some spurious stylesheet-related tabs, it’s not very user-friendly. It would be better to have a manager that could habdle stylesheets and – possibly – even program settings in a way where one could switch from one site’s formatting rules to another’s. Once again, this is a fairly difficult thing to implement correctly.
Custom rules — if there’s anything I’ve learned while publishing online, it’s that ever website or blog system has its own set of mess-ups and blatant developer mistakes, and no matter how good TypograFix is, most systems have flaws that have to be havigated around by hand[1]. Some of these weird features have already made their way into the program – e.g., the option to add blank lines after <p> tags. Having it all in a more manageable fashion would be nice.
Fun with Unicode — there are lots of characters in the Unicode character set that can be abused to make all sorts of strange images and the like. For example, one can emulate Japanese list numbering by using Unicode characters, and then provide an option to repaginate lists using paragraphs prefixed by those glyphs. Actually, what would be good to start with is a fast, JavaScript-driven Unicode browser.
Smarter Cut & Paste — right now, HTML copied off a website gets pasted as plain-text. It would be nice if the program could interoperate with HTML and maybe even RTF/Word clipboard formats in order to preserve at least some typographic information about the data being pasted in. Also – and this would be really good – I think that Excel importing and pasting in Excel or CSV-formatted data should also be processed correctly.
Print production — seeing how TypograFix is good for preparing technical documentation, maybe there is a way to prepare text in a format best suited for importing into InDesign. Of course, text has to be imported without any styles, but at least we can preserve the basic structure (e.g., line breaks) so that it doesn’t have to be redone.

This is a very rough list, but I imagine that I will me making a start on some of these features pretty soon.

Notes

↑ The only exception from this rule is the BlogEngine.net blogging platform which I use.

Working with pure C# auto-properties

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Sat, 04 Jul 2009 00:20:00 +0300

Trying to follow your own advice is something that only sounds easy. The case in point: ubiquitous use of auto-properties. The idea sounds nice, but it doesn't quite work.

public class C
{
public IList MySomethings { get; set; }
}

Having lots of lists in a class is tough: each IList deserves its own List initialization. In an ideal world, you could do the following:

public class C
{
[Dependency]
public IList MySomethings { get; set; }
}
// and then
var uc = new UnityContainer();
uc.RegisterType, List>();

But we cannot do it. So, aside from the option of creating everything manually (ugh) and using Nemerle (I'm almost there), the only way to init those stupid lists manually (I have many) is to write the following:

public class C
{
[LazyLoad(typeof(List))]
public IList MySomethings; // it's a field!
}

Using a PostSharp aspect certainly might do the trick, but it's only simple if it's a field. Nonetheless, since fields can be rewritten by PostSharp into properties, we might get the semantics similar to the following:

public class C
{
private IList mySomethings;
public IList MySomethings
{
get { return mySomethings ?? (mySomethings = new List(); }
// where is the set? :)
}
}

This bit of metaprogramming is naughty: we have a set-less property in place of a field. It's lazy-loaded so none of that null nonsense.

This solution can even be done without AOP. It works, and is moderately readable (I hope everyone recognizes the ?? operator). It's not very elegant in the sense where I might want to init this stuff from a container but then again, in my case all I need are empty lists. As a future task, I'll investigate Nemerle for a simpler, DSL=like way of defining these structures.

On a happy note: I found some code that used INotifyPropertyChanged and refactored it, removing any mention of that interface completely. AOP rules!

Promotion of engineering culture

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Mon, 29 Jun 2009 14:59:00 +0300

Seeing how I’m constantly looking for ways to promote .Net and software development in general, I’ve been thinking about ways in which the .Net community can be galvanized into action – i.e., how can people be motivated to write articles, post blog entries, meet in professional conferences, host worshops, et cetera? Putting this in the context of the city I live, it would appear that what I’m trying to achieve is impossible.

It seems somewhat foolish to try and project Western ideals onto a Russian culture. It doesn’t seem to work.

Well, it works to some extent. But not in the way one would like – there’s a fundamental lack of engineering culture, however we try to tell ourselves the opposite, and whatever we do is essentially a bit like trying to promote Communism – a goal that seems hopeless given the public’s general disinteterest. And while we won’t give up (after all, plenty of people are interested in programming to the extent they will visit our meetings), it does affect the strategy that can be chosen given the current social as well as economic climate.

I see two things happening. First, the group will contrinue to grow at a very small rate (I doubt we’ll ever have >50 people unless we cheat), so what we need to do is focus on the quality of the meetings rather than, say, on getting ourselves funding or even a better room.

The second thing I see coming is a shift of focus towards creating an environment where engineering culture can be nurtured in such a way that it doesn’t get significantly moderated by the fact that we live in Russia and not elsewhere. This is tough – there are some companies (e.g., this one) that manage to create great developer atmosphere and whatnot, but I’m after something more inclusive where the place you work doesn’t matter. It should be cool enough to just be a .Net developer.

Unfortunately, engineering culture can only be created in the context of a company, not a community. In other words, what makes great engineers great appears to be, at least given the situation, to be large amounts of money. We need to be able to afford changes financially before we try proposing them. Of course, this implies levying the burden from the decrepit and corrupt education systems that teaches people using last-century textbooks and whatnot. And that makes the whole proposition even harder.

To sum up, we won’t change the world: we’ll make it fun for ourselves and a few people but without business incentives, promotion of engineering culture is doomed to fail with all but the most fanatical of people. And I guess this is okay. After all, our jobs might one day end up at India or China, in which case we’ll pat ourselves on the back and say thanks that we didn’t invest into IT more than we should have.

More user group activity at St. Petersburg!

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Fri, 26 Jun 2009 15:38:00 +0300

Apart from caring for the wonderful St. Petersburg Alt.Net group (my baby), I’ve also been asked (volunteered, however you put it) to help run the St. Petersburg .Net User Group, which is an Ineta-affiliated enterprise. Unlike the Alt.Net group (which has just had its 8^th meeting, with ~20 people attending), the .Net UG has been somewhat quiet, with few meetings and hardly anything extraordinary. That’s about to change.

The idea behind Ineta’s group is rougly this: showing that Microsoft stuff is cool. I mean, that’s why we’re in this business of .Net – because it’s cool. But simply saying ‘it’s cool’ isn’t, in itself, cool enough – we need to show how to get the coolness out of tools like Visual Studio and technologies such as WPF. Great examples, lively demos and lots of fun are ways to guarantee that people who have never programmed .Net would want to move their carreers in this direction.

But what about the competition? Well, honestly, there isn’t much – except for Java, who was the topic of focus at the Sun TechDays here in St. Petersburg. Hundreds of people from St. Petersburg, Moscow and other places around Russia attended. It was a real high-profile show, with foreign speakers extolling the virtues of JavaFX and whatnot. I was there – it was a little crowded but fun overall.

The third developer camp is of course the C++ camp, largely driven by embedded, rather than desktop, development. As far as I know, the C++ people do not have a ‘user group’ of any kind, partly due to the fact that the C++ establishment is by its very nature more conservative and less agile than either C# or Java.

Whatever the case may be, I personally am excited about the new opportunity. We’ll see if I can make something out of it

Too Many Parallelization Frameworks?

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Sun, 21 Jun 2009 00:18:00 +0300

I think there's far too much stuff going on with respect to parallelization on the .Net scene. Just off the top of my head, I can think of Task Parallel Library, Axum, all those weird Spec#-esque languages (okay, they aren't exactly being released, but still), the Concurrency Coordination Runtime (which, let's face it, is similar to what Spec#/Polyphonic C#/Cω are doing, which is join calculus), and then there's things like join macros in Nemerle (I beleive they are called 'chords' by many people now). And let's not forget F# async workflows. But wait, we also have Intel releasing a huge unmanaged (plain C++) stack on us as well. And Microsoft! So the number of three-letter acronyms (PPL, TBB, MKL and IPP) is sharply on the rise, but are we actually better coders because of it? Do we really have time to learn it all?

I don't. I wish I did. Lots of stuff is a lot of fun, but some of the stuff is just a rehash of the same idea over and over. For example, we've always had task-level parallelism, we were just either too lazy to do it properly, or we were feeling helpless and waiting for some miracle framework that will give us a 'fair' ThreadPool (called TaskManager, as it happens) that will distribute work properly because, apparently, ThreadPool doesn't get it quite right. And async workflows? Come on, we have the AsyncEnumerator that does it in C#. Exploiting the F# workflow semantics is very cute but, given the abrupt change in the coding style that's required, I'm not sure it's worth the bother. I can download web pages asynchronously in C# too, thank you very much.

Then there's data-level parallelism, that ubiquitous Parallel.For that we now have (or are about to have) as a way of blind separation of data into chunks. But, seriously, as soon as this tech started getting previewed, lots of people wrote the same implementation for .Net 2.0. So we could have had that too, ages ago.

And let's not talk of instruction-level parallelism because, really, .Net could easily support it (hey, Mono does), but for some reason all that SIMD goodness is lost on the .Net code. Which is a pity because sometimes it is a real benefit, and I for one am tired of using those weird __m128 types and lots of pointer arithmetic and such.

And don't get me started on Axum and the join semantics.

Just before I go off and stop ranting, I have to take a shot at PPL (Parallel Pattern Library), too. Seriously, we already have OpenMP support (which is fantastic, and 3.0 ought to be even more brill), and TBB has been out - what - a year, maybe? Maybe less, but introducing PPL into VS2010 seems like a last-ditch attempt to get people to not spend money on the Intel compiler and parallelization stack.

To sum things up (because this is starting to sound like a rand instead of a normal blog post), I personally am sticking with OpenMP, SSE/2 and the Intel stack for all the parallelization stack. P/invoke only looks scary, but you only need to learn it once. As for .Net 4.0 and all that Task/Future business - well, we shall see what comes of it. But, beleive me, the people who want to get good parallelization and multicore scalability are getting it already. And to those who are still doing their matrix math in C# on a single thread - well, I leave you to your own devices.

A Crazy C# Question

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Tue, 16 Jun 2009 00:50:00 +0300

Today I really fell over while trying to answer a question about a (fairly stupid) C# expression. It went something like the following:

Consider the following code:

int i = 0; while (i < 10) { Console.WriteLine(i++); } while (i >= 10);

This code:

Will not compile
Will print numbers from 0..9
Will print numbers from 0..9 and then 9 forever
Will print numbers from 0 to ∞

Something like this. At first I thought this wouldn’t compile, but in fact my SnippetCompiler ate it just fine. I thought I was going insane. Then I assumed that this construct is, in fact, allowed (maybe a bug in the C# language spec?) and it acts as a double-ended do, cycling the precondition while before cycling the postcondition while, thus yielding the answer 4.

And then it finally hit me: a while loop doesn’t need a comma after it. What this meant is that you could write the code as follows:

int i = 0; while (i < 10) { Console.WriteLine(i++); } // empty while loop! while (i >= 10);

It was a trick question, of course, but it goes to show that vigilance against blatant trickery is often more useful than scientific analysis. Another lesson learned.

Are non-automatic properties evil?

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Sat, 06 Jun 2009 15:33:00 +0300

Ever since the introduction of automatic properties in C#, I’ve been thinking about whether they should, in fact, be the only type of property that’s allowed. The reasoning behind the idea is very simple: if something magical is happening inside an auto-property (i.e., something beyond the typical store/load), then this something is a piece of domain logic or a non-functional requirement that has somehow crept into the entity/persistence layer. The typical result of this is code like the following:


public int Age {
  get {
    return age;
  }
  set {
    // copious amounts of voodoo
    if (age != value)
    {
      age = value;
      NotifyPropertyChanged("Age");
    }
  }
}

This code is creepy for several reasons, the main being that end-users of this code won’t have a clue as to why the Age variable is behaving the way it does. Unless you document that it prevents self-assignment and notifies on change, that is. And even if you do that, would people then assume that every property in the class behaves this way? You never know, really! You’ve just introduced lots of ambiguity that is unresolvable without either a good look at the code or, failing that, Reflector. The above example is only the tip of the iceberg, because really, many people know about self-assignment checks and the INotifyPropertyChanged interface. But what if we do even more hackery inside the property?


public int Age {
  get {
    return person.Age;
  }
  set {
    // even more obscure voodoo
    if (value >= 16)
      person = new Adult(value);
    else
      person = new Minor(value);
  }
}

More trouble! Now you have some real voodoo happening inside the property. What if someone else decides to use the person field without knowledge of what your Age variable does? And what if every property in this class needs to be done this way, but developers forget about it? They are doomed! My take on making properties is to simply write the following:


public int Age { get; private set; }

That’s right – I make Age pseudo-immutable, i.e., only assignable from within the class. That way, we don’t get into a situation where people haphazardly assign the value from several different threads. Of course, we also lose all the interesting stuff like self-assignment checks or change notification, but we can get them back – all we have to do is sprinkle some AOP on top of the class, and we’re good to go. For example, we can declaratively turn every property into a WPF-style dependency property, all without having to use the propdp snippet and having extra static XxxProperty fields in our code. In fact, there are several more things you can do with properties. First, you can use fields instead! After all, PostSharp (if that’s what you’re using) doesn’t care what to rewrite, and making a field is easier—but you do lose declarative immutability. What I mean is, you can just write the following:


public int Age;

You might have FxCop complain about a public field, but feel free to ignore it. Public fields are perfectly okay in the general case too, of course, but that’s another story.

Visual Studio 2010 Beta 1 Screenshots

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Wed, 20 May 2009 12:25:00 +0300

I’ve been using Visual Studio since version 5. In those days, I ran it on a computer with 16Mb (!!!) RAM and a 166MHz processor and was, for the most part, happy with the performance. At the time, MFC was the way to make attractive GUI apps, but I also spent a lot of time obsessing over C++, replacing the compiler and the STL with versions that allowed me to weave that Generative Programming magic that was becoming popular.

Now, a decade down the line, we have a new release of Visual Studio. I have barely touched 2010, but with Beta 1 released I felt like downloading it off the subscription and having fun with it.

Installation

It’s weird, but there were two checkboxes for installation – to install support for .NET development and C++ development. That’s it. It’s totally weird. In other respects, the installation is the same as Visual Studio 2008, installing about 15 different components (such as .NET Framework 4, which casuese a restart) one after another.

I’m not a fan of these large 4Gb undiscriminated installs. Lots of stuff, such as Sync Framework and SQL Server should be installed only if you work with those technologies. I’m using a WinXP virtual machine, so these extra features carry a price in terms of both the waiting time and the performance costs.

First few screens

The opening screen is a re-hash of the same ideas that have been around since VS2005. I had to collapse the Solution pane because the whole thing wouldn’t fit on my 800×600 window.

One thing that VS and other programs haven’t learned to handle well yet is the absence of an internet connection. It’s not a given! And I don’t really want to be reminded that I’m missing on the RSS feed that goes into Visual Studio. Thanks but no thanks.

The project system is pretty much the same, with the idea that there is only one language (C#) and all languages fall into that elusive “Other Language” category. Care to guess what languages are kept there? C++, Visual Basic, and F#.

Making a WPF app

Just for fun, I decided to make a WPF app. One thing I noticed immediately is that the WPF property pane was properly done, with special editors and all.

It’s hard to notice that the editor uses WPF unless you try something like selecting text, shown below.

I was kind of expecting closing braces to ‘pop up’ like they do in XCode (the Mac IDE), but alas. The Add Reference dialog is, so far, still that same boring (and horribly inefficient) window. Notice that Blueprints, the tech that is meant to succeed Guidance Automation, is already in the GAC for some reason. I’m guessing it will be included in 2010.

Extra windows

There’s now a Call Hierarchy window that lets you determine who calls a particular method, and which other methods this one calls:

The Code Definition pane is even more useful – it can show you the actual body of the method you’re interested in. In our case, it can open the .g.cs file and show us the body of the (typically hidden) InitializeComponent() method:

One cute thing is ‘watch floats’ or whatever they are called. Basically, you can hover the mouse over a variable while debugging, and then detach the property strip, letting it stick on the screen as a kind of Post-It note:

That’s as much as I have explored so far… might post more screenshots later.

What is Data-Driven Design?

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Sun, 17 May 2009 18:55:00 +0300

When asked about the meaning of life, one colleague of mine commented once that “I don’t know, but I know that nobody else knows either.” The same approach, I suppose, could be taken with respect to data-driven design: it isn’t clearly defined (good luck on Wikipedia), and actually means different things to different people. Consequently, the best thing I can do is describe what it means to me, thereby making it a kind of ‘Dmitri’s data-driven design’. Still, given the relative scarcity of information on this paradigm (that Wikipedia article is a good indication), I think every little helps.

The idea of data-driven design, for me, is simple: data-driven design is a paradigm in which some nebulous data (which is, predictably, stored somewhere) drives all, or a large portion, of an application. In other words, the application infrastructure is malleable, and aligns itself to the needs of the data. Here, we can specify two different patterns of application:

Static data-driven design is akin to code generation: you have some data (an abstract model), and this data is transformed before compilation to produce an application, or a significant part of it. As an example, abstract data definitions in XML can be transformed via XSLT/XQuery/T4/whatever in order to produce necessary stored procedures, entity objects/mappings, and the user interface. In this respect, of course, data-driven design appears to be synonymous with either code generation or metaprogramming, depending on how you look at it.

Dynamic data-driven design is when data continuously defines how your application looks. For example, if a UI is generated dynamically from a database at run-time, you’ve got dynamic data-driven design. Admittedly, the dynamic approach is more powerful since it allows you to project changes to data immediately, instead of when the application is recompiled. However, this approach is also a lot more complicated and requires more metadata in the data you’ve got.

I am an advocate of static DaDD, because generation implies a higher degree of malleability. For example, in a dynamic scenario you corner yourself into a particular persistence scenario, and it is very difficult to suddenly change the persistence method mid-project. On the other hand, if your application is a compile-time derivative of some abstract data specification, you get massive benefits in the form of an ability to retarget the application to whatever language or platform you see fit. In a similar dynamic setting, you would need to re-engineer the persistence layer, unless of course you started out with persistence ignorance in mind.

Dynamic DaDD is infinitly more complex because you get the interplay of various actors in the system that can also use their data to drive your software. For example, an engineer writes a formula that gets fed as input data to your system. Great, right? Except it doesn’t work that well, and a much better approach is to pre-translate the formula to a programming notation beforehand, so that any bugs or inconsistencies can be discovered before the whole thing is shipped to the customers. After all, how do you prove that mathematical data will drive your system correctly in 100% of cases? That’s right – you cannot.

Here’s an example of dynamic data-driven design: games scripting. The idea behind scripting is that it’s not the programmers’ business to create, say, in-game interactions, but rather the story designers. So, some of the tasks are offloaded to the storywriters, and the scripts they write are plugged in at either compile-time or run-time into the system. When I was young, I remember fiddling with the scripting of Icewind Dale, which involved using a strange language to determine the AI that drove my characters’ actions. For simple decisions it was fine, but for a very complicated logic system, its interpret nature ground the game to a halt. Nowadays, this sort of data-driven thing still works, but if you look at language workbenches such as Microsoft DSL Tools, you’ll see that they are mainly geared towards compiled code, rather than the creation of scripts and an in-code interpreter.

Here’s one example where dynamic data-driven approach makes perfect sense. Suppose you’re writing some sort of Excel integration package where you want end users (who are familiar with Excel) to keep editing their sheets as always, but interoperate with some heavier tech (e.g., BizTalk) in order to aggregate data. In this case, precompiling the formulae in the spreadsheets would be somewhat pointless, because the interaction of formulae on a spreadsheet presents a very complicated, and inherently dynamic scenario of execution.

In my opinion, a dynamic, purely data-driven application only makes sense in cases where the variability of options and the frequency of changes is such that the application has to be constantly adaptable to the end-users’ wishes. But if there’s a need in maximum performance or maximum support for features inherent in the chosen framework, then the precompilation approach is more advisable.

Employer filtering

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Wed, 13 May 2009 21:32:00 +0300

As someone looking for a job, I often view the job description in order to try and analyze whether I should even bother applying to such-and-such a company. Recently, I realized that the process of determining whether a company is good or not can be completely automated. In other words, you can write a computer program that parses the job description and produces a score or some kind of index regarding the employer. Here are a few ideas:

# of listed technologies. Basically, there is a 'golden mean' of a number of technologies that a company should list for an opening. If the number is too small, the company is probably a bit clueless (or doesn't care) regarding how software is made. (For example, they mention C#, full stop. It's just plain silly.) On the other hand, there are job descriptions with 20 different skills required, sort of "C#/F#/C++ COM/IDL WinForms/WPF/WF NUnit TFS/Subversion" et cetera. It's silly too because finding a person with all those skills is near-impossible, so you'll have to compromise anyway. It's not a sign of wisdom.
Old technology. As soon as I see COM in a job description I just ignore it. Don't get me wrong - I've spent a lot of time working with COM/DCOM while at Uni, but I don't remember much, and these are not the skills I want to be picking up again. Whatever company is looking for old tech (e.g., Cobol), they are probably looking for 40-year-old developers who are either too lazy or too inept to actually learn new stuff.
Microsoft bias. I often pick this one up when a job posting is 100% Microsoft tech. Yeah, I'm a .NET programmer, but when I see VsTest on the job description, I just know that the people I will be working with are not very smart. Same goes for requirements of Linq2Sql or EF - I mean, you like the framework, great, you use it. Unfortunately, this also affects some of the tools that I really like (e.g., TFS) because, although TFS is a great idea, let's face it - the companies who do use it are large soulless corporate monstrosities.
Vague wording with respect to things that people probably don't understand themselves. Like when you write "should know how to manage requirements". WTF?!? So some HR chick got an order from high-up to post an IT position with some vague guidelines, she follows them verbatim. Heh, I guess I should be thankful: those vague features are easy to spot. Not sure about computer processing though. "Work in a nice company". Yeah, I like that. Very much. FAIL.

Overall, I think it might be possible to write a parser that can read in the job posting and determine whether it's even worth bothering to apply. Of course, some of the stuff (e.g., the prospective co-workers' ineptness) might not come out immediately, but at least you can diagnose the really bad cases. Oh, and whenever there's a line saying "overwork is nothing common" a big blue-and-yellow swedish flag can pop up. Yeah. That would be cool. Hehe.

Start-up Woes

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Mon, 11 May 2009 23:59:00 +0300

Over the past few months, I have been toying with the idea of opening a start-up company to explore some of the ideas that I've come up with over the years (or, ones that other people came up with). To date, nothing has come of it, and I kind of realize why. Basically, there are several essential hurdles that I haven't managed (yet) to overcome.

I don't have one central idea. Instead, I have about 4 proposals that can be said to be mature, but none of the ideas really 'shines'. Sure, they are all unique in their ways, but not to the extent where they make you go 'wow, this might really work'.
I am somewhat inept when it comes to doing a financial write-up. Truth be told, I'd actually prefer it if someone did this for me.
I am somewhat unsure about being part of a team. I am unlikely to meet anyone who, e.g., speaks English to the same standard as I would like. People sure have computer skills, but that's probably not as important.
I am anxious about starting a company in Russia. Of course, start-ups don't really get registered in Russia (I bet you knew that already), but operating any kind of business here seems like a risk, especially if you have no solid foundations elsewhere. By 'elsewhere', I mean either the UK or the USA.

One other thing that's a little disappointing (but only a little) is that one of my ideas was, ahemm, 'appropriated' by some nice people. Of course, my idea was subtly different than theirs, but it's still a bit weird.

So, just to sum up what's going on at the moment, I am sitting on a couple of nice ideas that are probably not going to work, for whatever reasons. However, all these thought processes might get me to come up with something truly sellable. That being the case, I will try to find an agency that can take the idea without a massive write-up because, truth be told, I am not particularly qualified in order to do one.

Defending code generation

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Fri, 08 May 2009 14:23:00 +0300

As a practitioner of code generation in its various guises, I’m always intrigued by anyone who proclaims that code generation is somehow bad or evil (e.g., this post). Since this subject has been raised in people’s blogs, like, a zillion times, I cannot help but think that anything I write to defend the practice of code generation is like a drop in a lake of tears shed by developers who have had negative experiences with code generation.

So here’s a collection of some (fairly strong) statements that I would offer as my way of explaining whether or not code generation is of any use in business.

Let’s be honest – code generation is very powerful, because as soon as you have figured out a method to save time by automating something, you are producing lots of code with ease. But, like they say in Spiderman, with great power comes great responsibility. What this means is that:

You can forget about modifying generated code. It’s just not kosher. Partial classes, partial methods, text injection and other tricks are used for that. If you need to modify generated code, you’re doing it wrong.
You can forget about computing the Maintainability Index or using a sly tool like NDepend to infer data about the generated assembly. Generated code != hand-written code. The former is not subject to the same quality control procedures, because the quality of generated code is a derivative of whatever template or alorithm was used to make it.
If you are using code generation effectively (i.e., getting an output between 3× and 10× the size of input), then you will never have perfect test coverage. Not even close. Generated code must be tested, but it is not subject to unit testing in the same degree as hand-written code. The reason should be obvious – if the same template results in the generation of 20 sprocs (one for each entity), it is unrealistic to aim for 100% coverage if they all contain variations on a single theme.

Now, with all this power, there comes a substantial skills risk that people write about. Essentially, it sounds something like this: “oh but what if other developers can’t make heads or tails of my code generation scaffolding?” Addmittedly, the skills risk is there, but let’s discuss it in detail:

One side effect of code generation is to simplify things – for example, the models I build in XML are fairly human-readable, and contain a fraction of the ‘footprint’ that would be generated by hordes of SQL and CS files that I would otherwise have. Everything is compact and understandable. So if the source materials for your generation are incomprehensible, well, guess what, you’re doing it wrong.
If your developers are fundamentally incapable of learning a language or two in order to become more effective then yeah, code generation is not for you. If that’s the case though, you might think of changing jobs because, personally, I think it’s more worthwhile working with people that can learn new things instead of stagnating their skills. It’s all about progress, isn’t it?
Likewise, if you’re thinking of later moving product support to another, cheaper location with less qualified developers, then code generation is not for you because the people you hire will start editing generatated code by hand, introducing magic numbers, et cetera. They won’t bother even finding out how the code generation thing worked, and they certainly won’t bother learning something as ‘complex’ as XSLT in order to improve the templates. In the end, you’ll most likely get an unmaintainable mess.

On the reverse side of the coin, code generation mitigates technological and financial risk. Having the model as a simple XML file means that on the output you can get anything – your code can be in C# or C++ or Java, your sprocs can target MySql, Sql Server or Oracle, your ORM can be anything from NHibernate/EF/Linq2Sql to a generated ADO.NET framework. And so on, and so forth. With code generation, you can always change your mind.

Here’s a real-life situation. A company that wanted a WinForms app chose DevExpress controls for its UI layer. Later, we as devs found out that some controls could not be tested by the automated testing tool. As a result, certain types of controls had to be changed back to WinForms. It was a nightmare, took ages, and the end result had bugs where the substitution wasn’t done just right.

To conclude, there’s something to be said for potential abuse of code generation, i.e., using code generation to generate everything (even one-off code blocks) or, what’s worse, using code generation to generate lots of code bloat, such as, e.g., creating a ToString() overload for every single class produced. Naturally, this is ineffective, which only goes to highlight that code generation requires judicious use.

Ideas for an IT book

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Wed, 06 May 2009 18:51:00 +0300

For a very long time, I have been tempted to write a book, save it for the fact that books don’t sell and nobody is going to read it anyway. However, I’m thinking that if I write, say, one page a day, I might actually get somewhere within the course of a year or two. It requires amazing discipline, of course, but maybe I can muster some – after all, I have several journals (Moleskine and Epica) that I have been writing in for about 5 years, every day, without fail. So persistence is not something I’m lacking.

What I’m lacking is something else – a good idea for a book. After all, there are already plenty of books on programming languages (e.g., C#, F#, C++) and technologies. What can I write that could be useful and people would read? Just off the top of my head, here are some ideas:

Write about a language for which no book exists, such as Nemerle or Io. Alternatively, I could write something on mixing and matching languages in the .NET stack. The trouble is, this requires a lot of experience with different languages.
Write about an unreleased or up-and-coming product such as Azure or Visual Studio 2010.
Write about DSLs. There aren’t that many books on DSLs, and I’m sure that many people are wondering why there’s been so much talk of DSLs lately, and many are interested how and where to use them. The most intriguing option is to use JetBrains MPS, which looks to be a complicated and powerful DSL technology (too bad it’s Java-driven).
Write a methodological book on something like AOP or metaprogramming that people kind of like but aren’t sure about.
Write about some tech that’s a bit far out than people are used to. For example, there is a book on DocBook and, let’s face it, not many people (even experienced developers) know what DocBook is, and there’s probably an even smaller number of people using it.
Write an algorithms book. This is a good idea because, apart from going over implementation of various algorithms (e.g., correlation calculation or A*) it’s also possible to cover various programming languages and contrast, e.g., imperative vs. functional approaches. Unfortunately it also requires explaining about the basics of numeric computation, e.g. what double.Epsilon means.
Write a statistics book. After all, collective intelligence of the type used on social networking websites is the kind of material that’s subject to statistics, so it could be a good topic.
Unmanaged parallelization. This is a subject of interest for me primarily because I want to write algorithms that scale, and the whole TaskManager related .NET 4.0 business consistently fails to impress me.

At the moment, I’m not prepared to start writing anything major. I prefer to sit on the ideas and write a few articles to explore the field and try to determine what people really want to read about. And who knows—I might find something that’s worth a long-term effort.

Organizational thoughts

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Mon, 04 May 2009 19:20:00 +0300

Ever since I ended up creating an ALT.NET group in my fair city I have been thinking about where I want all of this to be heading, and what I want the group to become. Currently, the activities of the group are driven by these observations:

The number of seminars and public meetings related to the .NET framework is rather small.
The technical level of such meetings is typically ‘beginner’ rather than ‘practitioner’, meaning that it’s not very interesting for people with work experience.
The level of people’s exposure to new technology in the .NET stack is greather than their exposure to existing technology. This is driven mainly by marketing concerns, and is understandable.

Consequently, the idea behind the group is to produce a .NET-oriented environment that caters to continous practice and excellence in .NET matters, and that is fit to house experienced professionals as well as people starting out (with the focus on the former). There is a lot to be said about the viability (and usefulness) of such an organization, mainly from the socio-economic point of view:

The prevalence of outsourced cost models in most cases hamper people’s professional development. In fact, with certain notable exceptions, such enterprises are entirely incapable of fostering professional development because their income is based on skills declared rather than ascertained. The immediate consequence is that a vast percentage of working programmers will never come to our meetings, not because the level of challenge is high, but because the motivation is not there.
The ‘worshop culture’ that’s become the norm elsewhere is nonexistent. This implies that the open space model simply does not work, because any random group of developers is unlikely to be able to self-organize and come up with a cohesive discussion plan. (I’m wrong about this in the sense that the progressive few developers might. But the .NET community is not the same as, say, a Ruby community, so YMMV, as always.)
Since the local business is essentially 90% outsourcing, it’s not interested in IT communities. This implies that the group will never have corporate sponsorship, which consequently means that the group’s profile can never be raised about ‘enthusiast’ level. This isn’t a problem for me—I’ve made a decision to never spend my own money on the group, so a zero-budget group is a perfect approach for me.

I suppose one advantage the group confers is a potential tie-in between my own company and the group. So, for example, I could provide corporate sponsorship for the group on my own terms. Because, looking at how things are right now, I doubt anyone else will.

New heading styles

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Thu, 30 Apr 2009 14:05:00 +0300

I have replaced post headings with images on my blog. The algorithm for making an image from text is as follows:

Grab heading text and turn it into a legal file name.
Render text using WPF typography and put it in a System.Drawing.Bitmap.
Post-process the bitmap to add subpixel antialiasing.
Save the result as PNG and serve.

Funnily enough, WPF’s terrible antialiasing is a good target for ClearTypification. All I had to do is render at triple-width, distribute the energy using a five-element low-pass windowing filter (ugh, can’t beleive I just wrote that), then coalesce the bitmap back to normal width. Which gives me a good subpixel hinting result (not perfect, mind you, because it’s not grid-fitted).

The next step for this tech is to iron out the bugs and make sure some automatic subsitutions happen (like I had with SVG). Also, all bitmap-processing code has to be rewritten in C++ using OpenMP so that it is rendered a little faster. Not that it matters – after all, the image is generated only once.

Rendering rich typography to bitmap

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Tue, 21 Apr 2009 12:36:00 +0300

In the dark ages of .NET 2.0, if you wanted OpenType text, you basically had to write your own OT engine. But in .NET 3.5, you can hijack all of the OT functionality in WPF rich typography, even if you are working with something like ASP.NET server-side image generation. Here’s how.

To begin with, you need to create the typographic elements that you are going to render:

Run r1 = new Run(text.Substring(0, 1))
{
  FontFamily = new FontFamily(fontName),
  FontSize = fontSize,
  FontStyle = FontStyles.Italic
};
if (char.IsLetter(text[0]))
  r1.SetValue(Typography.StandardSwashesProperty, 1);
Run r2 = new Run(text.Substring(1))
{
  FontFamily = new FontFamily(fontName),
  FontSize = fontSize
};
r2.SetValue(Typography.NumeralStyleProperty, FontNumeralStyle.OldStyle);
Paragraph p = new Paragraph {TextAlignment = TextAlignment.Left};
p.Inlines.Add(r1);
p.Inlines.Add(r2);
FlowDocument fd = new FlowDocument(p);

Now there’a challenge: how to render a FlowDocument onto a bitmap? We need to create a viewer and carry out the measure and arrange steps:

FlowDocumentPageViewer fdpv = new FlowDocumentPageViewer {Document = fd};
fdpv.Measure(new Size(800, 200));
fdpv.Arrange(new Rect(0, 0, 800, 200));

The last step is to render the text onto a target bitmap:

RenderTargetBitmap rtb = new RenderTargetBitmap(400, 100, 72, 72, PixelFormats.Pbgra32);
rtb.Render(fdpv);
PngBitmapEncoder enc = new PngBitmapEncoder();
enc.Frames.Add(BitmapFrame.Create(rtb));
using (Stream fs = File.Create(location))
{
  enc.Save(fs);
}

Of course, there’s a critical step missing: ClearType. Without it, all our beautiful typography is meaningless.

ClearType in Silverlight

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Sun, 05 Apr 2009 21:24:00 +0300

Silverlight (and WPF) font rendering sucks, but it is possible to render text with ClearType on the server side and send it to the applet. The following Silverlight app demonstrates it. Try comparing different fonts and sizes – some are almost identical, but some are strikingly different. Which version is better? It’s up to you to decide.

TypograFix 1.5 Released

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Sun, 05 Apr 2009 00:58:00 +0300

Today I added the last of the features that I wanted to have in TypograFix – the ability to easily add hyperlinks by using the vertical bar |http://like.so| or with text |http://like.so like so|. Thus, release 1.5 has been built and rolled out via ClickOnce. I hope people appreciate it.

Working with SQL Server, hierarchical data and Silverlight

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Wed, 01 Apr 2009 14:24:00 +0300

Introduction
Fun with SQL

A new UDT
Insertion
Descendants and Ancestors

Fun with C#… or not

Building a POCO
Procuring the data
Downloading and using

Download source code (1.88Mb)

Introduction

One of the new features of SQL Server 2008 is support for hierarchical data structures. These data structures can be used to store information pertaining to organization charts, online forums, network topologies, and more. This article seeks to present useful information regarding to the way you can work with hierarchical data in SQL Server.

To compile and run the examples in this article, you will need Visual Studio 2008 and SQL Server 2008.

Fun with SQL

As you may have guessed, we shall be working with data structures that help us basically build trees using SQL Server (as opposed to just ‘lines of data’ that we’re used to). Since we’re working with trees, here are some things we need to know:

Where the root of the tree is.
Who the node’s parents and children are.
The level (depth) of a node in the tree.
Ways of traversing our tree.
Ways of doing indexing over the tree.

A new UDT

In order to get all of this to work, SQL Server 2008 introduces a new UDT¹, hierarchyid (Msdn page). This data type contains the path from the root all the way to the node, and is thus used in lieu of, say, a foreign key, when linking tree structures together. The value that SQL Server writes to hierarchyid is itself unreadable (it’s made using a clever algorithm) but we can use ToString() if we want to get a human-readable representation.

Before we jump into C# (the scary part), let’s play a bit with good old SQL in order to get a feel for what this new feature looks like and what to do with it. Let’s imagine, for example, that we are using SQL Server to keep a hierarchical structure of help topics that can be compiled into a help application². We begin by defining a somewhat unusual table schema³:

create table HelpItem
(
  Id      hierarchyid primary key clustered,
  Lvl     as Id.GetLevel(),
  ItemId  int unique,
  Name    nvarchar(32) not null,
  Content ntext
)

Okay, that’s probably a lot different to what you’re used to, especially if, like me, your first row of (almost) every table is defined an int identity. Having a hierarchyid as the primary key allows us to do different types of indexing over the table, which we will discuss later. The second column, Lvl, is a computed column that calls on a GetLevel() method of the hierarchyid type (remember it’s a CLR type)⁴.

Insertion

To appreciate both the Id and Lvl columns, let’s add some data to our table. This has to be done programmatically:

insert into HelpItem
  (Id, ItemId, Name, Content)
values
  (hierarchyid::GetRoot(), 1, 'My Product', 'How to use my product')

If you’ve never met the :: syntax, it’s basically used for calling class-level (static) methods inside SQL Server – instance methods still use the dot notation. In the above insertion, we just specified the hierarchyid of our first entry to be the root of the tree.

To view the data, you must fiddle the query somewhat. Doing a typical select * is useless because the hierarchyid column will have the value of 0x in every row. Don’t ask me why (I have no idea). To get a readable value, we simply use the ToString() method I mentioned earlier, and execute the following query instead:

SELECT [Id].ToString()
      ,[Lvl]
      ,[ItemId]
      ,[Name]
      ,[Content]
  FROM [Hierarchical].[dbo].[HelpItem]
 
// here is the output:
 
Id Lvl ItemId Name       Content
-- --- ------ ---------- ---------------------
/  0   1      My Product How to use my product

As you can see, the textual version of the root Id element is a simple slash (/), and the root node has a level of 0 (zero).

Descendants and Ancestors

We’ve managed to insert the root through some hackery, and we also ended up providing our root element with an ItemId instead of auto-generating one (which can be a problem). Now, to add more elements to the hierarchy, we need to get acquainted with a new method called GetDescendant.

Essentially, GetDescendant generates the hierarchyid of a new element that is a) a descendant of the element you called it on; and b) is located between the two elements provided to GetDescendant as parameters. Of course, either of the parameters can be null, which means that there’s no constraint on where the element can occur.

That said, let’s add a child element called ‘Introduction’ as a child of our root element:

insert into HelpItem
  (Id, ItemId, Name, Content)
values
  ((select Id from HelpItem where ItemId = 1).GetDescendant(null,null),
   2,
   'Introduction',
   'Some general info about our product')

Just to reiterate, what’s happening above is that, for our new element, we chose a descendant root from the parent element. We passed two null values, this making the placement of the new item unconstrained. Turning the hierarchyid value of the new item yields the value /1/. This piece of information is fairly useless per se though, because we are mainly interested in the actual (binary) hierarchyid value.

In addition to being able to get at the descendant of a particular element (which is essential for random insertion), it’s also possible to get an ancestor N levels up by calling GetAncestor(N) on the element you’re interested in. The result is, predictably, a hierarchyid of the parent N steps up the tree, or null if there aren’t that many elements up the tree.

Another method available is IsDescendantOf, which checks whether our hierarchyid is a descendant (direct or indirect) of the hierarchyid provided as parameter.

All right, if you’re like me, you are probably bored with all this theory. Let’s try to make a hierarchical database and bind it to a simple Silverlight app.

Fun with C#… or not

It really amuses me when someone uses the phrase modern ORM framework – if frameworks were modern, we would have support for hierarchyid in both Linq2Sql and Entity Framework. That said, it’s not the case: ORM frameworks do not support our brand new type out of the box.

After playing around with trying to get a Linq2Sql T4 tempalate to play nice with SqlHierarchyId (it worked, but Linq2Sql failed to agree with the idea of an hierarchyid identity column), I decided to let it go at that and work without an ORM⁵, especially seeing how I only have one very simple table and I’m interested in working with Silverlight which, as you probably know, cannot directly use an O-R mapper anyway.

Building a POCO

Seeing how we aren’t going to get an auto-generated entity class (big deal), we have to make one ourselves. This is even more appropriate if we take into account that a Silverlight app cannot use an ORM layer directly, and we end up interfacing through a web service. Here’s our service’s HelpItem definition:

[DataContract]
public class HelpItem
{
  internal SqlHierarchyId Id;
  [DataMember]
  public int ItemId;
  [DataMember]
  public string Name;
  [DataMember]
  public string Content;
  private List<HelpItem> children;
  [DataMember]
  public List<HelpItem> Children
  {
    get
    {
      return children ?? (children = new List<HelpItem>());
    }
  }
}

This structure is tricky, and requires some explanation. Some of its elements cannot be transmitted over WCF and so are kept private – for example, you cannot use the SqlHierarchyId type in Silverlight anyway, so it doesn’t get its own [DataMember] attribute. On the other hand, in order to transmit the hierarchy we have to condense it by aggregating the items in their respective parents – thus the need for a Children collection (which is lazily created, to boot).

Procuring the data

To get the data from the server, we use good old-fashioned SqlConnection and SqlCommand classes – the way things were done when .NET was just appearing. At the moment, we define just one function, which returns all the elements as an object graph:

[OperationContract]
public HelpItem GetHelpItems()
{
  SqlConnection dbConn = new SqlConnection(
    "Data Source=(local);Initial Catalog=Hierarchical;Integrated Security=True");
  List<HelpItem> results = new List<HelpItem>();
  try
  {
    dbConn.Open();
    SqlCommand cmd = new SqlCommand(
      "select Id.ToString(), ItemId, Name, Content from HelpItem order by Id.GetLevel()",
      dbConn);
    using (var reader = cmd.ExecuteReader())
    {
      while (reader.Read())
      {
        HelpItem i = new HelpItem
        {
          Id = SqlHierarchyId.Parse(reader.GetSqlString(0)),
          ItemId = reader.GetInt32(1),
          Name = reader.GetString(2),
          Content = reader.GetString(3)
        };
        results.Add(i);
        // make sure its parent knows
        foreach (HelpItem parent in results.Where(r => r.Id.Equals(i.Id.GetAncestor(1))))
          parent.Children.Add(i);
      }
    }
  }
  finally
  {
    if (dbConn.State == System.Data.ConnectionState.Open)
      dbConn.Close();
  }
  return results.Count() > 0 ? results[0] : null;
}

The only really ‘smart’ thing we’re doing above is using LINQ and the GetAncestor() method of SqlHierarchyId in order to add each child to its respective parent⁶. You’ll also note that, even though we return but a single element, this element aggregates everything else, thus yielding the whole object graph.

Downloading and using

Unfortunately, the POCOs we get from the service aren’t very useful on their own – they have weak support for things like editing, so we copy them into corresponding client-side HelpItem structures:

private static void ProcessChildren(HelpItem orig, HelpService.HelpItem curr)
{
  foreach (HelpService.HelpItem hi in curr.Children)
  {
    HelpItem i = new HelpItem { Name = hi.Name, Content = hi.Content };
    orig.Children.Add(i);
    ProcessChildren(i, hi);
  }
}
void hsc_GetHelpItemsCompleted(object sender, GetHelpItemsCompletedEventArgs e)
{
  // convert and assign
  if (e.Result != null)
  {
    HelpItem root = new HelpItem
                      {
                        Name = e.Result.Name,
                        Content = e.Result.Content
                      };
    ProcessChildren(root, e.Result);
    items.Clear();
    items.Add(root);
  }
}

This copying procedure lets us use better collections and more features in the entities – something that will come in handy later. Meanwhile, we can use our acquired elements in the model – and here’s what we get:

On the next iteration, we shall continue by looking at how edited tree items can be posted back to the server.

Notes

^ A UDT is a user-defined data type. Unlike the primitive server types such as bit and int, a UDT is a CLR type, thus having typical CLR methods such as ToString().
^ The idea comes from Andrej Tozon’s blog entries which describe how to work with the tree control in Silverlight.
^ As you might have guessed, UI-driven schema creation or editing isn’t supported by tools such as Management Studio. This is due to the absense of editors (and inability to do, say a string → hierarchyid conversion) as well as lack of support for computed columns.
^ When we come to interfacing with an ORM, we shall completely get rid of this generated column. It is only shown here as an illustration – in real life, it’s probably easier to simply create generated columns as elements of a partial class that ORM frameworks make.
^ In case you’re wondering, I looked at EF, Linq2Sql and NHContrib.
^ One interesting thing to note is the == operator doesn’t work on SqlHierarchyId – at least not how one would expect. Equals() did the trick, though.

F# exploration notes

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Sun, 15 Mar 2009 22:36:00 +0300

Having spent some time programming in F#, I though I’d post some observations that might be useful to C# programmers.

F# has an ambiguity with regard to the return statement. It’s not required in a function, but when you are in a match block, you do need it:
```
match root.Name.ToString() with
| "rss" -> return ProcessAsRss(root, url)
| "atom" -> return ProcessAsAtom(root, url)
| _ -> return 0
```
F# has something against null. Apparently, you cannot really have null objects without hacks. Which was a problem for me because I was trying to write
```
let id = null
```
to use it as an output parameter in a sproc. I got so confused I had to ask on SO, and ended up writing somewhat bizarre code:
```
let id = new Nullable<int>()
```
Somewhat predictably, the int? syntax is not working – F# tells me that the use of a ? symbol after an identifier is reserved for future use.
You cannot call a function that returns something and then continue doing things. Without the return statement, F# thinks that a returning function is the return value. Which leads to code like the following
```
let _ = ctx.ChannelUpsert(ref id, url,
                          ⋮
                         )
```
The underscore stores the end result, even though we don’t really care.
One of the really useful methods for downloading from the web was removed from F# in order to allow for compatibility with Silverlight and Mono. This function defines a matching pair of Begin and End clauses for a web request, thus allowing asynchronous operation:
```
type WebRequest with
  member x.GetResponseAsync() =
    Async.BuildPrimitive(x.BeginGetResponse, x.EndGetResponse)
```
The async keyword is really fun syntactic sugar that also exists in C# (in a way) via PowerThreading’s AsyncEnumerator, though this usage is somewhat more clever. Essentially, in addition to an immediate variable-definition operator let, F# also has a ‘whenever’ operator let! (with an exclamation mark). This operator appears precisely at an async junction, i.e., on the left-hand side of an async call:
```
let UpsertFromUrl(url:string) =
  async {
    try
      let r = WebRequest.Create(url)
      let! resp = r.GetResponseAsync()
      use stream = resp.GetResponseStream()
      use reader = new StreamReader(stream)
      let xml = reader.ReadToEnd()
      ⋮
  }
```
It seems the whole point of async ops is to fire them off on arrays (sorry, sequences). It is this way in PowerThreading, and it appears this way in F#.
```
let UpdateExistingChannels =
  use ctx = new PrioryDataContext()
  ctx.Channels
  |> Seq.map (fun f -> UpsertFromUrl(f.Source))
  |> Async.Parallel
  |> Async.Run
```
The funny thing is: I wonder how many people will use this instead of, say, doing Parallel.Invoke() instead. ← This is C# 4 library stuff, btw. We’re not even there yet.
F# has a use keyword to match C#’s using, but it’s not clear where its scope is. F# does not really delimit its scope with curly braces (exce, and indenting doesn’t help either. Figures.
Strings lose their == operator, so you can no longer compare them this way. Actually, I was also comparing string and XName and the like, and I suspect that I might have panicked. Also – and this is annoying – the compiler did not list the error! It just failed to compile and I had to check the build output to see what was going on. How crazy is that?!?
I managed my first F# hack!!! (err, of sorts) I was trying to do an upsert on all elements of an array, and then return the array, but whatever I did, F# was complaining that my List.iter code was not to its liking. So, instead, I used List.map taking the return value of my upsert sprocs and returning the length of that list. All in one statement. Witness my perfection:
```
  first.Elements() 
  |> Seq.filter(fun f -> f.Name.Equals("item"))
  |> Seq.map(fun f -> ctx.ItemUpsert(ref itemId, id, 
                                     ⋮
                                    )
  |> Seq.length
```
Programming is great!

TypograFix: the end of an open-source project

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Fri, 13 Mar 2009 22:20:00 +0300

I like the principle ‘quit while you’re ahead’. It’s indicative of a strong character, of someone showing restraint instead of attempting to get more water out of a dry well. In this vein, I have decided to cease development on my beloved TypograFix project. Not because there isn’t anything interested that can be added, not because there isn’t more .NET goodness that can be exploited, not because I’ve written a Godlike application what does everything possible.

I want to quit while I’m ahead.

The program works. It does what I intended it to do. I use it for all my HTML work nowadays, and it covers 99% of situations. I could keep thinking of more features to add, but then I would be like those other app developers who have already got a good product, but want to keep users upgrading, so they begin adding useless features that maybe 1% of their target audience will appreciate. That’s not the way I want to go with my little app.

So, after release 1.5, TypograFix will be in maintenance mode, meaning if anyone logs a bug, I’ll consider fixing it but, unless something unusual happens, development effort will be frozen.

One reason open-source projects are great is that you learn lots of interesting lessons about development. For me, with this project, the lessons were as follows:

C# is not the best language for everything. If I had to do this all over again, I’d write it in F# or Nemerle because of their much better support for pattern matching. Since the bulk of ‘business logic’ is, in fact, pattern-matching against HTML, I could have ended up with much more readable, but also much slower code. Which is okay, I don’t mind slow code for an application such as this.
Doing WPF is narrow-minded. I ended up in a situation where I wanted to use the program on a Mac. Clearly doing it in either a WinForms-esque toolkit like GTK# or doing it on the web (jQuery + web services / Silverlight) would have made more sense. Instead, I went for a heavy, badly refactorable platform. Whatever my next project is, it will most likely run on the web, just so people can access it easier. Right now, I wouldn’t even discount the idea that it will be a Flex app – I’m really tempted to try something new.
The transformation function in this project has a complexity that renders TDD useless. If I had used TDD, I would have to write an insane number of tests to cover eventualities that could reasonably be predicted just by looking at the code. This project is one good example of where test-driven development and – especially – the concept of code coverage is practically useless.
At times, I fell victim of overengineering. I repeatedly caught myself thinking that I ended up adding a lot of superflous features (like being about to write x^3 and have it turned into x³, that even I would hardly ever use. Many nice-to-haves got in the way of really important features that made the application easier to work with. If this was a commercial app, I’d be paying for these mistakes with market capitalization or whatnot.
ClickOnce just works, but I should have done it sooner. Since I went this ‘rich client’ path, I should have done it this way from day 1 instead of packaging the app with an installer or (worse) as a ZIP file for ‘xcopy deployment’. ClickOnce is a lot easier, because people get the updated app automatically without any action on their part. If I ever do a relatively simple thick-client app again, I’d go for ClickOnce to make everyone’s life a little easier.
Leaving minor bugs in is okay. No app is perfect. I know that some features don’t work as well as I’d like, but just because it’s an open-source project doesn’t imply that I won’t do cost-benefit analysis on it. Minor bugs will either get ironed out if they become real annoyances, or they’ll stay there ignored. Either way, they aren’t something to lose sleep over.
Small is beautiful. Simplicity rules. Three large text areas, a couple of links, a disappearing property grid. That’s all you need for a sensible user interface.

Overall, TypograFix became a permanent resident on all my desktop machines, and I’m happy about the way it works to this day. I’m using it to type this blog post right now, and have many slots open for articles and things that I want to publish. One thing I realized is that, with the typography nicely done, the output result can easily be opened in Word or InDesign, should I ever feel the need to publish.

The employer's market?

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Thu, 12 Mar 2009 18:38:00 +0300

If there’s one good thing that has come from the global economic recession, it’s the fact that things will never be the same again in the land of IT. At long last, people are being given a push to re-evaluate the way IT businesses or branches operate, and adjust the rules of the game in order to make more business sense. So, while I sit around waiting for a decent job to come by, here are some observations about how things are changing on the Russian IT market.

Telecommuting is starting to make sense

Now that cost cutting is all the rage, employers are realizing that having people work in offices full-time is not such a good idea: it’s a lot of extra costs that could be negated if you let people work from home. These costs can then be taken as savings or used to increase employee motivation in other, creative ways. Taking myself as an example, my productivity when working from home is several times greater because:

I don’t spend extra 2 hours a day going to and from work
My computers are vastly superior to anything any office can provide
I have a library greater than any a Russian firm can possibly have. (then again, some don’t even have books at all)
My working conditions are ideal. Space, adequate lighting, furniture, cooked food, everything is better

It’s always funny when, oftentimes, I hear from some employer or other that ‘work from home is not real work’ or that ‘it doesn’t count’. I hope such employers change their minds because, with the changing climate, they might be rejected more and more.

The myth of cheap programmers is disappearing

I guess there was a time ‘before my time’ when lost of western businessmen went east with hopes of finding ‘great programmers, cheaply’. However, one thing that Russian companies realized is that competing on price is pointless, because countries like India will always win, especially if you compare hourly rates – if you compare eventual project costs, my guess is things would be different. But that’s now really what I want to talk about.

What I’m talking about is using this ‘economic crisis’ as a pretext to hire good programmers cheaply. It doesn’t work. People just stay at home, work freelance, sit on their savings. People who came to the larger cities from smaller ones simply go back to cut living costs and wait things out. That’s not to say that employers do not try – they do. I personally have been to interviews where the offered rate was 1/2 or 1/3^rd of the ‘expected market rate’ (I’m saying ‘expected’ because market rates have, in fact, gone down). Admittedly, I did enjoy the interviews, but summarily rejected the offers – something I’m hearing from other people looking for lead positions.

Sure, people get hired into PM/Lead roles for measly money. I mean, someone has to take the job, right? And I’m guessing that anyone looking for a very fast ‘career’ climb (notice I’m using quotes here) will get exactly that – a difficult position with inadequate compensation. Hey, so long as everyone is happy, I won’t really complain.

Employers are more picky

There is two observations I have with regard to employment nowadays. The first is, the relationship between emloyers and employees in a hiring situation have deteriorated greatly. I used to see a situation where companies would express concrete desire to hire a particular person, entice them, etc. Not anymore! In 2009, the hiring process is strict, methodical and ruthless. And this is great!

Why is it great! Because, previously, many firms had such a bad process filtering out the losers. Ask a few general questions, get a few sensible answers, hey, you’re hired! But now, nobody is spending a dollar until they are sure that that dollar is an investment, and not a ballast expense. Consequently, if you’re smart, it’s becoming easier to outgun people who, despite having more experience than you, don’t know jack about technology.

There is, of course, lots of stuff missing. For example, employers still do not care about community involvement. They care about projects done, but they don’t ask about talks given or papers written. This differentiating factor is taken out of the equation. Another factor is that nobody bothers to check for references. I mean, seriously, it’s just a phone call to find out whether the person quit or whether they were fired for being grossly incompetent. It seems that Russian HR simply cannot be bothered. Maybe the situation will change some day, who knows?

Also, though social skills are often listed as a desire for a project management position, nobody actually asks to demonstrate in any fashion the fact that those skills are there. In fact, this is another requirement that is completely missing from the job description of most development positions, often to the subsequent detriment of collective work. Still, we aren’t at the point yet where everyone realizes it’s important.

People are more mindful of quality

It’s the same old thing reiterated: if you don’t care about quality, you may as well choose the cheapest outsourcing provider you can find. If you do, it’s a lot more difficult to find the right one. In these challenging times, I find that people are becoming more concerned with the way quality is controlled during software production. TDD, continuous integration, coverage, mocking, dependency analysis – these used to be words of myth that people used to shrug off just ‘so long as it works’. Well, now they are coming to the fore, since there’s a growing realization that a developer that isn’t constantly mindful of the testability of their software is not such a great developer after all.

It has become very hard to measure quality, though, primarily because institutions like the CMM got discredited somewhat, leaving companies with few means beyond the good old chat with the customer to demonstrate that they take quality control seriously. For example, if the only code checking you do on a .NET project is with FxCop, I’m sure you’ll get laughed out of most meetings simply because nowadays, everyone realizes how awfully inadequate any single tool is. Once again, I’m loving this, because the time invested into learning about the different test/mock frameworks is going to start to pay off – hopefully for me and not just for whatever company I choose to work for.

A post-crisis reshuffle is almost guaranteed

If I was desperate to find a job now, my guess is that I would accept any paycheck that would feed me and my family. Many people have done precisely that, which is why the elusive market rate has slipped (it has also slipped because of a strong dollar, but anyone smart enough would have hedged against this anyway). However, if we assume that the recession is about to end at some point (say, within a year), my guess is there would be a reckoning where companies would, once again, be able to pay top dollar (rouble, in some cases) for lead positions. And this would most likely cause the ‘middle management’ layer of IT companies to get reshuffled once again, as large companies with renewed economic strength will headhunt the top talent and get it. That is, if things ever get back to normal.

Bilingual programming

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Thu, 05 Mar 2009 10:41:00 +0300

Sometimes, using English in coding is utterly impossible. For example, when working on a medical app, there’s no way I can translate special terms to English just so I can work with an all-English model. Funnily enough, the design process does not suffer that much. For example, my abstracted Patient entity looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<table name="Пациент" script="•script.xml">
  <field name="ИндексЛечащегоВрача" type="int" filter="true"/>
  <field name="Фамилия" type="string" size="32" null="false" />
  <field name="Имя" type="string" size="32" null="false" />
  <field name="Отчество" type="string" size="32" null="false" />
  <field name="ПолЖенский" type="bool" null="false" text="Пол"
         radio="true" option1="Мужской" option2="Женский"/>
  <field name="Адрес" type="string" size="128"/>
  <field name="Телефон" type="string" size="16"/>
  <field name="ДатаРождения" type="datetime"/>
  <field name="МестоРаботы" type="string" size="64"/>
  <field name="Профессия" type="string" size="64"/>
  <field name="Страховка" type="string" size="256"/>
</table>

This works so long as I keep everything in UTF-8 perpetually. Of course, I also had to alter CodeGenUtils to handle UTF-8 but, end result it that I can manufacture Latin-Cyrillic sprocs from the model:

/* Creates the 'Пациент' table. */
create procedure [dbo].[ПациентCreate] as
create table [dbo].[Пациент] (
  [Индекс] int identity primary key not null,
  [ИндексЛечащегоВрача] int,
  [Фамилия] nvarchar(32) not null,
  [Имя] nvarchar(32) not null,
  [Отчество] nvarchar(32) not null,
  [ПолЖенский] bit not null,
  [Адрес] nvarchar(128),
  [Телефон] nvarchar(16),
  [ДатаРождения] datetime,
  [МестоРаботы] nvarchar(64),
  [Профессия] nvarchar(64),
  [Страховка] nvarchar(256)
);
return @@error
go

SQL Server supports Unicode names, so there’s really no problem. Firing off this stored procedure (once you get over Visual Studio’s bugs) is easy and, predictably, gets you the structure you want. Now, an ORM like the Entity Framework can grab the (cyrillic) tables easily. It won’t be able to rename tables of course, but still, it works:

This is where things get tricky. Entity Framework has a bunch of inherent bugs where it expects Latin key names. However, for the most part, things work well enough. For example, binding works against Cyrillic identifiers just fine:

<StackPanel Orientation="Horizontal">
  <Label Content="Пол:"  />
  <RadioButton Content="Мужской" 
    IsChecked="{Binding Path=ПолЖенский, Converter={StaticResource ic} }"/>
  <RadioButton Content="Женский" IsChecked="{Binding Path=ПолЖенский}"/>
</StackPanel>

The above code is, interestingly enough, auto-generated from the model. Thanks to WPF’s WrapPanel, making groups of controls has never been easier:

HTML parsing with F#

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Tue, 24 Feb 2009 23:33:00 +0300

While working on TypograFix, I managed to get myself into the situation where I have a massive state machine that goes through HTML one character at a time. However, due to its specifics, it starts doing look-ahead using code similar to the following:

case '-':
  if (CannotReplace) goto default;
  // if surrounded by spaces, it's an en dash
  if (i > 0 && i + 1 < text.Length && text[i - 1] == ' ' && text[i + 1] == ' ')
    hb.Append("&ndash;");
  else if (i + 1 < text.Length && text[i + 1] == '-' &&
           (i + 2 == text.Length || text[i + 2] != '&'))
  {
    hb.Append("&mdash;");
    ++i; // ignore the second dash
  }
  else if (i + 5 < text.Length &&
           text[i + 1] == '-' &&
           text[i + 2] == '&' &&
           text[i + 3] == 'g' &&
           text[i + 4] == 't' &&
           text[i + 5] == ';')
  {
    hb.Append("&rarr;");
    i += 5;
  }
  else goto default;
  // same for em dash
  break;
  #endregion

This got me thinking: apart from maybe improving the code by doing ">".StartsWith(text.Substring(i, 5)) kind of code, is there anything that can fundamentally improve the way the code looks? My attention turned, predictably enough, to using F# lists for string processing. (Disclaimer: using lists is probably not the best way of going through strings, but it might just be the clearest.)

F# actually treats strings as sequences (think IEnumerable<char>) instead of, err, lists (think List<T>). Consequently, if you want to pass a string as a list, you would write

Parse(List.of_seq "hello")

However, for a string parsing function, it’s OK to just declare the string parameter in the constructor.

let Proc (text : string) =

Then, we can define some recursive processor that goes through the string one character at a time:

let Proc (text : string) =
  let rec Other html =
    match html with

Now comes the part with matching. Getting something like --> is, however, a bit ugly:

let Proc (text : string) =
  let rec Other html =
    match html with
    | '-' :: '-' :: '>' :: tail -> ("&rarr;" |> List.of_seq) @ Other tail

It’s already looking weird, because the three parts of --> are spread out as list elements. What I can do is declare a temporary variable to store the first few elements of a list (Seq.of_list) and then use string.StartsWith. Which gets me more or less back to where I started.

Returning identity from SQL Server upsert sproc

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Mon, 23 Feb 2009 17:54:00 +0300

An upsert is an update-or-insert operation realized via a stored procedure. Typically, we check the parameters against some criteria, and if the criteria match, we update an existing record. Otherwise, we create a new one. One problem I encountered is getting the primary key (@@identity) of the record after it has been upserted. However, introducing a parameter (@Id) seems to do the trick. Here is an example:

Suppose I have a definition of an RSS channel as follows:

create table [dbo].[Channel] (
  [Id] int identity primary key not null,
  [Title] nvarchar(128) not null,
  [Description] nvarchar(128) not null,
  [Link] nvarchar(128) not null,
  [Docs] nvarchar(128) not null,
  [Generator] nvarchar(128) not null,
  [Language] nvarchar(8) not null);

My upsert operation would look as follows:

create procedure [dbo].[ChannelUpsert](
  @Id int output,
  @Title nvarchar(128),
  @Description nvarchar(128),
  @Link nvarchar(128),
  @Docs nvarchar(128),
  @Generator nvarchar(128),
  @Language nvarchar(8))
as
 begin
  merge Channel as tbl
   using (select
     @Id as Id,
     @Title as Title,
     @Description as Description,
     @Link as Link,
     @Docs as Docs,
     @Generator as Generator,
     @Language as Language) as row
   on
     tbl.Link = row.Link
when not matched then
  insert(Title,Description,Link,Docs,Generator,Language)
  values(row.Title,row.Description,row.Link,row.Docs,row.Generator,row.Language)
when matched then
 update set
  @Id = tbl.Id,
  tbl.Title = row.Title,
  tbl.Description = row.Description,
  tbl.Docs = row.Docs,
  tbl.Generator = row.Generator,
  tbl.Language = row.Language
;
if @Id is null
  set @Id = SCOPE_IDENTITY()
return @Id
end

The parameter @Id is part of the sproc, tagged with the output keyword, meaning that when calling from C#, you’ll need to initialize an int? parameter and pass it by reference. Subsequently, we do the upsert, and if an update happens, we simply set the @Id to whatever table we just updated.

If towards the end the @Id is null, this meanins we did an insert – so we just assign @Id to the identity value of the inserted record. We also return it from the sproc, which is somewhat unnecessary.

Here is the actual usage code:

int? id = null;
ctx.ChannelUpsert(
  ref id,
  getData(elem, "title"),
  getData(elem, "description"),
  getData(elem, "link"),
  getData(elem, "docs"),
  getData(elem, "generator"),
  getData(elem, "language"));

Now that we have the id, we can use it as a foreign key in child upsert operations.

Side note: it turns out that it’s impossible to call custom sprocs (such as this one) automatically through the Entity Framework. How depressing. Well, at least Linq2SQL works.

No Comment

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Fri, 20 Feb 2009 16:45:00 +0300

Adding a Tag property to any object - without AOP

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Wed, 18 Feb 2009 18:22:00 +0300

Here is a class that lets us add a Tag property to any object without using AOP.

static class ExtensionMethods
{
  private static Dictionary<WeakReference, object> tags = 
    new Dictionary<WeakReference, object>();

  public static void SetTag(this object obj, object tagValue)
  {
    Cleanup();
    var found = tags.Select(o => o.Key)
      .Where(k => ReferenceEquals(obj, k.Target));
    if (found.Count() > 0)
    {
      tags[found.Single()] = tagValue;
    }
    else
      tags[new WeakReference(obj)] = tagValue;
  }

  public static T GetTag<T>(this object obj)
  {
    return (T)tags.Where(o => ReferenceEquals(o.Key.Target, obj)).First().Value;
  }

  private static void Cleanup()
  {
    var toRemove = tags.Select(o => o.Key).Where(p => !p.IsAlive);
    foreach (var tr in toRemove)
      tags.Remove(tr);
  }
}

Lazy loading with PostSharp

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Wed, 18 Feb 2009 10:05:00 +0300

Recently I wanted to use PostSharp for some simplistic lazy loading, but ended up hitting a couple of problems along the way. It all started very well – I defined an OnFieldAccessAttribute, removed the field in GetOptions() and overrided OnGetValue() to lazily create the type. My creation code was something like the following:

public override void OnGetValue(FieldAccessEventArgs eventArgs)
{
  if (eventArgs.StoredFieldValue == null)
    eventArgs.StoredFieldValue = Activator.CreateInstance(eventArgs.DeclaringType, args);
  eventArgs.ExposedFieldValue = eventArgs.StoredFieldValue;
}

Big mistake. For an uninitialized field, the declaring type is null. This means that it’s impossible to apply a [LazyLoad] attribute, since the aspect is clueless about the type of object to lazily load. The solution – passing the type :( Here it is:

[Serializable]
class LazyLoad : OnFieldAccessAspect
{
  private readonly Type type;
  private readonly object[] args;
  public LazyLoad(Type type, params object[] arguments)
  {
    this.type = type;
    args = arguments;
  }
  public override OnFieldAccessAspectOptions GetOptions()
  {
    return OnFieldAccessAspectOptions.RemoveFieldStorage;
  }
  public override void OnGetValue(FieldAccessEventArgs eventArgs)
  {
    if (eventArgs.StoredFieldValue == null)
      eventArgs.StoredFieldValue = Activator.CreateInstance(type, args);
    eventArgs.ExposedFieldValue = eventArgs.StoredFieldValue;
  }
}

This takes care of both the type to create and the arguments, so it can be used as follows:

class B
{
  public B()
  {
  }
  [LazyLoad(typeof(A))]
  public A A;
}

You don’t want to know what happend if A is a value type – really, you don’t. But the above code works, and lazy-loads the A variable only when accessed. So far so good, right? Well, yes, except there is a problem: sometimes I want to lazy-load a member, passing this into the constructor. I could make this super-complicated by overloading aspect constructors, but I decided to create a new aspect instead:

[Serializable]
class LazyLoadWithThis : OnFieldAccessAspect
{
  private readonly Type type;
  public LazyLoadWithThis(Type type)
  {
    this.type = type;
  }
  public override OnFieldAccessAspectOptions GetOptions()
  {
    return OnFieldAccessAspectOptions.RemoveFieldStorage;
  }
  public override void OnGetValue(FieldAccessEventArgs eventArgs)
  {
    if (eventArgs.StoredFieldValue == null)
      eventArgs.StoredFieldValue = Activator.CreateInstance(type, new[] {eventArgs.Instance});
    eventArgs.ExposedFieldValue = eventArgs.StoredFieldValue;
  }
}

There. That covers my needs. However, since there’s a myriad of Activator.CreateInstance() overloads, there’s probably a lot more one can do with lazy-loading aspects. Shame about the (somewhat) ugly syntax though – but then again, this lets you lazy-load polymorphically, if that’s your cup of tea. And I strongly suspect that lazy-loading semantics are domain-specific in most cases (they certainly are in my projects), so the above is only an example.

Cross-thread collection updates in WPF

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Tue, 17 Feb 2009 02:50:00 +0300

It’s often very tempting to write your own version of ObservableCollection to accomodate cross-thread calls. However, this is not a very good idea, because you never know what your usage scenario might be. For example, I had to create a binding to an ObservableCollection with set semantics (i.e., duplicates are not allowed) in order to get SQL Server Management Objects to periodically scan the network for SQL Server instances. Even though I already had a MyObservableCollection class ready, it didn’t work for me because I needed to get a result from the conditional insert method, indicating whether it actually succeeded.

As always, I got tempted to use the PostSharp aspect to make the call async and to handle errors. However, what I ended up doing is isolating my async code from the code that still had to happen in the UI thread. The end result – another rather ugly call on the Dispatcher:

[WorkerThread, ExceptionDialog]
private void LoadDatabases()
{
  bool added;
  do
  {
    added = false;
    foreach (string db in State.FindDatabases())
    {
      added = (bool)Dispatcher.Invoke(
        (Func<string, bool>) delegate(string s)
        {
          bool a = false;
          if (!Databases.Contains(s))
          {
            a = true;
            Databases.Add(s);
          }
          return a;
        }, db);
      }
  } while (added);
}

Code like this makes me want to be an Erlang programmer.

AOP-based reporting of exceptions in separate threads

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Sun, 15 Feb 2009 22:35:00 +0300

The WorkerThread aspect is a really cheap way of running a method asynchronously in the thread pool. Sure, you don’t get to pick priority and such, but for a general-puspose background task, it’s perfect:

[Serializable]
public class WorkerThreadAttribute : OnMethodInvocationAspect
{
  public override void OnInvocation(MethodInvocationEventArgs eventArgs)
  {
    ThreadPool.QueueUserWorkItem(state => eventArgs.Proceed());
  }
}

There’s one problem with this method, though – it becomes more difficult to catch exceptions. For example, if you have a VS add-in throwing an exception on a separate thread, VS itself will not catch it and your add-in will crash. The solution? Another aspect:

[Serializable]
public class ExceptionDialogAttribute : OnExceptionAspect
{
  public override void OnException(MethodExecutionEventArgs eventArgs)
  {
    ExceptionMessageBox emb = new ExceptionMessageBox(
      eventArgs.Exception,
      ExceptionMessageBoxButtons.OK,
      ExceptionMessageBoxSymbol.Error);
    emb.Show(null);
  }
}

Rather than using a boring MessageBox to report the exception, I went for a richer user interface – namely, the ExceptionMessageBox that comes with SQL Server. Just what the doctor ordered.

Fun with Wait and Pulse

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Tue, 03 Feb 2009 02:31:00 +0300

Anyone who has used Microsoft Project for time tracking is probably familiar with terms like Finish-to-Shart and Start-to-Start. For example, in a Waterfall model, we say that Requirements have to ‘finish’ (i.e., we must have them) before Coding starts.

The same idea also appears in multithreading – having a process begin only when another has started or is finished. In an ideal world, such problems in C# would be resolved using a clear, sensible API. Unfortunately, even waiting on several processes to complete is not easy. Here’s an illustration:

ThreadStart delA = a;
ThreadStart delB = b;
IAsyncResult arA = delA.BeginInvoke(null, null);
IAsyncResult arB = delB.BeginInvoke(null, null);
// no, you cannot have a pony
WaitHandle.WaitAll(new[]{ arA.AsyncWaitHandle, arB.AsyncWaitHandle});

That’s right – the neat API features such as WaitAll() are unusable simply because it only works under [MtaThread], which ruins the fun for far too many people to be useful.

Pulse & Wait

The static methods Pulse() and Wait() of the Minitor class are a solution to our problem of waiting on many concurrent operations to complete. That covers the finish-to-start scenario, but these mechanisms work for the start-to-start scenario, too. Let’s see how.

Imagine you’re making breakfast, consisting of jam on toast and a cup of tea. Now, toasting bread and making tea are async operations; you can only try to take the jam out once you’ve started toasting the bread; you can only have breakfast when you’ve got jam on toast and a cup of tea. Here’s a graphic representation:

Just in case you haven’t guessed, solid lines are finish-to-start ones, while the dashed line is start-to-start. So how would code for such a scenario look? Well, the first thing you must realize with Wait() and Pulse() is that every operation that waits on something has a lock:

private readonly object MakeSandwichLock = new object();
private readonly object EatBreakfastLock = new object();
private readonly object GetJamLock = new object();

Now, pay attention, here’s the second part of the puzzle: every operation that someone waits on has a bool indicating whether it is done:

private bool MakeTeaIsDone;
private bool ToastBreadIsDone;
private bool GetJamIsDone;
private bool MakeSandwichIsDone;

Okay, if you look at the diagram, the two parts of nomenclature I just mentioned should be obvious. MakeSandwich, EatBreakfast and GetJam all wait on someone – although GetJam does a ‘soft wait’ because it waits for something to start and not finish. Then each function that is waited upon got a bool flag to indicate whether it is done. Hey, we need some indicator, right?

Now here’s the third part of the puzzle: both finish-to-shart and start-to-start operations are enforced by a)locking on your lock; b) doing a Monitor.Wait() if the combination of boolean flags is unsatisfactory. Here’s some code for actually eating the breakfast:

protected internal void EatBreakfast()
{
  lock(EatBreakfastLock)
    if(!(MakeTeaIsDone && MakeSandwichIsDone))
      Monitor.Wait(EatBreakfastLock);
  EatBreakfastImpl();
}

Just to reiterate: to get to eat breakfast, we a)lock on the breakfast lock; and b)wait on that same lock if we cannot proceed yet. Okay, we’re almost there – all we need to know is when to actually set those boolean flags and Pulse(). Well, this depends: if you are doing a start-to-start, you do it at the start of the method; otherwise, do it at the end. Do what? Well, to inform someone you’re done, lock on the target lock (i.e., on the lock of someone waiting on you), then set your Done boolean to true, then Pulse(). For example, after tea is ready, here’s how to tell the EatBreakfast method that it can (theoretically) proceed:

protected internal void MakeTea()
{
  MakeTeaImpl();
  lock(EatBreakfastLock)
  {
    MakeTeaIsDone = true;
    Monitor.PulseAll(EatBreakfastLock);
  }
}

Ordering

Before we move on, here’s the whole Breakfast class (without the Impl() methods, of course):

partial class Breakfast
{
  private readonly object MakeSandwichLock = new object();
  private readonly object EatBreakfastLock = new object();
  private readonly object GetJamLock = new object();
  private bool MakeTeaIsDone;
  private bool ToastBreadIsDone;
  private bool GetJamIsDone;
  private bool MakeSandwichIsDone;
  private bool ToastBreadStarted;
  protected internal void MakeTea()
  {
    MakeTeaImpl();
    lock(EatBreakfastLock)
    {
      MakeTeaIsDone = true;
      Monitor.PulseAll(EatBreakfastLock);
    }
  }
  protected internal void ToastBread()
  {
    lock(GetJamLock)
    {
      ToastBreadIsDone = true;
      Monitor.PulseAll(GetJamLock);
    }
    ToastBreadImpl();
    lock(MakeSandwichLock)
    {
      ToastBreadIsDone = true;
      Monitor.PulseAll(MakeSandwichLock);
    }
  }
  protected internal void GetJam()
  {
    lock(GetJamLock)
      if(!(ToastBreadStarted))
        Monitor.Wait(GetJamLock);
    GetJamImpl();
    lock(MakeSandwichLock)
    {
      GetJamIsDone = true;
      Monitor.PulseAll(MakeSandwichLock);
    }
  }
  protected internal void MakeSandwich()
  {
    lock(MakeSandwichLock)
      if(!(ToastBreadIsDone && GetJamIsDone))
        Monitor.Wait(MakeSandwichLock);
    MakeSandwichImpl();
    lock(EatBreakfastLock)
    {
      MakeSandwichIsDone = true;
      Monitor.PulseAll(EatBreakfastLock);
    }
  }
  protected internal void EatBreakfast()
  {
    lock(EatBreakfastLock)
      if(!(MakeTeaIsDone && MakeSandwichIsDone))
        Monitor.Wait(EatBreakfastLock);
    EatBreakfastImpl();
  }
}

So, how does this monstrosity get fired off? Well, here’s one possibility:

ThreadStart[] ops = new ThreadStart[] {
  MakeTea,
  GetJam,
  ToastBread,
  MakeSandwich,
  EatBreakfast };
foreach (ThreadStart op in ops)
  op.BeginInvoke(null, null);

There’s one caveat with all of this: what if we had ToastBread before GetJam in the list? That’s right – you would have a hanging application. It’s important to remember that Pulse() is only useful is someone is Wait()ing on it. If you cause a situation where Pulse() is fired before the Wait()ing code is running, you’re in trouble. Luckily, it’s a fairly obvious mistake that you’re likely to spot even if you’re not into unit tests.

In Closing

I don’t seriously recommend anyone writing code like the one above – in fact, the above picture and code come from a DSL that facilitates P&W-based asynchronous design and a lot more.

Cheap way of speeding up Visual Studio I/O

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Wed, 28 Jan 2009 21:06:00 +0300

Visual Studio loves file I/O. It really, really does, and it cares very little about caching and the like. What this means is that, when you open a solution file, it always reads it from disk. Which means that the speed of operation when generating lots of files (e.g., via T4 transformations) is typically atrocious. The fact that file I/O in VS does not appear to be multithreaded is another hindrance. In short, it’s a mess.

So here’s a stupid solution that I’ve been using for ages with excellent results – make a RAM disk! By sacrificing a small amount of RAM (I give it 128Mb), you can move your whole solution into RAM, where I/O speeds are far greater – noticeably greater when working with Visual Studio.

To get a RAM disk working, I followed advice on this page to the letter, and got it all set up under Vista. Then, when I need to, I simply snapshot my Subversion projects onto the RAM drive and do my work. If the stuff I work on is sensitive, I can even erase the RAM drive after work – committing beforehand, of course.

Typographic reflections

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Wed, 28 Jan 2009 02:48:00 +0300

Having spent far too much time working on TypograFix, I now have a few moments to recount my experiences – in particular, the useful things I’ve learned.

AOP and Cross-Thread Calls

First, I finally added AOP support to the project. I’m not using AOP for validation, since there’s very little, but I am using two AOP features to drive the app. The first is a [WorkerThread] attribute, which causes a function to be called in a separate thread. It’s probably the simplest, most brainless aspect you can think of:

/// <summary>
/// Makes the method run async.
/// </summary>
[Serializable]
public class WorkerThreadAttribute : OnMethodInvocationAspect
{
  public override void OnInvocation(MethodInvocationEventArgs eventArgs)
  {
    ThreadPool.QueueUserWorkItem(state => eventArgs.Proceed());
  }
}

What surprised me was that this attribute worked surprisingly well – I just put it in front of the DoUpdate() function, and it started running async, fired every 5 seconds by a timer. The only thing that annoys me is that the WinForms WebBrowser control makes a clicking sound whenever I update and, at the moment, I don’t have a clue how to remove it.

My application uses both WPF and WinForms controls (you’ve guessed it – the WebBrowser control), so having this attribute implies that I have to do those ugly cross-thread calls. One ‘gotcha’ I got from this is that not only does setting a WPF control property require the use of a Dispatcher, but getting the property requires one too (which makes sense in retrospect, I guess). This led me to write the following, slightly bizarre, piece of code:

string text = (string)tbOut.Dispatcher.Invoke((Func<string>) delegate
{
  ConversionStatus cs;
  string msg;
  tbOut.Text = Generator.Generate(
    tbIn.Text,
    App.ConversionOptions,
    5000,
    true,
    out cs,
    out msg);
  ConversionMessage = msg;
  Status = cs;
  return tbOut.Text;
});

This is the first time I’ve done a return from an Invoke() on a delegate – and is probably the last time, since I don’t like the notation. I think this piece of code is somewhat hard to read, and is confusing to anyone but me.

The second AOP attribute I used is [INotifyPropertyChanged], which adds support for change notifications to a class in just one line of code (instead of all that event and firing stuff). Annoyingly enough, the PostSharp sample had a bug which took me some time to track down, but I got it working in the end. The attribute implementation itself weighs in at 226 lines of code, which is quite a bit – but then the functionality is so useful – it works on auto-properties just fine!

The only real snag I had (again, due to inexperience) was a question – how do I cast an object to an INotifyPropertyChanged interface if it doesn’t implement this interface at compile-time? Luckily, the solution is really simple – a double cast via object:

INotifyPropertyChanged npc = (INotifyPropertyChanged)(object)App.ConversionOptions;

Incidentally, PostSharp does provide a generic method Post.Cast<> to do this, but I had some trouble with it, so I’m sticking to the double cast for now. You wouldn’t beleive the kinds of options I tried though! For example, I tried to do a GetInterfaces() at runtime to find the object having an INotifyPropertyChanged interface, but to my amazement, it didn’t, probably because it was extended by PostSharp’s own interface wrapper.

Strong Names

Rule of thumb for library makers: provide a strong name. This is something that was missing from AOP-based validation package (which I later decided not to use). Basically, if you make a reusable library and don’t provide a strong name, I can’t use it in strongly-named assemblies myself. And even if I have your source code, I’m a bit too lazy to provide a strong name myself.

Why? Because, apart from having to generate the snk or pfx file, I also have to get the public key and update all the [InternalsVisibleTo] attributes – not in my solution, but in the 3^rd party library one! How crazy is that?

This is definitely something to remember for the future.

Maintainability

I finally had a change to take a look at the metrics that VSTS gives me for the transformation function. Needless to say, the situation is dire:

195 lines of code
Class coupling of 15
Cyclomatic complexity of 192 (!!!)
Maintainability index of 0 (zero)

If I was managing a project and someone wrote this, I’d ask them to refactor it. Clearly, having a massive state machine in one function is not the best way to go, especially seeing how I have conditionals everywhere – that’s right, I’m using the evil goto statement all over the place, because within switch statements, the goto instruction is capable of jumping to the default label – a situation that happens far too often when a match against an expected condition fails.

Although I cannot refactor the method manually, it might be possible to do it with programmatic means – some sort of static transformation can turn this huge switch monstrosity into a series of separate functions (perhaps in a nested class) or even a WF or some sort of DSL. At the moment, I’m happy with the code, so I’m not bothered that much. But it’s something I might do later.

Cross-Platform Plans

Since I spend a small amount of time on a Mac, I’m thinking of transforming this application into something that can run on either Mono, ASP.NET (via AJAX, of course), or Silverlight. (I wish I could consider AIR, too, but I doubt it can understand C#.) The idea is that anyone can use my script to prepare documents for publication.

On a side note, I finally managed to do a screencast for the program, so anyone who wants to get some web-based typographic goodness can take a look at the program in action. Admittedly, it’s not very exciting, but then again, what excites me is the fact that I’m using my own program for pretty much all the HTML publication I do. Including this blog post.

Diversification

dmitrinesteruk.nospam@nospam.gmail.com (dmitri) — Mon, 19 Jan 2009 01:55:00 +0300

This is a new blog, which has arisen as a result of restructuring of my own domain, as well as my desire to leave the main one (mindstudies.org) where it is, and maybe enhance it somewhat. My stuff must go, and now it’s all completely separate, though I will be leaving the old blog where it is.

That was number one. The second thing I want to announce (to whom, I wonder), is the fact that after two open-source projects written in C# and WPF (I’m talking about the MathML → C# Converter and TypograFix, which I use to type all my stuff now), I decided that my next widget, whatever it happens to be, will be done in a different language. Which one? Well, it will most likely be F#, but I’ve been considering Nemerle, IronRuby and IronPython. I’ve also thought of crazy things like Erlang (there is no .net variant, to my knowledge), but that will only be used if there is clear reason for it.

Two more interesting things. First, I have put up all of my stuff on Subversion, as I cannot wait to get a reliable TFS install anywhere. I did have one, but then the hard drive decided to destroy itself – I doubt TFS was to blame, but whatever. Being happy with SVN, I also got a free account on FogBugz so now I track my stuff exclusively there. I must admit that I could just as easily host a JIRA personal license, but since I do commercial stuff, I’m really not allowed to. So FogBugz it is.

More fun in store – the first meeting of the St. Petersburg ALT.NET Group is probably not that far away, so I’m excited to see how things pan out – assuming the meeting happens at all. It’s nice to realize that the developer community hasn’t died off due to an economic crisis or something – though at times it seems that way.

That’s all for now. I better get back to coding useful programs, lest I lose my .net touch.

Dmitri Nesteruk

Simpler MVC localization

Improving business orientation

Use ASP.NET to collect visitor stats from blogs you do not own (but can post in)

Outsourcing vs Custom Development

Looking for projects on freelance sites

TypograFix 1.6 Released

VS2010, please, can I have a faster editor?

Html encoding bugs in AntiXss

A tip for managing complex CSS

Internal DSL in C# via reflection

Correlation of pay and benefits

Sending a Bitmap to a web service

The road ahead

Status Quo

So What?

Didactic inefficiencies

More fun with C++ and DirectWrite

Block-level formatting in DirectWrite

Reviewing My First Asp.Net MVC Project

Syntax

Linq2Sql

jQuery

Styling and Layout

Content

Use of AOP

Error handling

Using Direct2D and DirectWrite in .Net/C# Development

Community and Project News

Unwelcome Advice

Like Asp.Net MVC or Silverlight? We need you!

TypograFix and Local Image Generation

How to unit-test asynchronous operations?

Making DSLs in F#

Contents

Introduction

Source Code

Problem Statement

End Users

Continous Process Improvement

Choosing a Language

First DSL Statement

Handling Lists

Referencing with Strings

Greater Fluency

Generating the Project

Conclusion

Windows 2008 R2 out on MSDN!

Late-night musings on Russian Outsourcing

Beware of C-style macros in Boo

TypograFix Improvements

Working with pure C# auto-properties

Promotion of engineering culture

More user group activity at St. Petersburg!

Too Many Parallelization Frameworks?

A Crazy C# Question

Are non-automatic properties evil?

Visual Studio 2010 Beta 1 Screenshots

Installation

First few screens

Making a WPF app

Extra windows

What is Data-Driven Design?

Employer filtering

Start-up Woes

Defending code generation

Ideas for an IT book

Organizational thoughts

New heading styles

Rendering rich typography to bitmap

ClearType in Silverlight

TypograFix 1.5 Released

Working with SQL Server, hierarchical data and Silverlight

Contents

Introduction

Fun with SQL

A new UDT

Insertion

Descendants and Ancestors

Fun with C#… or not