Code, code and more code.

Fun with dynamic - how to build a dynamic object

2013-04-25T14:06:00.001-07:00

The project: build a redis client

The dynamic feature in C# 4 is powerful when used appropriately. I thought it was about time I wrote a piece on how to do that. And … because I like redis so much I thought it would make a perfect example.

Now, the observant among you may be thinking “but Marc, you’ve already written a redis client” – in which case I agree (and incidentally I congratulate you on getting hyperlinks into speech – no mean feat); but that isn’t the point! This toy client isn’t meant to compete: BookSleeve is heavily optimised to allow really fast and efficient usage. This one is just for fun. If you like it and it works for you, great! All the code for this is available on google-code.

What we want

I want to be able to do things like:

// increment and fetch new
int hitCount = client.incr("hits");
// fetch next, if any (else null)
string nextWorkItem = client.lpop("pending");

but without the client knowing anything about redis except the binary wire protocol – those commands are entirely dynamic. Actually a nice advantage of this is that the client doesn’t need to be updated as new redis features are released… but I digress!

Getting started

The key in implementing a dynamic API is implementing IDynamicMetaObjectProvider – although frankly I don’t propose doing that; I’m just going to subclass DynamicObject which does a lot of the work for us. So here's our first step:

public sealed class RedisClient : DynamicObject, IDisposable {...}

This gives us the start of a client that will respond to dynamic; although it doesn't actually do anything yet - we have to tell our object to handle dynamic method calls, which we do by overriding TryInvokeMember. Again, keep in mind that this is only a toy, and we’ll do this by simply writing the command name and parameters (in redis format) down the wire, and reading one result in redis format (note that this means that we’ll pay the full latency price per operation, which isn’t ideal – and that we can’t act as a redis pub/sub subscriber – that would simply not work, since replies don’t match neatly to commands then):

public override bool TryInvokeMember(
    InvokeMemberBinder binder,
    object[] args, out object result)
{
    WriteCommand(binder.Name, args);
    result = ReadResult();
    var err = result as RedisExceptionResult;
    if (err != null) throw err.GetException();
    return true;
}

In particular, note that the name of the method requested is available as binder.Name, and the parameters are just an object[].

Writing the command

I won't dwell on the redis protocol details (feel free to read the specification), but basically we need to write the number of arguments for the command (where the command-name itself counts as an argument), followed by the command-name, followed by each of the parameters. To avoid packet-fragmentation, we’ll use some buffering into a BufferedStream which we hold in outStream, which in turn writes to a NetworkStream which we hold in netStream - and obviously after each command we need to flush those to ensure they get to the server, so we get:

private void WriteCommand(string name, object[] args)
{
    WriteRaw(outStream, '*');
    WriteRaw(outStream, 1 + args.Length);
    WriteEndLine();
    WriteArg(name);
    for (int i = 0; i < args.Length; i++)
    {
        WriteArg(args[i]);
    }
    // and make sure we aren't holding onto any data...
    outStream.Flush(); // flushes to netStream
    netStream.Flush(); // just to be sure! (although this is a no-op, IIRC)
}

And for each argument, we need to write the length of the data, followed by the data itself. I won't detail each the various formats for different types of data, but: to avoid having to test "what type of data is this?", we'll cheat by having a few overloads of a method we'll call WriteRaw, and use dynamic to get the runtime to pick between the overloads for us... sneaky:

private void WriteArg(object value)
{
    // need to know the length, so: write to our memory-stream
    // first
    buffer.SetLength(0);
    WriteRaw(buffer, (dynamic)value);
    // now write that to the (bufferred) output
    WriteRaw(outStream, '$');
    WriteRaw(outStream, (int)buffer.Length);
    WriteEndLine();
    WriteRaw(outStream, new ArraySegment(buffer.GetBuffer(), 0, (int)buffer.Length));
    WriteEndLine();
}

Did you spot the cheeky dynamic in there? Since we're already in dynamic-land, it is hard to say that this is going to have any negative impact... so; why not? It means that if I need to support a new data-type, I just add a new WriteRaw to match, and: job done - and frankly, that's about it for sending the data.

Reading the response

After that, we need to refer to the specification again to see what replies look like - it turns out that they're basically the same format as the outbound data. But we have some ambiguity - does the user want their data as a byte[]? or as a decoded string? or maybe they want int? The nice thing is: we can let them tell us, by providing a result that supports conversions via dynamic - so when they type:

byte[] blob = client.get("my_image");
string name = client.get("name");

they get the binary and text correctly. So we can subclass DynamicObject again for a class that holds the raw result, and override another method - TryConvert this time. We get passed in a different binder, this time with access to the requested type in binder.Type - which we can then use to unscramble the data accordingly. This implementation is a less interesting and more tedious (testing different matched types), so I’ll leave that out of the blog. The only thing left to do (as shown in the TryInvokeMember) is to check if the response is an error-message from the server, and turn that into a thrown Exception, so that it feels intuitively .NET. The reason I can’t do this directly when reading the reply comes down to some implementation details – some replies are themselves composed of multiple nested replies, and I want to re-use the code internally for reading those. We can’t do that if it throws when hitting an error a few levels down – the stream could be left in an incomplete state (i.e. we might not have finished reading the outer-most reply).

Summary

And: that's it! A redis client written from scratch in a little over an hour; but more importantly, a complete dynamic API illustration. Well, maybe not complete, but as you can imagine: the other available operations (properties, indexers, operators, etc) work very similarly. I hope it is illustrative. Again, all the code for this is available on google-code.

How many ways can you mess up IO?

2013-02-25T13:14:00.000-08:00

There are so many interesting ways that people find to make a mess of reading binary data, but actually it doesn’t need to be hard. This post is intended to describe some of the more common misunderstandings people have.

Text encodings: what they are not

Text is simple, right?

Text encodings are great; you use them (whether you know it or not) all the time, whenever your computer opens any kind of text file, or downloads text (such as the ubiquitous html and json) from the internet. If you don’t know what an encoding is, then first: go and read Joel’s treatise on the subject. No really, go read that and then come back.

So now you know: an encoding is a way of working with character data (you know... words etc) over a binary medium (such as a file on disk, or a network socket). Common encodings are things like UTF-8, UTF-7, UTF-16 (in either endian-ness), UTF-32, or “Windows-1252” - however, there is a wide range of encodings available. Some (in particular the UTF-* encodings) can handle most-any unicode character, but many are restricted to the characters used most commonly in a particular locale. Ultimately, an encoding defines a map between characters and bytes, for example defining that “abc” should be represented as the bytes 00-61-00-62-00-63 (and equally, that the bytes 00-61-00-62-00-63 can be interpreted as the text “abc”).

Here’s the thing an encoding is not: it is not a way to turn arbitrary bytes into text for storage. You would be amazed how often people try this, but: that simply doesn’t work. If an encoding tries to read something that isn’t text data, then at best: it will throw an error. At worst, it will silently corrupt the data without realizing it has made a mistake. If you want to store arbitrary binary data as text, then there are other tools for that: primarily, things like base-n. In this case, it is the text that is specially formatted. For example, we might need to convey the bytes 07-A2-00-B2-EE-02 using only text.The observant will notice that I’ve just done exactly that using hexadecimal (base-16), but that it took 17 characters (12 without the dashes) to represent just 6 bytes of actual data. A common alternative to reduce this overhead is base-64, which uses characters that avoid the “control characters” likely to cause problems, while also staying in the first range 0-127, which is the most reliable region. Our 6 bytes become “B6IAsu4C” when stored as a base-64 string. Most platforms have utility methods that provide translation to and from base-64.

The key in choosing between a text-encoding and something like base-64 is simple: all you need to figure out is: which of these two things can have arbitrary contents, and which is required to follow rules (otherwise it is nonsensical)?

arbitrary* text / rules-based binary: use a text encoding
arbitrary binary / rules-based text: use base-64 (or similar)

*=at least, for the characters that the encoding supports, since not every encoding supports every character.

One last thing to say about encodings: don’t use the “system default” encoding - ever. This goes back to the old days of 8-bit systems where your text console needed a code-page that fitted into a single byte but reached at least most of the characters the user was likely to need. For me, this is code-page 1252, but yours may be different. It is exceptionally rare that this is what you want; if in doubt, explicitly specify something like UTF-8. Sometimes you can use byte-order-mark detection, but this is also pretty unreliable - many files omit a BOM. The best answer, obviously, is: define and document what encoding you are planning to use. Then use it.

Network packets: what you send is not (usually) what you get

TCP usually works as a stream of data. When you send things, it is usually not guaranteed that it will arrive at the destination in exactly the same chunks that you send. For example, let’s say that
you try to “send” 3 messages over the same socket using your chosen network library- one with 10 bytes, one with 4 bytes, and one with 8 bytes. You might think that the receiving client can call “read” 3 times and get 10 bytes, 4 bytes and 8 bytes - but it is much more interesting than that. Because it is just a stream, this is simply very unlikely to be what happens. The client could find they get 1 message with 22 bytes. Or 22 messages each with 1 byte. Or any other combination. They could get 10 bytes, 10 bytes, and the last 2 bytes never arrive (because of some network issue). All that TCP guarantees is that whatever portion of the data does make it to the receiver, it will be the correct bytes in the correct order.

Because of this, if you want to send multiple messages over the same socket it is necessary to implement some kind of “framing” protocol - i.e. some way of splitting the data back into logical pieces at the other end. How you do this is up to you, and often depends on the data being sent. For example, text-based protocols often detect a “sentinel” value to split messages (possibly a carriage return, line-feed, or some other “control character” rarely seen in regular text; the characters with values 0 (“nul”) and 3 (“etx”) are also popular choices.

If your messages are binary then it is more challenging, as usually there aren’t really any safe “sentinel” bytes to choose from - so commonly some header information is sent before each message that includes (perhaps only includes) the length of the message that follows. This could be as simple as dumping the 4-byte (32-bit) or 8-byte (64-bit) native representation of the length onto the stream (although you also need to decide whether it is “big endian” or “little endian”, obviously). There are also a range of variable-length representations, for when most messages are expected to be small, but the protocol needs to allow for much longer messages. The key thing here is that you must decide how you are going to do this, and clearly document it for consumers. This of course has the unfortunate requirement that you must actually know the length of the data you want to write before you write it - not always convenient. To help with this, some protocols (for example, the more recent web-sockets protocols) allow you to further break down a single message into multiple fragments that the client must stitch back together into a single logical piece - with an additional marker (perhaps a trailing zero-length message, or a special bit set in the header message) to indicate the end of the logical message.

Learning to read

Most frameworks have a “read” API that looks something like:

int read(byte[] buffer, int offset, int count)

(or something appropriately similar for asynchronous access). The point being that you supply a buffer (somewhere for it to put the data), tell it where in that buffer to start writing (often, but not always, 0, and how much data you want it to read. It then goes away and fetches some data for you. Here’s the rub: in most cases, the last parameter is just a maximum - it is not “get this much data”; it is “get at most this much data”. The return value tells you what it could do: this could be non-positive if no more data will ever be available (the stream has ended), it could be count - which is to say: it could fetch all the data you wanted, or it could be any other value greater than zero and less than count. Because of this, reading often involves a loop; for example, let’s say we want to write a method to read an exact number of bytes; me might do that as:

    void readExact(byte[] buffer, int offset, int count) {
        int bytesRead;
        if(count < 0) {
            throw new ArgumentOutOfRangeException(“count”);
        }
        while(count != 0 &&
          (bytesRead = source.read(buffer, offset, count)) > 0) {
            offset += bytesRead;
            count -= bytesRead;
        }
        if(count != 0) throw new EndOfStreamException();
    }

Dissecting that:

first we check that we aren’t requesting a negative amount of data, which is clearly silly
then in a loop, we:
- check to see if we want more data, then try to read at most count more data
- if the stream ends, we break out
- otherwise, we increment out offset, so that if we need to do another read, we don’t over-stamp the data we just fetched) - and decrement count, because we now have less work required
finally, we check whether we still have data outstanding, because the stream terminated before we expected to - perhaps raising an error to indicate this

This is a fairly classic pattern for reading data that shows how to process the number of bytes obtained in each iteration.

Gotchas when buffering data in memory

Sometimes it isn’t possible to process all the data in a single buffer; in many cases it becomes necessary to use an in-memory store of data that you have received but still need to process (perhaps because you need yet more data before you can make proper sense of it). A common way to do this is to make use of a “memory-stream” - an in-memory object that acts like a stream, but that simply uses local memory rather than a disk or network socket. You basically “write” to the memory-stream until you think you have something you can process, then “read” from it. But did you spot the deliberate mistake? Most memory-streams acts like an old-fashioned VCR: if you record a show, hit “stop”, and then hit “play” - you will find that you are unexpectedly either watching a black screen, or something that you recorded 3 weeks ago. We forgot to rewind it. Essentially, the memory-stream has a single cursor position; after writing the data, the cursor is at the end of the data: what follows is either all zeros, or garbage from the last thing that happened to be in that space. Fortunately, rewinding a memory-stream is usually trivial:

stream.Position = 0;

There’s a second very common confusion with memory-streams; sometimes, after writing to it you want to get the contents, but as a raw buffer (byte[]) rather than as a stream. Often, there are two different APIs for this:

one which gives you a brand new buffer of the current logical length of the stream, copying the data into the new buffer
one which hands you the oversized buffer that it is using internally - oversized so that it doesn’t need to allocate a new backing buffer every time you write to it

They both have uses; the second is often more efficient, as it avoids an allocation - but it is only useful if you also track the logical length of the data (often via stream.Length or stream.getLength()). To go back to our VCR analogy: one creates a brand new cassette of exactly the 37.5 minutes we recorded, and copies the video - the other simply hands us the existing 180 minute cassette: this is only useful if we also say “the bit you want is the first 37.5 minutes”.

Conclusion

This isn’t actually all that complex, but people tend to bring a lot of incorrect preconceptions to the table, which results in the same bugs – over and over and over. So let’s all save some time, and do it right. Please?

Multiplexed transactions in BookSleeve

2012-10-10T15:29:00.001-07:00

Short version, for the impatient

BookSleeve now has support for transactions using WATCH and pre-conditions. If you don’t know what BookSleeve or redis is, this won’t be of interest to you.

Background

We use redis extensively as part of our caching (and a few other things) technologies at Stack Exchange, and as part of doing that efficiently, we put together BookSleeve, which is a high-performance multiplexer over a redis connection. In English, what that means is: you can share a single redis connection from lots of different threads, and it will worry about delivering the commands efficiently and shipping results back to the appropriate callers, without 200 commands having to pay the penalty of 200 latency. Redis even includes transaction support for complex operations, however, previously I’ve always been a bit stumped how to fit this into BookSleeve…

How do transactions work in redis?

You see, redis transactions are not like SQL transactions. A transaction is defined by a MULTI … EXEC block, with everything between the MULTI and the EXEC being processed as a single atomic unit (meaning: redis doesn’t process any requests from other connections during that time - don’t worry if that sounds draconian, redis manages extraordinarily well on a single thread anyway). That’s pretty easy to fit into a multiplexer, as we can just have a structure to buffer up commands, and send them when complete – so that already exists in BookSleeve:

using (var tran = conn.CreateTransaction()) 
{ 
    tran.Strings.Set(0, "foo", "abc"); 
    tran.Strings.Set(0, "bar", "def"); 
    tran.Execute(); 
}

But here’s the gotcha: between the MULTI and the EXEC, you don’t get results – you only get results when you commit the transaction. This means that you can’t easily request information in the middle of a transaction and then make a decision on that for what you do next – which makes a lot of sense really, since while you’re doing that thinking and network-IO, every other connection would be stalled. A sane decision really.

To get around this limitation, redis transactions actually allow a bit more cleverness… you can issue a WATCH command against keys of your choice, and then if somebody changes that key, your transaction automatically gets killed. So, a typical cycle might be:

WATCH SomeKeyName
GET SomeKeyName
{some test on the value back in your client code, then}
MULTI
SET SomeKeyName SomeNewValue
EXEC
{or if you decided you didn't like it after all}
UNWATCH

And importantly, because of the WATCH, if another connection comes along and changes SomeKeyName, then the EXEC does nothing, and returns a reply that robustly indicates the cancellation. This actually allows a lot of very subtle usage, while allowing everything to happen at the server as a single unit of work.

So what is the problem? And what did you do?

The painful bit here is simply: WATCH doesn’t work in a multiplexer. As soon as one caller issues a WATCH, either you need to stop accepting work from other callers, or you now have no idea what each caller thinks it is watching (which is very important). Another approach would be to somehow have the user write callbacks that happen as part of a transaction, but what I absolutely don’t want to do is expose anything in BookSleeve that inadvertently allow one caller to break everybody, by being too slow – so callbacks: not really an option. So for a long time, BookSleeve only made use of WATCH etc internally, and didn’t expose it to the caller. But that has finally changed!

I’ve just committed my first stab at improving this, basically by implementing a number of pre-canned pre-conditions that can be enforced on a transaction. The “pre-canned” is there to avoid the issue of opening up massive performance issues, but in reality they are very simple. For example, a previous internal usage of WATCH was to take an exclusive lock (by marking a key as in-use). Until today, that was about 100 lines of complex code that needed to know about the full gory details of the redis protocol – where-as now it is just:

TaskCompletionSource<bool> result = new TaskCompletionSource<bool>();
            
using (var tran = CreateTransaction())
{
    tran.AddCondition(Condition.KeyNotExists(db, key));
    tran.Strings.Set(db, key, value, expirySeconds);
    tran.Execute().ContinueWith(takeLockContinuation);
}
return result.Task;

where takeLockContinuation is just a static delegate instance that take the Task<bool> result from the Execute call, and sets that into result (plus handling task faults, cancellation, etc). Crucially, the condition (Condition.KeyNotExists in this case) takes care of both the WATCH steps, and the actual validation / enforcement. Basically, if there are pre-conditions (there can be multiple), then they are all sent to the server, and only if they all return the expected results is the MULTI / EXEC block sent – otherwise an UNWATCH is sent instead. Simple, elegant (IMO), and efficient.

The available pre-conditions are pretty limited at the moment, but I’ll try to add all the obvious ones I can think of (exists/not-exists for keys/hashes, equals/not-equals for keys/hashes, maybe a few inequalities, maybe something involving ttl). But finally I have a WATCH implementation that isn’t entirely sucky for a multiplexer. At the moment this is in the code repository only; I'll push to NuGet when I have a few more pre-conditions implemented.

Iterator blocks, missing methods, and .NET 4.5

2012-09-06T06:47:00.001-07:00

We had some fun a while ago where we had a flurry of MissingMethodException occurring on some boxes, in particular, complaining that Environment.CurrentManagedThreadId was missing. And sure enough, this is a 4.5 method, and the affected servers were 4.0 – so it was kinda valid, but completely unexpected. At the time, our most pragmatic option was just: deploy 4.5, but we’ve finally had time to go back and understand it!

This happened after we upgraded our build server to 4.5, for some initial 4.5 usage – but all our core projects were still 4.0 at the time.

Fix first:

MAKE SURE YOUR BUILD SERVER HAS THE REFERENCE ASSEMBLIES FOR 4.0!

These are typically somewhere like:

C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.0

Now, the full story:

fact 1: 4.5 is an in-place over-the-top install on top of 4.0, in the GAC; once you have installed 4.5, 4.0 runs with the 4.5 assemblies

fact 2: if your build server doesn’t have the reference assemblies, it looks in the GAC – so… once you’ve installed 4.5, it will get 4.5 even if you asked (in your project) for 4.0

fact 3: the c# compiler has a detail relating to iterator blocks (aka "yield return" and "yield break") that involves threading, i.e. "which thread am I?"

fact 4: the implementation for this changes between 4.0 and 4.5; in 4.0, it uses (in the generated iterator-type’s constructor) Thread.CurrentThread.ManagedThreadId – but in 4.5 it uses Environment.CurrentManagedThreadId; it makes this decision based on the framework version it finds it has loaded (taking into account targeting etc)

Put these pieces together, and you can get a scenario where a 4.0 project that uses 4.0-only methods, seems to build successfully, but builds in a way that will only work on 4.5; it will never work on 4.0.

Fun fun fun!

Taking meta-programming beyond crazy

2012-08-17T12:45:00.001-07:00

Firstly, a brief note that protobuf-net is up to r580 now, on both nuget and google-code; mainly small tweaks while I build up enough energy to tackle a few larger pieces (some knotty interface / dynamic / base-class improvements are next on the list)

Pushing both ends

Aimed to time with the .NET 4.5 release (and perhaps more notably, the .NETCore and .NETPortable profiles), I’ve recently spent a lot of time on meta-programming, culminating in the new precompiler that allows these slimmed down and highly restrictive frameworks to still have fast serialization (static IL, etc).

While there, I set myself a silly stretch goal; for the main purpose simply of to see if I could do it – which was: to get the whole shabang working on .NET 1.1 too. This would give me a fair claim to supporting the entire .NET framework. So, for a bit of reminiscing – what does that need?

Generics

Generics were introduced in .NET 2. v1 of protobuf-net made massive usage of generics; so much so that it actually killed the runtime on some platforms (see here, here and here). So removing most of the generics was already a primary design goal in v2.

Perhaps the most significant problem I hit here was trying to decide on a core collection type for the internal state. As it turns out, there’s no free lunch here; there is no collection type that is common to all frameworks – some don’t have ArrayList. In the end, I wrote my own simple collection – not just for this, but also because I wanted a collection that was thread-safe for iterations competing with appends (iterators only read what existed when it started iterators).

Language

You’d be amazed what you miss when you try to design something to compile on down-level compilers. For a dare, go into project-properties, the “Build” tab, and click “Advanced…” – and change the “Language Version” to something like ISO-1 (C# 1.2) or ISO-2 (C# 2.0) – see what breaks. Obviously you expect generics to disappear, but you also lose partial methods, partial classes, iterator blocks, lambdas, extension methods, null-coalescing, static classes, etc – and just some technically legal syntax that the early compilers simply struggle with. protobuf-net is configured to build in ISO-2 in the IDE, but with #if-regions to rip out the last few generics in the .NET 1.1 build. Writing iterator blocks… not fun.

Framework

There’s a silly number of variances in the core BCL between different frameworks; even things like string.IsNullOrEmpty or StringBuilder.AppendLine() aren’t all-encompassing. I ended up with a utility class with a decent number of methods to hide the differences (behind yet more #if-regions). But by far the craziest problem: reflection. And protobuf-net, at least in “Full” mode (see here for an overview of “CoreOnly” vs “Full”), uses plenty of reflection. Oddly enough, the reflection in .NET 1.1 isn’t bad – sure, it would be nice to have DynamicMethod, but I can live without it. Getting this working on .NET 1.1 was painless compared to .NETCore.

Aside / rant: how much do I hate “.GetTypeInfo()” on .NETCore? With the fiery rage of 2 stars slowly crashing into each-other. Oh, I’m sure that the differences to Type / TypeInfo make perfect sense for application-developers in .NETCore, who probably should be limiting their use of reflection, but for library authors: this change really, really hurts. The one things that lets me keep civil about this change is that in “CoreOnly” + “precompiler” we do all the reflection work up-front using the regular reflection API, so for me at least most of this ugly is just a cruel artefact. But still: grrrrrrrrrrrrrr.

Opcodes

There are a number of opcodes that simply don’t exist back on 1.1; if I’ve done my compare correctly, this is: Unbox_Any, Readonly, Constrained, Ldelem and Stelem. The good news is that most of these exist only to support generics, and are pretty easy to substitute if you know that you aren’t dealing with generics.

Metadata Version

.NET 1.1 uses an earlier version of the metadata packaging format than all the others use. This is yet another thing that the inbuilt Reflection.Emit can’t help, but - my new favorite metaprogramming tool to the rescue: IKVM.Reflection supports this. I have to offer yet another thanks to Jeroen Frijters who showed me the correct incantations to make things happy: beyond the basics of IKVM, the key part here is a voodoo call to IKVM’s implementation of AssemblyBuilder:

asm.__SetImageRuntimeVersion("v1.1.4322", 0x10000);

The 0x10000 here is a magic number that specifies the .NET 1.1 metadata format. For reference, 0x20000 is the version you want the rest of the time. As always, IKVM.Reflection seems to have considered everything; it is the gold standard of assembly writing tools. Awesome job, Jeroen. I jokingly half-expect to find that Roslyn has a a reference to IKVM.Reflection ;p

Putting the pieces together

But! Once you’ve dealt with all those trivial problems; it works. I’m happy to say that protobuf-net now has “CoreOnly” and “Full” builds, and support from “precompiler”. So if you still have .NET 1.1 applications (and I promise not to judge you… much), you can now use protobuf-net with as many optimizations as it is capable of. Which is cute:

C:\SomePath>AnotherPath\precompile.exe Net11_Poco.dll -o:MyCrazy.dll -t:MySerializer

protobuf-net pre-compiler
Detected framework: C:\Windows\Microsoft.NET\Framework\v1.1.4322
Resolved C:\Windows\Microsoft.NET\Framework\v1.1.4322\mscorlib.dll
Resolved C:\Windows\Microsoft.NET\Framework\v1.1.4322\System.dll
Resolved protobuf-net.dll
Adding DAL.DatabaseCompat...
Resolved C:\Windows\Microsoft.NET\Framework\v1.1.4322\System.Xml.dll
Adding DAL.DatabaseCompatRem...
Adding DAL.OrderCompat...
Adding DAL.OrderLineCompat...
Adding DAL.VariousFieldTypes...
Compiling MySerializer to MyCrazy.dll...
All done

C:\SomePath>peverify MyCrazy.dll

Microsoft (R) .NET Framework PE Verifier  Version 1.1.4322.573
Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.

All Classes and Methods in MyCrazy.dll Verified

In case it isn’t obvious, “Net11_Poco.dll” is a .NET 1.1 dll created in Visual Studio 2003; “precompiler” has then detected the 1.1-ness, bound IKVM to the .NET 1.1 framework, and compiled a protobuf-net custom serializer for that model, as a legal .NET 1.1 dll.

Questionable sanity

Another way of reading all this is: I’ve possibly now crossed the line between “eccentric” and “batshit crazy”. I don’t have a need to use .NET 1.1, but I would be overjoyed if someone else gets some genuine usage out of this. Mainly, I just wanted to learn some things, challenge myself, and take a bit of professional pride in doing something fully and properly – just because: I can.

Introducing the protobuf-net precompiler

2012-07-16T16:15:00.001-07:00

Over the last few posts (here and here) I’ve given a few hints as to compiling for other platforms. Basically, this all relates to how well protobuf-net works on platforms like iOS, WinRT/Metro, Silverlight, Phone 7, etc. These heavily restricted runtimes don’t allow much meta-programming, and they might be running on low-power CPUs, so reflection (even if possible) is not ideal.

I’ve played with assembly generation before, but with mixed results. For example, here for Phone 7. This could just about work for some frameworks, but was patchy on some, and won’t work at all for others.

Well, all that IKVM shininess has opened up a whole new set of tools. The small beauty that I’m ridiculously pleased with is a new utility exe in the SVN trunk (will be part of a proper release soon): precompile.

This unassuming little tool works a bit like “sgen”, but with the ability to target multiple frameworks. What it does is:

inspect the input assembly (or assemblies) to resolve (if it can) the target framework
initialize an IKVM universe targeting that framework
load the core framework libraries, protobuf-net, and the input assemblies into the IKVM universe
scan the input assemblies for types marked [ProtoContract]
add those to a protobuf-net model
compile the model to a type / assembly of your choosing

To use that in a project you might:

create a new DTO project in your chosen framework, and compile it
execute precompile to generate to a serializer assembly
from your application project, reference the DTO and serializer assemblies
use the type you created

For example, say I create a new Phone 7 DTO assembly, called PhoneDto (because it is late and I lack imagination). I can then create a serialization assembly via:

precompile {some path}\PhoneDto.dll –o:PhoneSerializer.dll –t:MySerializer

This will generate a library called PhoneSerializer.dll, which you can reference from your main project (in addition to the DTO and the protobuf-net core).

Then, just use MySerializer:

var ser = new MySerializer();
ser.Serialize(output, obj);

I hope this finally solves a number of tooling issues. I’m pretty pleased with it. I’ve tested it against a range of different frameworks, and it has worked well – but if you get problems, just let me know (comment here, or email me, or log an issue on the protobuf-net project site).

Enter the IKVM

2012-07-14T03:35:00.001-07:00

aka Meta-programming and Metro for .NET

In my previous blog entry I gave an overview of how protobuf-net is arranged internally, and hinted that it all falls apart for Metro. So: what goes wrong? Currently, protobuf-net has a lot (and I do mean a lot) of code built on top of ILGenerator, the reflection class that underpins meta-programming. This class is great for hardcore library builders: you can build individual methods (via DynamicMethod) or entire assemblies (via AssemblyBuilder). It is very low level, since you are writing raw IL – but I’m not completely crazy, so I wrap that up in some utility methods. There are a few problem., though:

ILGenerator usually expects be be working on the current framework
Even though you can use a “reflection only load”, this still fails horribly for System.Runtime.dll (aka Metro for .NET)
In early versions of .NET (and I try to support .NET 2 and upwards), the ability to properly inspect attribute data against reflection-only types/members (GetCustomAttributesData) is simply missing

So… basically, it all falls to pieces for generating as assembly targeting Metro for .NET, based on a Metro for .NET input assembly.

What I would need is some kind of utility that is:

Broadly similar to Reflection.Emit, so I don’t have to rewrite a few thousand lines of code (since I want to keep that code for the runtime meta-programming on full .NET)
Able to load and work with alternative frameworks without getting confused
Able to give full attribute information, even on .NET 2
Inbuilt or free, to match the existing license

Who would go to the trouble of writing such a thing?

Maybe somebody who is already writing compiler-like tools that aren’t tied to a specific framework, and who has run into the existing limitations of Reflection.Emit? Maybe somebody writing a Java/.NET bridge? Like IKVM.NET ?

Actually, for my purposes I don’t need most of IKVM. I just need one tiny piece of it: IKVM.Reflection. This library was specifically written to mimic the existing Reflection.Emit API, but without all the problems. In particular, it has a Universe class that is a bit like an AppDomain; you load assemblies into a Universe, and that is what is used for resolution etc after that. Here’s a simple example – which will immediately make sense to anyone used to Reflection.Emit – or another example specifically targeting Metro for .NET. The nice thing about the API is that mostly I can change between IKVM and Reflection.Emit via a simple:

#if FEAT_IKVM
using Type = IKVM.Reflection.Type;
using IKVM.Reflection;
#else
using System.Reflection;
#endif

OK, I would be grossly exaggerating if I claimed that was the only change I had to make, but that’s the most demonstrable bit. What this means is that just for my cross-platform compiler, everything in the Model, Strategy and Compiler modules (see previous blog) switches to IKVM terminology. The Core still uses System terminology, and the Runtime doesn’t apply (I obviously can’t instantiate/execute types/methods from another framework, even if I can inspect them).

With this technique, I can now successfully load a Metro for .NET assembly (such as a DTO), and generate a fully static-compiled assembly (“Standalone” on the diagram) targeting Metro for .NET. No reflection at runtime, no nasty hacks – just: an assembly that wasn’t generated by the MS tools.

I’m still working on turning this into a slicker tool (comparable to, say, SGEN), but a working illustration is in the repo; in particular, see MetroDTO (some sample DTOs), TestIkvm (my proof-of-concept compiler hard-coded to MetroDTO), and Metro_DevRig (which shows the generated assembly working in a Metro for .NET application, including performance comparisons to XmlSerializer and DataContractSerializer).

Additionally, it seems extremely likely that the same tool should be able to write platform-targeted assemblies for Silverlight, Phone 7, XNA, CF, etc. Which is nice.

Thanks and Acknowledgements

I’m hugely indebted to Jeroen Frijters in all of this; both for providing IKVM.Reflection, and for his direct and very prompt assistance when playing with all the above. Mainly for correcting my own brain-dead mistakes, but also for a very quick bug-fix when needed. The library is awesome, thanks.

(as a small caveat, note that protobuf-net is currently exposing a locally built version of IKVM.Reflection; this will be rectified after the next dev build of IKVM)

Some internals of protobuf-net

2012-07-13T05:59:00.001-07:00

This post is a bit of background explanation for my next, much more interesting (IMO) post. But in order for that one to make sense without being too long, I wanted to talk about the “what?” and “why?” first, and then discuss the “how?” after. Make sense?

First, I’ll whet your appetite with this:

This is protobuf-net running nice and fast (much faster than XmlSerializer or DataContractSerializer) on .NET for Metro style apps. Which is a pretty neat trick, because Metro doesn’t allow any of the usual tricks that protobuf-net uses.

So what does it use?

A walkthrough of protobuf-net

The first piece of protobuf-net is the core – this is basically the fundamental reader/writer API (think: XmlReader/XmlWriter), and some utility methods. Every build of protobuf-net includes the core, although many parts may have different implementations on a per-platform basis.

Next we have the model; this is where the user defines the types that are going to be serialized by the model. In many cases, the model is populated automatically as you serialize items (it figures out what is needed). The model doesn’t know how to do anything – it only knows the configuration.

Next comes the strategy; a series of decorators and other structures that turn that configuration into something resembling a sensible way to handle, each configuration, building from simpler steps.

And then given a strategy, we need some mechanism to implement that strategy. A simple option is reflection, which is convenient and available on most platforms, but comparatively slow.

A much more elegant approach is to use meta-programming to turn a strategy into regular IL code. This is pretty complex, but leads to very fast results. Unfortunately, a lot of light-weight platforms do not allow this.

Which is a shame, because once you have a working compiler, you can go so far as to generate (via a utility exe at build time) a standalone serialization assembly, which then doesn’t need to do any thinking at runtime.

In fact, if you do this you don’t even need anything else – just the core and a serialization assembly. This technique allows you to generate, with a little voodoo, a pair of small assemblies that can be used on platforms such as MonoTouch, as described here.

So you just did that, right? Doesn’t sound too bad…

You’d think so, wouldn’t you? Unfortunately, there’s a stumbling block. Generating a standalone assembly is … complex. It only works on full .NET for starters, but also: Reflection.Emit only works with full .NET assemblies. You can’t load your Metro DTO into regular .NET, spin up a TypeBuilder, and run with it; a: you won’t be able to load the types correctly (since your Metro DTO targets an entirely different framework), and b: what you generate won’t be convincing as a Metro dll. We get away with it for MonoTouch, but that is just because the Xamarin folks are evil geniuses.

What we need, then, is some way of generating a Metro serialization dll, after inspecting a Metro DTO dll, but presumably as a build tool running on regular .NET. That, as they say, gets tricky. And that is for next time.

The question I was asked at DDD South West…

2012-06-12T04:50:00.001-07:00

…about 20 times during and after my last presentation sessions had nothing to do with my content. I am choosing (hoping?) to believe that this is because I articulated myself so clearly and accurately.

No, the question everyone asked me was:

What was putting those method-end markers in the IDE?

i.e.

Well, I didn’t want to mess around with my laptop too much at the event, as I had a re-run session later that day. The first rule of presenting: don’t mess around with your laptop just before you present!

But – I finally tracked it down; this is VSCommands “Enable Code Block End Tagger”. It comes with cute options too, for example: to only display the end tag when the start is not visible (which you can see above – no noise is added for the constructor or indexer, since you can see the start and end at the same time).

It’s a nice tool, and it has VS 2012 support already, apparently (I haven’t had time to test it). Heck, how many IDE tools do you know that have an option to display your stackoverflow reputation and recent activity? (this is not a joke). Or more importantly, has a tool to stop the annoying “drag/drop a file in solution explorer, when I clearly didn’t intend to totally and utterly shaft the solution” (they call it by a different name, but that’s what I would have called it).

(anti-disclosure; no I don’t get anything from them for saying that it is a good tool – it just is)

Abstractions are only a tool; you don’t need to be

2012-02-06T00:30:00.001-08:00

Abstractions abstractions abstractions. IAbstraction and AbstractionFactory.

Ah, joy.

I get very very confused when I see people saying things like “but it must be done via EF” (substitute EF for any other tool), when the thing they are trying to do is clearly not suited to the abstraction.

Abstractions, as tools, are simply a way of reducing the overhead / complexity of doing something specific (managing the relationship between objects and a database in the case of EF). Sometimes they are fairly tight; sometimes they are leaky. But; to use any abstraction, it is absolutely critical to be aware of two related things:

where the limits of the abstraction end
how the underlying “thing” works

I get so weary of people using things like EF in the hope that they then “don’t need to learn SQL”. This is a false and destructive aim. You, as a professional developer, wouldn’t try to write a web-page without knowing at least the fundamentals of html and css? Would you? Really?

In the same way; occasionally, a tool simply isn’t aimed at the job you want to do. If you want to avoid pain, you need to recognise this as soon as possible. I’ve seen some horrible horrible ugliness intended to strong-arm an existing tool to do something that it didn’t want to. In most cases, simply using a different tool for that piece (yes, you are allowed to use more than one), or writing the code directly, would have been simpler, less buggy, and wouldn’t take a great big core-dump all over your shiny code. Code matters; don’t contaminate it.

[end rant]

Playing with your member

2012-01-12T15:20:00.001-08:00

(and: introducing FastMember)

Toying with members. We all do it. Some do it slow, some do it fast.

I am of course talking about the type of flexible member access that you need regularly in data-binding, materialization, and serialization code – and various other utility code.

Background

Here’s standard member access:

Foo obj = GetStaticTypedFoo(); 
obj.Bar = "abc";

Not very exciting, is it? Traditional static-typed C# is very efficient here when everything is known at compile-time. With C# 4.0, we also get nice support for when the target is not known at compile time:

dynamic obj = GetDynamicFoo(); 
obj.Bar = "abc";

Looks much the same, eh? But what about when the member is not known? What we can’t do is:

dynamic obj = GetStaticTypedFoo(); 
string propName = "Bar";  
obj.propName = "abc"; // does not do what we intended!

So, we find ourselves in the realm of reflection. And as everyone knows, reflection is slooooooooow. Or at least, it is normally; if you don’t object to talking with Cthulhu you can get into the exciting realms of meta-programming with tools like Expression or ILGenerator – but most people like keeping hold of their sanity, so… what to do?

Middle-ground

A few years ago, I threw together HyperDescriptor; this is a custom implementation of the System.ComponentModel representation of properties, but using some IL instead of reflection – significantly faster. It is a good tool – a worthy tool; but… I just can’t get excited about it now, for various reasons, but perhaps most importantly:

the weirdness that is System.ComponentModel is slowly fading away into obscurity
it does not really address the DLR

Additionally, I’ve seen a few bug reports since 4.0, and frankly I’m not sure it is quite the right tool now. Fixing it is sometimes a bad thing.

Having written tools like dapper-dot-net and protobuf-net, my joy of meta-programming has grown. Time to start afresh!

FastMember

So with gleaming eyes and a bottle of Chilean to keep the evil out, I whacked together a fresh library; FastMember – available on google-code and nuget. It isn’t very big, or very complex – it simply aims to solve two scenarios:

reading and writing properties and fields (known by name at runtime) on a set of homogeneous (i.e. groups of the same type) objects
reading and writing properties and fields (known by name at runtime) on an individual object, which might by a DLR object

Here’s some typical usage (EDITED - API changes):

var accessor = TypeAccessor.Create(type); 
string propName = // something known only at runtime  
while( /* some loop of data */ ) {  
accessor[obj, propName] = rowValue;  
}

or:

// could be static or DLRvar wrapped = ObjectAccessor.Create(obj); string propName = // something known only at runtime Console.WriteLine(wrapped[propName]);

Nothing hugely exciting, but it comes up often enough (especially with the DLR aspect) to be worth putting somewhere reusable. It might also serve as a small but complete example for either meta-programming (ILGenerator etc), or manual DLR programming (CallSite etc).

Mary Mary quite contrary, how does your member perform?

So let’s roll some numbers; I’m bundling read and write together here for brevity, but - based on 1M reads and 1M writes of a class with an auto-implemented string property:

Static C#: 14ms
Dynamic C#: 268ms
PropertyInfo: 8879ms
PropertyDescriptor: 12847ms
TypeAccessor.Create: 73ms
ObjectAccessor.Create: 92ms

As you can see, it somewhat stomps on both reflection (PropertyInfo) and System.ComponentModel (PropertyDescriptor), and isn't very far from static-typed C#. Furthermore, both APIs work (as mentioned) with DLR types, which is cute - becaues frankly they are a pain to talk to manually. It also supports fields (vs. properties) and structs (vs. classes, although only for read operations).

That's all; I had some fun writing it; I hope some folks get some use out of it.

The autistic elephant in the server-room

2011-11-26T03:18:00.001-08:00

A bit of a serious blog-entry this time. Normally I blog about language lemmas, IL intricacies, serialization subtleties, and materialisation mechanisms; but occasionally bigger topics present.

This blog entry is probably more for my purposes than yours. But you might like it anyway. If not, just come back next time.

Woah, lots of words; why should I read this?

Because knowledge never hurts. You might start to understand a colleague better; maybe it might help you recognise autistic traits in a family-member; or maybe you'll just think for a second before "tutting" when you see a child apparently misbehaving in the supermarket - perhaps you don't know what is happening quite as much as you think.

Why am I putting this on my geek blog?

Because, while the reasons still aren't exactly clear [edit: alternative], it tends to present more commonly in the children (mainly sons) of people in geeky professions. Indeed, our profession has a pretty high rate of successful autistic or Asperger's members; some very notable, some doing the day job like everyone else.

More specifically, I care because my eldest son is autistic. Now, autism (or more correctly, ASD) is a pretty large spectrum. In the UK, there is a trend not to bother trying to diagnose more specific groupings (like Asperger's) because it doesn't actually help with treatment: everyone with ASD needs to be assessed as an individual to understand their needs.

Evan is at the (perhaps unfairly named) "high functioning" end of the spectrum, which means he is of pretty normal intelligence and largely independent, but experiences the world in a slightly different way.

If you want to very quickly get a feel for stereotypical ASD at the "high" end, then Sheldon Cooper is perhaps the easiest place to look, with two caveats:

- there is no automatic "brilliance"; normal intelligence (albeit it, focused) is more likely

- everybody truly is individual; it helps to remember that generally speaking, generalisations don't help

Also, AFAIK The Big Bang Theory has never explicitly labelled Sheldon with ASD or Asperger's

(boo, someone deleted my Sheldon image... take your pick from here

Typical observations would include (not exhaustive):

- attention / interest can be intensely focused on a narrow topic, to the exclusion of others

- sensory perception (sound, touch, taste, smell, sight, etc) can be extreme (both hyper- and hypo-); for example, simply the washing/size-label on a t-shirt can cause agitation

- cognitive processing can get focused on a particular detail rather than processing the wider context; in programming terms think "depth first" rather than "bredth first"

- language and interpretation can be very literal (for example, "laughing your head off" could be concerning) and repetitive

- routine and predictability are important (in particular, this reduced the stress of processing unfamiliar scenarios), to the point where an individual can seem inflexible and stubborn

- social interaction tends to be limited (people aren't predictable, and often aren't very interesting - even less so if they aren't talking about your preferred topic) and a little awkward (not least, brutal honesty doesn't always go down well - "that lady smells bad" etc)

- there can be a definite need for "quiet time", or just time to unwind in whatever way works. That might be "stimming", or it might be sitting under a table for half an hour

- recollection (in particular visual memory) can be more acute

Of course, at the other end of ASD are far more debilitating issues, which may mean dedicated life-long care, maybe institutionalisation. It is not always gentle. I can't speak of this from experience, so I won't try.

So... What is the point of this blog entry?

It is a serious issue that affects our industry, perhaps more than most. Also, our industry is well suited to ASD. Computers are predictable; IT has a use for massively-focused detail-oriented people (experts perhaps in a very narrow and specialised field), and people who can quickly spot some really minor and subtle differences.

Perhaps more importantly though: it shouldn't be something we don't talk about. ASD tends to be an invisible and private thing (you can see a wheel-chair or hearing-aid; you can't necessarily see that the person you are talking to is actually massively stressed to breaking point because a road was closed and they took a different route to work, and then the 2nd lift on the right - THE LIFT THEY USE - was out of order).

Ultimately, no matter where on the spectrum someone is, we should not feel anything like embarrassment. I'm not embarrassed by my son - he's awesome! It took us as a family a little time to properly understand him and his needs, but I think we have something that works well now. And his current school is really great (mainstream, but really understand ASD; his last two schools..... not so much).

If you have a child with ASD

Moving from denial through to acceptance can take time. There's only so many birthday parties you can take them to where opening and closing a door for an hour is more interesting that the hired entertainer - eventually you need to accept that they are simply different. Not better or worse; just different.

Yes, I know, it can be hard sometimes. You may feel like a chess grand-master constantly thinking 12 steps ahead to avoid some meltdown later in the day. Remembering to give advance notice of change, and implementing damage limitation when something truly unexpected happens. Talking openly about it can be hard (yet is also very liberating). There are often local groups of other parents with similar experiences that you may find helpful (assuming you can arrange cover - regular babysitters may not work out so well).

Laughing helps. Try reading #youmightbeanautismparentif. Or watching Mary and Max

Also, don't buy into any of the snake-oil.

If you have ASD

Then you know ASD better than me. I've seen some folks with ASD get annoyed before because it is always "parents of..." doing the talking. Well, the world is your podium: I'd love to hear your thoughts. The "parents of" also have an awful lot to bring to the table, though.

And for Evan

He's doing great. He likes maths and his handwriting is almost as bad as mine. He can probably name a handful of the children in his class, but I'm sure he could tell me how many lights there are in each room in the school (and how many have broken/missing bulbs). We have good days, and we've had some really horrible days - but as we get better, together, at knowing what each of us needs, we get more of the former and less of the latter.

Final words

"I'm not crazy; my mother had me tested"

"I'm not insane; my mother had me tested"

Dr. Sheldon Cooper

Assault by GC

2011-10-24T05:12:00.000-07:00

TL;DR;

We had performance spikes, which we eased with some insane use of structs.

History

For a while now, we had been seeing a problem in the Stack Exchange engine where we would see regular and predictable stalls in performance. So much so that our sysadmins blogged about it back here. Annoyingly, we could only see this problem from the outside (i.e. haproxy logging, etc) – and to cut a very long story short, these stalls were due to garbage collection (GC).

We had a problem, you see, with some particularly large and long-lived sets of data (that we use for some crazy-complex performance-related code), that would hobble the server periodically.

Short aside on Server GC

The regular .NET framework has a generational GC – meaning: new objects are allocated “generation 0” (GEN-0). When it chooses, the system scans GEN-0 and finds (by walking references from the so-called “roots”) which (if any) of the objects are still in use to your application. Most objects have very short lives, and it can very quickly and efficiently reclaim the space from GEN-0; any objects that survived move to GEN-1. GEN-1 works the same (more-or-less), but is swept less often – moving any survivors into GEN-2. GEN-2 is the final lurking place for all your long-lived data – it is swept least often, and is the most expensive to check (especially if it gets big).

Until .NET 4.5 rolls into town with a background server GC, checking GEN-2 is a “stop the world” event – it (usually briefly) pauses your code, and does what it needs to. Now imagine you have a huge set of objects which will never be available to collect (because you will always be using them) all sat in GEN-2. What does that look like? Well, using StackExchange Data Explorer to analyse our haproxy logs, it looks a bit like this:

I’ve omitted the numbers, as they don’t matter; but interpretation: normally the server is ticking along with nice fast response times, then WHAM! A big spike (which we have correlated with GC) that just hammers the response-times.

So what to do?

We take performance very seriously at Stack Exchange, so as you imagine this got a bit of attention. The obvious answer of “don’t keep that data”, while a valid suggestion, would have hurt a lot of the overall performance, so we needed to find a way to remove or reduce this while keeping the data.

Our initial efforts focused on removing things like unnecessary copy/extend operations on the data, which helped some, but didn’t really make a step-change. Eventually, we concluded…

Break all the rules

Important: the following is merely a discussion of what has helped us. This is not a magic bullet, and should only be applied to some very limited scenarios after you have profiled and you know what you are doing and why. And it helps if you are just a little bit crazy.

First, a brief overview – imagine you are holding Customer objects in memory for an extended period; you have a lot of them in a List<Customer>, and occasionally add more (with suitable threading protection, etc). Further, you have some pre-calculated subsets of the data (perhaps by region) – so a number of smaller List<Customer>. That is entirely inaccurate, but sets the scene ;p

After exhausting alternatives, what we did was:

change Customer from a class to a struct (only within this crazy code)
change the main store from a List<Customer> to a Customer[]
change the subsets from List<Customer> to List<int>, specifically the offset into the main Customer[]

eek; so what does that do for us?

the main data is now a single object (the array) on the "large object heap", rather than lots of small objects
by using direct access into an array (rather than a list indexer) you can access the data in-situ, so we are not constantly copying Customer values on the stack
the int offsets for subsets are essential to make sure we don't have multiple separate values of each record, but on x64 using int offsets rather than references also means our subsets suddenly take half the memory

Note that for some of the more complex code (applying predicates etc), we also had to switch to by-ref passing, i.e.

void SomethingComplex(ref Customer customer) {...}
...
int custIndex = ...
SomethingComplex(ref customers[custIndex]);

This again is accessing the Customer value in-situ inside the array, rather than copying it.

To finish off the bad-practice sheet, we also had some crazy changes to replace some reference data inside the record to fixed sized buffers inside the value (avoiding an array on the heap per item), and some corresponding unsafe code to query it (including the rarely-used stackalloc), but that code is probably a bit complex to cover properly - essentially: we removed all the objects/references from this data. And after removing the objects, there is nothing for GC to look at.

Impact

It helped! Here’s the “after”, at the same scales:

As you can see, there are still a few vertical spikes (which still tie into GC), but they are much less dense, and less tall. Basically, the server is no longer tied up in knots. The code is a bit less OOP than we might usually do, but it is a: constrained to this very specific scenario (this is absolutely not a carte-blanche “make everything a struct”), and b: for understood reasons.

Plus, it was pretty interesting (oh dear; I really, really need to get out more). Enjoy.

Disclaimers

You need to be really careful messing with this – especially with updates etc, since your records are certainly over-size for atomic read/write, and you don’t want callers seeing torn data.
This is not about “struct is faster” – please don’t let that be what you take away; this is a very specific scenario.
You need to be really careful to avoid copying fat values on the stack.
This code really is well outside of the normal bell-curve. Indeed, it is pretty resonant of XNA-style C# (for games, where GC hurts the frame-rate).

Credit

A lot of input, data-analysis, scheming, etc here is also due to Sam Saffron and Kyle Brandt; any stupid/wrong bits are mine alone.

The trouble with tuples

2011-10-07T15:22:00.000-07:00

A little while ago, I mentioned how new tuple handling in protobuf-net meant that likely-looking tuples could now be handled automatically, and even better this meant I could remove my hacky KeyValuePair<,> handling. All was well in the world – after all, what could go wrong?

Answer: Mono.

Actually, I’m not criticizing the Mono implementation really, but simply: Mono has a subtly different implementation of KeyValuePair<,> to Microsoft – nothing huge; simply there exist some set accessors (private ones, visible here). And the library was pretty fussy – if there was a set it wouldn’t be treated as a sure-thing tuple.

This is now fixed in r447, but: if you are using Mono and your dictionaries stopped serializing – then honestly and humbly: sorry about that.

And if you are a library author – watch out: sometimes the simplest most subtle differences between implementations can kill you.

Automatic serialization; what’s in a tuple?

2011-08-15T04:48:00.001-07:00

I recently had a lot of reason (read: a serialization snafu) to think about tuples in the context of serialization. Historically, protobuf-net has focused on mutable types, which are convenient to manipulate while processing a protobuf stream (setting fields/properties on a per-field basis). However, if you think about it, tuples have an implicit obvious positional “contract”, so it makes a lot of sense to serialize them automatically. If I write:

var tuple = Tuple.Create(123, "abc");

then it doesn’t take a lot of initiative to think of tuple as an implicit contract with two fields:

field 1 is an integer with value 123
field 2 is a string with value “abc”

Since it is deeply immutable, at the moment we would need to either abuse reflection to mutate private fields, or write a mutable surrogate type for serialization, with conversion operators, and tell protobuf-net about the surrogate. Wouldn’t it be nice if protobuf-net could make this leap for us?

Well, after a Sunday-night hack it now (in the source code) does.

The rules are:

it must not already be marked as an explicit contract
only public fields / properties are considered
any public fields (spit) must be readonly
any public properties must have a get but not a set (on the public API, at least)
there must be exactly one interesting constructor, with parameters that are a case-insensitive match for each field/property in some order (i.e. there must be an obvious 1:1 mapping between members and constructor parameter names)

If all of the above conditions are met then it is now capable of behaving as you might hope and expect, deducing the contract and using the chosen constructor to rehydrate the objects. Which is nice! As a few side-benefits:

this completely removes the need for the existing KeyValuePairSurrogate<,>, which conveniently meets all of the above requirements
it also works for C# anonymous types if we want, since they too have an implicit positional contract (I am not convinced this is significant, but it may have uses)

This should make it into the next deploy, once I’m sure there are no obvious failures in my assumptions.

Just one more thing, sir…

While I’m on the subject of serialization (which, to be fair, I often am) – I have now also completed some changes to use RuntimeHelpers.GetHashCode()for reference-tracking (serialization). This lets met construct a reference-preserving hash-based lookup to minimise the cost of checking whether an object has already been seen (and if so, fetch the existing token). Wins all round.

Shifting expectations: non-integer shift operators in C#

2011-08-12T13:21:00.000-07:00

In C#, it is quite nice that the shift operators (<< and >>) have had their second argument constrained to int arguments, avoiding the oft-confusing piped C++ usage:

cout << "abc" << "def";

But wait a minute! The above line is actually C#, in an all-C# project. OK, I cheated a little (I always do…) but genuinely! This little nugget comes from a stackoverflow post that really piqued my curiosity. All credit here goes to vcsjones, but indeed the line in the documentation about restricting to int (and the return type, etc) is about declaring the operator – not consuming it – so it is seemingly valid for the C# dynamic implementation to use it quite happily.

In fact, the main cheat I used here was simply hiding the assignment of the result, since that is still required. Here’s the full evil:

  using System;

  using System.Dynamic;

  using System.Linq.Expressions;



  class Evil : DynamicObject {

      static void Main() {

          dynamic cout = new Evil();

          var hacketyHackHack =

          cout << "abc" << "def";

      }

      public override bool TryBinaryOperation(

              BinaryOperationBinder binder,

              object arg, out object result) {

          switch(binder.Operation) {

              case ExpressionType.LeftShift:

              case ExpressionType.LeftShiftAssign:

                  // or whatever you want to do

                  Console.WriteLine(arg);

                  result = this;

                  return true;

          }

          return base.TryBinaryOperation(

               binder, arg, out result);

      }

   }

I've seen more evil things (subverting new() via ContextBoundObject is still my favorite evil), but quite dastardly, IMO!

Additional: grrr! I don't know why blogger hates me so much; here it is on pastie.org .

BookSleeve, transactions, hashes, and flukes

2011-07-04T03:54:00.001-07:00

Sometimes, you just get lucky.

tl;dr; version : BookSleeve now has transactions and Hashes; new version on nuget

Transactions

For some time now I’ve been meaning to add multi/exec support to BookSleeve (which is how redis manages transactions). The main reason it wasn’t there already is simply: we don’t use them within StackExchange.

To recap, because BookSleeve is designed to be entirely asynchronous, the API almost-always returns some kind of Task or (more often) Task-of-T; from there you can hook a “continue-with” handler, you can elect to wait for a result (or error), or you can just drop it on the floor (fire-and-forget).

Over the weekend, I finally got around to looking at multi/exec. The interesting thing here is that once you’ve issued a MULTI, redis changes the output – return “QUEUED” for pretty much every command. If I was using a synchronous API, I’d be pretty much scuppered at this point; I’d need to duplicate the entire (quite large) API to provide a way of using it. I’d love to say it was by-design (I’d be lying), but the existing asynchronous API made this a breeze (relatively speaking) – all I had to do was wrap the messages in a decorator that expects to see “QUEUED”, and then (after the EXEC) run the already-existing logic against the wrapped message. Or more visually:

conn.Remove(db, "foo"); // just to reset
using(var tran = conn.CreateTransaction())
{   // deliberately ignoring INCRBY here
    tran.Increment(db, "foo");
    tran.Increment(db, "foo");
    var val = tran.GetString(db, "foo");

    tran.Execute(); // this *still* returns a Task

    Assert.AreEqual("2", conn.Wait(val));
}

Here all the operations happen as an atomic (but pipelined) unit, allowing for more complex integrity conditions. Note that Execute() is still asynchronous (returning a Task), so you can still carry on doing useful work while the network and redis server does some thinking (although redis is shockingly quick).

So what is CreateTransaction?

Because a BookSleeve connection is a thread-safe multiplexer, it is not a good idea to start sending messages down the wire until we have everything we need. If we did that, we’d have to block all the other clients, which is not a good thing. Instead, CreateTransaction() creates a staging area to build commands (using exactly the same API) and capture future results. Then, when Execute() is called the buffered commands are assembled into a MULTI/EXEC unit and sent down in a contiguous block (the multiplexer will send all of these together, obviously).

As an added bonus, we can also use the Task cancellation metaphor to cleanly handle discarding the transaction without execution.

The only disadvantage of the multiplexer approach here is that it makes it pretty hard to use WATCH/UNWATCH, since we wouldn’t be sure what we are watching. Such is life; I’ll think some more on this.

Hashes

Another tool we don’t use much at StackExchange is hashes; however, for your convenience this is now fully implemented in BookSleeve. And since we now have transactions, we can solve the awkward varadic/non-varadic issue of HDEL between 2.2+ and previous versions; if BookSleeve detects a 2.2+ server it will automatically use the varadic version, otherwise it will automatically use a transaction (or enlist in the existing transaction). Sweet!

Go play

A new build is on nuget; I hope you find it useful.

Profiling with knobs on

2011-06-08T15:54:00.001-07:00

A little while ago, I mentioned the core part of the profiling tool we use at stackexchange – our mini-profiler. Well, I’m delighted to say that with the input from Jarrod and Sam this tool has been evolving at an alarming rate, and has now gained:

full integration with our bespoke SQL profiler, which is now also included
- basically, this is a custom ADO.NET DbConnection that you wrap around your existing connection, and it adds logging
an awesome UI (rather than html comments), including a much clearer timeline
full support for AJAX activity
some very basic highlighting of duplicated queries, etc
webform support
and a ton of other features and enhancements

To illustrate with Jarrod’s image from the project page:

The custom connection has been exercised extensively with dapper-dot-net, LINQ-to-SQL and direct ADO.NET, but should also work with Entity Framework (via ObjectContextUtils, provided) and hopefully any other DB frameworks built on top of ADO.NET.

We use it constantly. It is perhaps our primary tool in both understanding problems and just keeping a constant eye on the system while we use it – since this is running (for our developers) constantly, 24×7.

All too often such fine-grain profiling is just “there’s a problem, ask a DBA to attach a profiler to the production server for 2 minutes while I validate something”.

Enjoy.

http://code.google.com/p/mvc-mini-profiler/

So soon, beta 2

2011-05-25T16:45:00.001-07:00

I’m delighted by the level of response to my protobuf-net v2 beta; I have been kept really busy with a range of questions, feature requests and bug fixes (almost exclusively limited to the new features).

So, before I annoy people with too many “fixed in source” comments, I thought I’d better re-deploy. Beta 2 gives the same as beta 1, but with:

reference-tracked objects
- root object now included (BREAKING CHANGE)
- fixed false-positive recursion issue
full type metadata
- support custom formatting/parsing
interface serialization
- support serialization from interfaces
- support known-implementations of interfaces
- support default concrete type
WCF
- expose WCF types on public API to allow custom models
misc
- support shadow set-methods
- fix IL glitch
- fix cold-start thread-race condition
- fix missing method-forwarding from legacy non-generic API

In addition to a very encouraging level of interest, I’m also pleased that things like the interface-based support mentioned above were pretty effortless to push through the v2 type-model. These are things that had been proposed previously, but there was just no way to push them into the codebase without hideous (and cumulative) hackery. With v2, it was pretty much a breeze.

But for v2 fun see the project download page

And please, keep pestering me ;p

Note that the BREAKING CHANGE marked above only applies to data serialized with the new full-graph AsReference option. No v1 data is impacted, but if you have stored data from earlier v2 alpha/beta code, please drop me a line and I can advise.

protobuf-net v2, beta

2011-05-19T13:23:00.001-07:00

It has been a long time in the coming, I know. I make no excuses, but various other things have meant that it didn’t get much input for a while…

But! I’m happy to say that I’ve just pushed a beta download up onto the project site

So… this v2… what is it?

Basically, it is still the core protobuf stream, but against all the safe advice of my employer I rewrote the entire core. For many reasons:

the original design grew organically, and ideas that seemed good turned to make for some problematic code later
the vast overuse of generics (and in particular, generics at runtime via reflection) was really hurting some platforms
there were many things the design could not readily support – structs, runtime models, pre-built serialization dlls, usage on mobile devices, etc
the code path at runtime was just not as minimal as I would like
and a myriad of other things

So… I rewrote it. Fortunately I had a barrage of integration tests, which grew yet further during this exercise. The way I figured, I had 2 choices here:

go with a richer code-generator to use as a build task (not runtime)
go crazy with meta-programming

After some thought, I opted for the latter. In particular, this would (in my reasoning) give me lots of flexibility for keeping things runtime based (where you are able to, at least), and would be a good opportunity to have some fun and learn IL emit to dangerous levels (which, incidentally, turns out to be very valuable – hence mine and Sam’s work on dapper-dot-net).

So what is new?

I don’t claim this list is exhaustive:

the ability to define models at runtime (also useful for working with types you don’t control) (example)
the ability to have separate models for the same types
pre-generation to dll
runtime performance improvements
support for structs
support for immutable objects via surrogates (example)
the ability to run on mobile platforms (unity, wp7 – and perhaps moot for a while: MonoDroid, MonoTouch – but presumably Xamarin)
the ability to serialize object graphs preserving references (example)
the ability to work with types not known in advance (same)
probably about 40 other things that slip my mind

I’ll try to go into each of these in detail when I can

And what isn’t there yet?

This is beta; at some point I had to make a milestone cut – but some things aren’t quite complete yet; these are not usually needed (hint: I’m happy to use protobuf-net v2 on my employer’s code-base, so I have confidence in it, and confidence that the core paths have been smoke-tested)

But things that definitely aren’t there in the beta

WCF hooks
VS tooling (it is just the core runtime dlls)
round-trip safe / “extension” fields
extraction of .proto (schema) files from a model
my build / deploy script needs an overhaul
tons of documentation / examples
tooling to make the “mobile” story easier (but; it should work – it is being used in a few, for example)

But; have at it

If you want to play with it, go ahead. I only ask that if something behaves unexpectedly, drop me a line before declaring loudly “it sux, dude!”. It might just need some guidance, or maybe even a code fix (I’m far from perfect).

Likewise, any feature suggestions, etc; let me know.

Completion tasks - the easy way

2011-04-27T03:57:00.001-07:00

One of the good things about doing things in public – people point out when you’ve missed a trick.

Just the other day, I was moaning about how Task seemed too tightly coupled to schedulers, and wouldn’t it be great if you could have a Task that just allowed you to indicate success/failure – I even went so far as to write some experimental code to do just that.

So you can predict what comes next; it already existed – I just didn’t know about it. I give you TaskCompletionSource<T>, which has SetResult(…), SetException(…) and a Task property. Everything I need (except maybe support for result-free tasks, but I can work around that).

And wouldn’t you believe it, the TPL folks shine again, easily doubling my lousy performance, and trebling (or more) the performance compared to “faking it” via RunSynchronously:

Future (uncontested): 1976ms // my offering
Task (uncontested): 4149ms // TPL Task via RunSynchronously
Source (uncontested): 765ms // via TaskCompletionSource
Future (contested): 5608ms
Task (contested): 6982ms
Source (contested): 2389ms

I am hugely indebted to Sam Jack, who corrected me via a comment on the blog entry. Cheers!

Musings on async

2011-04-25T13:45:00.001-07:00

OUT OF DATE: SEE UPDATE

For BookSleeve, I wanted an API that would work with current C#/.NET, but which would also mesh directly into C# 5 with the async/await pattern. The obvious option there was the Task API, which is familiar to anyone who has used the TPL in .NET 4.0, but which gains extension methods with the Async CTP to enable async/await.

This works, but in the process I found a few niggles that made me have a few doubts:

Completion without local scheduling

In my process, I never really want to schedule a task; the task is performed on a separate server, and what I really want to do is signal that something is now complete. There isn’t really an API for that on Task (since it is geared more for operations you are running locally, typically in parallel). The closest you can do is to ask the default scheduler to run your task immediately, but this feels a bit ungainly, not least because task-schedulers are not required to offer synchronous support.

In my use-case, I’m not even remotely interested in scheduling; personally I’d quite like it if Task supported this mode of use in isolation, perhaps via a protected method and a subclass of Task.

(yes, there is RunSynchronously, but that just uses the current scheduler, which in library code you can’t assume is capable of actually running synchronously).

Death by exception

The task API is also pretty fussy about errors – which isn’t unreasonable. If a task fails and you don’t explicitly observe the exception (by asking it for the result, etc), then it intentionally re-surfaces that exception in a finalizer. Having a finalizer is note-worthy in itself, and you get one last chance to convince the API that you’re sane – but if you forget to hook that exception it is a process-killer.

So what would it take to do it ourselves?

So: what is involved in writing our own sync+async friendly API? It turns out it isn’t that hard; in common with things like LINQ (and foreach if you really want), the async API is pattern-based rather than interface-based; this is convenient for retro-fitting the async tools onto existing APIs without changing existing interfaces.

What you need (in the Async CTP Refresh for VS2010 SP1) is:

Some GetAwaiter() method (possibly but not necessarily an extension method) that returns something with all of:
A boolean IsCompleted property (get)
A void OnCompleted(Action callback)
A GetResult() method which returns void, or the desired outcome of the awaited operation

So this isn’t a hugely challenging API to implement if you want to write a custom awaitable object. I have a working implementation that I put together with BookSleeve in mind. Highlights:

Acts as a set-once value with wait (sync) and continuation (async) support
Thread-safe
Isolated from the TPL, and scheduling in particular
Based on Monitor, but allowing efficient re-use of the object used as the sync-lock (my understanding is that once used/contested in a Monitor, the object instance obtains additional cost; may as well minimise that)
Supporting typed (Future<T>) or untyped (Future) usage – compares to Task<T> and Task respectively

My local tests aren’t exhaustive, but (over 500,000 batches of 10 operations each), I get:

Future (uncontested): 1993ms
Task (uncontested): 4126ms
Future (contested): 5487ms
Task (contested): 6787ms

So our custom awaitable object is faster… but I’m just not convinced that it is enough of an improvement to justify changing away from the Task API. This call density is somewhat artificial, and we’re talking less than a µs per-operation difference.

Conclusions

In some ways I’m pleasantly surprised with the results; if Task is keeping up (more or less), even outside of it’s primary design case, then I think we should forget about it; use Task, and move on to the actual meat of the problem we are trying to solve.

However, I’ll leave my experimental Future/Future<T> code as reference only off on the side of BookSleeve – in case anybody else feels the need for a non-TPL implementation. I’m not saying mine is ideal, but it works reasonably.

But: I will not be changing away from Task / Task<T> at this time. I’m passionate about performance, but I’m not (quite) crazy; I’ll take the more typical and more highly-tested Task API that has been put together by people who really, really understand threading optimisation, to quite ludicrous levels.

Practical Profiling

2011-04-19T04:38:00.001-07:00

If you don’t know what is causing delays, you are doomed to performance woes. Performance is something I care about deeply, and there are no end of performance / profiling tools available for .NET. Some invasive and detailed, some at a higher level. And quite often, tools that you wouldn’t leave plugged into your production code 24/7.

Yet… I care about my production environment 24/7; and trying to reproduce a simulated load for the sites I work on can be somewhat challenging. So how can we get realistic and detailed data without adversely impacting the system?

Keep It Simple, Stupid

A common strapline in agile programming states:

Do the simplest thing that could possibly work

Now, I don’t profess to be strictly “agile”, or indeed strictly anything (except maybe “pragmatic”) in my development process, but there is much wisdom in the above. And it makes perfect sense when wanting to add constant (live) profiling capabilities.

So what is the simplest thing that could possibly work with profiling? Automated instrumentation? Process polling on the debugging API? Way too invasive. IoC/DI chaining with profiling decorators? Overkill. AOP with something like PostSharp? Unnecessary complexity. How about we just tell the system openly what we are doing?

Heresy! That isn’t part of the system! It has no place in the codebase!

Well, firstly – remember I said I was “pragmatic”, and secondly (more importantly) performance is both a requirement and a feature, so I have no qualms whatsoever changing my code to improve our measurements.

So what are you talking about?

Frustrated by the inconvenience of many of the automated profiling tools, I cobbled together the simplest, hackiest, yet fully working mini-profiler – and I thought I’d share. What I want is as a developer, to be able to review the performance of pages I’m viewing in the production environment – sure, this doesn’t cover every scenario, but it certainly does the job on sites that are read-intensive. So say I browse to “http://mysite/grobbits/MK2-super-grobit” – I want immediate access to how that page was constructed (and where the pain was). And in particular, I want it live so I can hit “refresh” a few times and watch how it behaves as different caches expire. Nothing rocket-science, just a basic tool that will let me hone in on the unexpected performance bumps. Finally, it can’t impact performance the 99.99% of regular users who will never see that data.

I’m currently having great success using this mini tool; the concept is simple – you have a MiniProfiler object (which would be null for most users), and you just surround the interesting code:

using (profiler.Step("Set page title"))
{
    ViewBag.Message = "Welcome to ASP.NET MVC!";
}

using (profiler.Step("Doing complex stuff"))
{
    using (profiler.Step("Step A"))
    { // something more interesting here
        Thread.Sleep(100);
    }
    using (profiler.Step("Step B"))
    { // and here
        Thread.Sleep(250);
    }
}

(the Step(…) method is implemented as an extension method, so it is perfectly happy operating on a null reference; which is a crude but simple way of short-circuiting the timings for regular users)

Obviously this isn’t very sophisticated, and it isn’t meant to be – but it is very fast, and easy to make as granular as you like as you focus in on some specific knot of code that is hurting. But for the simplicity, it is remarkably useful in finding trouble spots on your key pages, and reviewing ongoing performance.

So what do I see?

The output is very ~~basic~~ functional – you simply get a call tree of the code you’ve marked as interesting (above), to whatever granularity you marked it to; no more, no less. So with the sample project (see below), the home-page displays (in the html markup):

<!--
MYPC at 19/04/2011 11:28:12
Path: http://localhost:3324/
 http://localhost:3324/ = 376.9ms
> Set page title = 0ms
> Doing complex stuff = 349.3ms
>> Step A = 99.3ms
>> Step B = 250ms
> OnResultExecuting = 27.5ms
>> Some complex thinking = 24.8ms
-->

(for your developers only; there is no change for users without the profiler enabled)

Stop waffling, Man!

Anyway, if you have similar (high-level, but live) profiling needs, I’ve thrown the mini-profiler onto google-code and NuGet, along with a tweaked version of the MVC3 (razor) sample project to show typical usage. Actually, you only need a single C# file (it really is basic code, honest).

If you find it useful, let me know!
If it sucks, let me know!
If I’ve gone mad, let me know!
I’ve you’ve gone mad, keep that to yourself.

async Redis await BookSleeve

2011-04-11T04:48:00.001-07:00

At Stack Exchange, performance is a feature we work hard at. Crazy hard. Whether that means sponsoring load-balancer features to reduce system impact, or trying to out-do the ORM folks on their own turf.

One of the many tools in our performance toolkit is Redis; a highly performant key-value store that we use in various ways:

as our second-level cache
for various tracking etc counters, that we really don’t want to bother SQL Server about
for our pub/sub channels
for various other things that don’t need to go direct to SQL Server

It is really fast; we were using the redis-sharp bindings and they served us well. I have much thanks for redis-sharp, and my intent here is not to critique it at all – but rather to highlight that in some environments you might need that extra turn of the wheel. First some context:

Redis itself is single threaded supporting multiple connections
the Stack Exchange sites work in a multi-tenancy configuration, and in the case of Redis we partition (mainly) into Redis databases
to reduce overheads (both handshakes etc and OS resources like sockets) we re-use our Redis connection(s)
but since redis-sharp is not thread-safe we need to synchronize access to the connection
and since redis-sharp is synchronous we need to block while we get each response
and since we are split over Redis databases we might also first have to block while we select database

Now, LAN latency is low; most estimates put it at around 0.3ms per call – but this adds up, especially if you might be blocking other callers behind you. And even more so given that you might not even care what the response is (yes, I know we could offload that somewhere so that it doesn’t impact the current request, but we would still end up adding blocking for requests that do care).

Enter BookSleeve

Seriously, what now? What on earth is BookSleeve?

As a result of the above, we decided to write a bespoke Redis client with specific goals around solving these problems. Essentially it is a wrapper around Redis dictionary storage; and what do you call a wrapper around a dictionary? A book-sleeve. Yeah, I didn’t get it at first, but naming stuff is hard.

And we’re giving it away (under the Apache License 2.0)! Stack Exchange is happy to release our efforts here as open source, which is groovy.

So; what are the goals?

to operate as a fully-functional Redis client (obviously)
to be thread-safe and non-blocking
to support implicit database switching to help with multi-tenancy scenarios
to be on-par with redis-sharp on like scenarios (i.e. a complete request/response cycle)
to allow absolute minimum cost fire-and-forget usage (for when you don’t care what the reply is, and errors will be handled separately)
to allow use as a “future” – i.e request some data from Redis and start some other work while it is on the wire, and merge in the Redis reply when available
to allow use with callbacks for when you need the reply, but not necessarily as part of the current request
to allow C# 5 continuation usage (aka async/await)
to allow fully pipelined usage – i.e. issue 200 requests before we’ve even got the first response
to allow fully multiplexed usage – i.e. it must handle meshing the responses from different callers on different threads and on different databases but on the same connection back to the originator

(actually, Stack Exchange didn’t strictly need the C# 5 scenario; I added that while moving it to open-source, but it is an excellent fit)

Where are we? And where can I try it?

It exists; it works; it even passes some of the tests! And it is fast. It still needs some tidying, some documentation, and more tests, but I offer you BookSleeve:

http://code.google.com/p/booksleeve/

The API is very basic and should be instantly familiar to anyone who has used Redis; and documentation will be added.

In truth, the version I’m open-sourcing is more like the offspring of the version we’re currently using in production – you tend to learn a lot the first time through. But as soon as we can validate it, Stack Exchange will be using BookSleeve too.

So how about some numbers

These are based on my dev machine, running redis on the same machine, so I also include estimates using the 0.3ms latency per request as mentioned above.

In each test we are doing 5000 INCR commands (purely as an arbitrary test); spread over 5 databases, in a round-robin in batches of 10 per db – i.e. 10 on db #0, 10 on db #1, … 10 on db #4 – so that is an additional 500 SELECT commands too.

redis-sharp:

to completion 430ms
(not meaningful to measure fire-and-forget)
to completion assuming 0.3ms LAN latency: 2080ms

BookSleeve

to completion 391ms
2ms fire-and-forget
to completion assuming 0.3ms LAN latency: 391ms

The last 2 are the key, in particular noting that the time we aren’t waiting on LAN latency is otherwise-blocking time we have subtracted for other callers (web servers tend to have more than one thing happening…); the fire-and-forget performance allows us to do a lot of operations without blocking the current caller.

As a bonus we have added to ability to do genuinely parallel work on a single caller – by starting a Redis request first, doing the other work (TSQL typically), and then asking for the Redis result. And let’s face it, while TSQL is versatile, Redis is so fast that it would be quite unusual for the Redis reply to not to already be there by the time you get to look.

Wait – did you say C# 5?

Yep; because the API is task based, it can be used in any of 3 ways without needing separate APIs:

synchronous (blocking on the response) via Wait
asynchronous (callback-based) via ContinueWith
async/await by using the Async CTP (edit or for VS2010 SP1) and C# 5 language features

As an example of the last:

IMPORTANT: in the above “await” does not mean “block until this is done” – it means “yield back to the caller here, and run the rest as a callback when the answer is available” – or for a better definition see Eric Lippert’s blog series.

And did I mention…

…that a high perfomance binary-based dictionary store works well when coupled with a high performance binary serializer? ;p

Objects, Graphs and all that Jazz

2011-03-07T11:52:00.001-08:00

protobuf-net v2 ticks along slowly; I’m embarrassed to say that due to a combination of factors progress has been slower than I would have liked – and for purely human reasons (availability etc).

But; I thought I’d better update with the tweaks I’m looking at currently; they have all been ticking along in the back of head for ages, but frankly people kept nagging me to provide them, so who am I to argue?

Caveat

This is all work-in-progress; don’t try to pull the trunk and shove it into production! Soon… I should also stress that all of the thoughts described below are outside the interoperable protobuf spec; it’ll still be a valid protobuf stream, but this only applies to protobuf-net talking to protobuf-net.

So; here’s some common questions I get…

Type meta

The serialization is fine, but I don’t know (and cannot know) all of my types up front. How can I do this?

Well, protobuf is a contract based format; if you don’t know the types, it will struggle – as will any contract based serializer…

Yes, I get that; now: how do I do it?

Now, I’ve held off putting any meta in the stream for various reasons:

it steps far outside the core protobuf spec
it flashes the warning signs of BinaryFormatter, my nemesis

But, so many people seem to want this that I think I have to buckle; but on my terms! So in v2, I’m adding the ability to indicate that (on a per-member basis) objects should resolve their type information from the stream. By default, by embedding the assembly-qualified-name, but providing an abstraction layer over that allowing you to provide your own string<===>Type map (and thus avoiding the knots in by stomach caused by too much type dependency).

Full graphs

The serialization is fine, but my data is a graph, not a tree. How can I do this?

Well, protobuf is a tree format; it doesn’t work like that…

Yes, I get that; now: how do I do it?

(by the way, are you spotting a pattern in the questions I get?)

Type meta is something I didn’t want to add, but graph support is something I have really wanted to sneak in; breaking all the rules with type meta seems a reasonable excuse. So in v2, I’m adding the ability (on a per-member basis) to use reference tracking to (de)serialize a complete object graph (except events; don’t get me started on that…). A minor complication here is that for technical reasons it is a nightmare to support this concurrently with inheritance (which, btw, the core protobuf doesn’t support – I feel quite the anarchist here…); but I can support it in conjunction with type meta; so you still get to keep inheritance, but implemented differently.

Repeated strings

It is fairly common to have repeated string data in a large graph. As an offshoot of full-graph support, we also get a mechanism to support string re-use for free; woot! So again, in v2 I’ll be enabling this on a per-member basis.

Footnote

All of this goes so far outside of the protobuf spec that I have little right to even call it protobuf-related any more; maybe I should force the caller to explicitly set an EnableImplementationSpecificOptions flag? So there is an portable core with some opt-in hacks for these random options.

And again; this whole area is under development; I’m working on it, honest!

That’s great! My full-graph of dynamic objects using inheritance and string re-use works fine! Now how do I load it into my c++/java/php/python/whatever client?

Don’t make me hurt you…