(disclaimer: these are 8 requirements for me, yours may differ!)

Something to remember: When your hard drive dies, do you have your data safely stored somewhere else? A good test would be if all the hard drives in your pc all stopped working right now, would you lose any critical or valuable data? Do you have backups? Can you restore your backups? What about if your whole house burned down?

I’ve found that available backup software out there is either to complicated, doesn’t do what I want, or too expensive. I’ve listed out what I feel are my requirements for backup software (which means that they’re probably not your requirements!). Note that since my primary machine is running Windows, I’m looking at requirements that Windows supports.

I don’t need a full disk backup, rather on a file-by-file basis. When my drive crashes, I’d probably want to use the opportunity to re-install my operating system anyway.

1. Simple and easy to use

Obviously all software should be easy to use, but why are a lot of backup solutions so complex? Ask me what to backup, where to store the backup and how often to run. Otherwise, leave me alone (obviously if there is a problem, let me know!).

2. Simple backup format

I should be able to open windows explorer, and view the files in a backup set without having to run any special software. This also helps with the first requirement – if your files are stored as files on the drive, anything should be able to read them. When you lose your main drive, it makes it easier if you can plug in your backup drive, and just copy files off it.

3. Actually backup (and restore!) your files

This seems obvious, but I’ve seen backup solutions that will silently backup all your files, but then barf when asked to restore them (looking at you, Windows Backup). The only way to be sure is to try restore your data at regular intervals, no matter which system you use. If the storage format is simple, as above, this is less of an issue, but you should still test it on a regular basis!

4. Stay out of my way

You should not have to remember to run a backup. Backup software should run automatically, and quietly in the background. Once you’ve scheduled it, it should do the rest. Disconnecting an external drive half way through a backup should not cause any problems when you plug the drive in again – it should carry on, ideally where it left off.

5. Handle multiple destinations

I would like to backup to external hard drives, USB flash drives and network computers. The software should automatically handle things like drive letter changes for external USB drives.

I want to define a set of files to backup, then point the software at multiple places. When plugging in a USB drive, the software should automatically kick off a backup if one is scheduled. When connecting to a network, if the destination machine is available, start the backup.

6. Keep history

Being able to go back a couple of months and retrieve a file from then is very handy. The software should do intelligent diffing as well as cleanup of old backup sets. If I’ve changed one 1MB file within of a couple of hundred gigabytes of files, I don’t expect the next backup to take much more than 1MB of extra space.

7. Offline

I have too much data to realistically store online. Living in South Africa exacerbates it because we have such constrained bandwidth availability (e.g. ADSL with 3GB limit). An online backup solution, while nice, is not really an option for most South Africans. If you have a couple of megabytes of word docs, then by all means install something like Dropbox or SugarSync and be done.

8. Quick

Obviously there is a limit to how quick backup software can be – it still needs to enumerate over all files you’re choosing to backup, but a backup of a couple of hundred changed files (from a typical day of work) should take under 5 minutes to complete.

Other nice-to-have features

While these would be nice, they are not critical for me at the moment.

Encryption: Encrypting your files would be nice, but this would break the “Simple format” requirement above. Maybe have an external usb drive encrypted using TrueCrypt?

Handle locked files: Files that are in use should also be backed up. This would involve using Volume Shadow Copy.

Primarily, a backup solution needs to be simple, just work and stay out of my way. Anything I’ve left out?

Right, back to work.

(for those of you that don’t know, Mono is an open-source implementation of the .net CLR. Go here for more details. It is actually quite a bit more than that, but coming from using .net, that is the part the interests me – at least at the moment.)

Firstly, one of the interesting things about Mono, is that you can take an assembly compiled using Microsoft’s tools, and run it under Mono, without any changes. Obviously, if you’re talking to the windows registry or using MS specific p-invoke calls, it wont work, but if your code is 100% managed, the chance that it will just work is pretty high.

Tooling

In terms of actually developing directly on Linux and using the Mono tools (rather than the Microsoft ones), there is an IDE called MonoDevelop that comes very close to the functionality of Visual Studio – in places its better (it can even open and save Visual Studio solutions and projects).

All the other usual command line tools are there, most importantly, there is a MSBuild clone called x-build, which lets you use your existing build scripts.

And something that is important for me, NUnit runs fine on mono.

For porting existing projects to Mono, there is a tool, called MoMA, which scans your source code, flagging any areas it identifies as being a potential problem.

Benefits

The most obvious one is that of licensing. Since Mono is open-source, and runs on an open-source operating system, there are zero licensing costs. This is attractive if you’re wanting to have multiple servers running your application. Windows licenses for 20 servers is a non-trivial cost.

Mono is also cross-platform. It runs in environments and on devices where you won’t find the Microsoft .net framework. Taken from the mono site:

Mono is built to be cross platform. Mono runs on Linux, Microsoft Windows, Mac OS X, BSD, and Sun Solaris, Nintendo Wii, Sony PlayStation 3, Apple iPhone. It also runs on x86, x86-64, IA64, PowerPC, SPARC (32), ARM, Alpha, s390, s390x (32 and 64 bits) and more. Developing your application with Mono allows you to run on nearly any computer in existence.

Distribution

I’m not sure about older versions, but the latest version of Ubuntu has Mono installed already. A couple of the pre-installed applications are written in C#.

To be honest, I’ve only used Mono under Ubuntu, so cannot comment on its availability in other distributions, but the Mono website recommends openSUSE.

Conclusion

So, in conclusion, I’m very impressed, well done guys. I’ve not yet had time to take one of our larger projects and get it running on Mono, but that’s the next step.

This is more a note to myself, but below is a list of things I want to investigate further around Mono:

• Ease of porting a largish existing application to Mono
• Do common frameworks run on Mono? (things like ASP.NET MVC, NHibernate, Castle Windsor etc)
• Windows Forms support (this was spotty when I looked at it a couple of years ago – its meant to be better)
• Performance vs the Microsoft stack

We’ve recently seen a problem with the installer not upgrading older versions, but rather just installing on top of the existing version, which results in the application being listed twice in the “Add/remove programs” (Programs and Features in Vista) section, one for each version. This results in weird things happening, like uninstalling one of them leaves the other one still installed with all the application files removed.

I ensured that we kept the same upgrade guid, with the version number incrementing, but all looked good. After searching and trying different things, I finally found this gem, hidden away in the MS documentation:

Note that Windows Installer uses only the first three fields of the product version. If you include a fourth field in your product version, the installer ignores the fourth field.

Why?!??

Our version numbers are in the order of 3.2.0.1234, 3.2.0.1456 (where the 4th number increments based on the Subversion revision number). The installer was seeing the version numbers as 3.2.0, i.e. the same, and not uninstalling the old version. Changing the installer version number to 3.2.1234 fixes this.

The WP Super Cache plugin for WordPress is a pretty awesome plugin. The other day I was having problems with it not doing any caching and spitting out the following just before the head tag (where the wp_head() call is):

<!-- Page not cached by WP Super Cache. No closing HTML tag. Check your theme. -->

This was very odd. Everything I could find on the web regarding this problem was seeing this tag at the end of the html. Eventually I found a post that sounded similar to mine (and of course now I cannot find it again, unfortunately). It mentioned a plug-in using output buffering incorrectly. This makes sense, as looking in the source code of WP Super Cache, it outputs the message above in its output buffer handler if it cannot find a </html> closing tag. Needless to say, after a couple of hours fiddling, it turns out that it was a plug-in that I had written especially for the site, and it was doing exactly what the post mentioned (i.e. it was calling ob_end_flush() without calling ob_start()).

So it was all my fault, and I’ll probably do it in the future, hence this post!

So, note to self: If you see this in the future, check the plug-ins.

The reason for this is that @@IDENTITY gives you the last generated Id in the current session vs. SCOPE_IDENTITY() gives you the last generated Id in the current scope.

The current session includes any triggers that run on the table after the insert. The current scope is just the current stored procedure that you’re in.

We were bitten by this recently. We were using @@IDENTITY when inserted into a table that is replicated. Upon insert the replication trigger was firing, inserting a record somewhere else. @@IDENTITY was giving us the identity of the record the trigger inserted.

Changing the @@IDENTITY to SCOPE_IDENTITY() fixed the problem.

This was even nastier in that by total fluke, the identity was matching a record that already existed, so when we used the retrieved identity to insert into another table, we didn’t violate the foreign key constraint.

Lesson learnt!

Update: I’ve found a page with an in-depth look at the different techniques. Its for SQL Server 2000, but applies to 2005/2008 too.

Unfortunately, google managed to find the test site and ranked it higher than the proper site for a couple of the keywords. Not good! Its a simple fix to put in a mod_rewrite redirect rule, but its quite incredible how quickly google managed to find a site that has no incoming links.

I have a feeling that its related to the site map plugin that we’re using for wordpress, I think that it pinged google when we were developing the site, and google happily went along and indexed the site.

So, if you’re developing a site, and put up a test version, make sure that you either take it down when making the site live, or ensure that it redirects to the correct site.

In general, we’re very happy with the database, it is most of the time pretty fast, and easy to work with. The fact that it runs within our process also helps, as its easy to install and use. This is especially useful, as we are going to be running on relatively low-spec machines (laptops with 500MB ram).

We are however seeing a performance problem in SELECT statements with multiple joins (> 5).

This article warns:

In general, if a significant number of your queries require joins of more than five or six tables, you should consider denormalization.

We are seeing queries that take around 4 minutes, whereas running the same query on a proper sql server takes less than a second.

When dumping out the query plans, and reviewing it, we are seeing in most cases, a full table scan taking place, and joining of all rows before filtering. Running the same query on the full sql server (with the same schema), it uses the indexes, and filters as expected. I’m going to demonstrate this issue, and a possible solution to the problem (tests were done using SQL Compact 3.5 SP1).

I’ve created a simple schema, as defined in the diagram below (should be pretty familiar):

(all tables had 1000 records in them, with the exception of the order_line table, which had ~16 000 records)

I’m going to use the following query, which is a bit contrived, but it is complex enough to display the issue:

SELECT   product.name,
         COUNT([order].id) AS order_count
FROM     order_line
         LEFT OUTER JOIN [order]
           ON [order].id = order_line.order_id
         LEFT OUTER JOIN product
           ON product.id = order_line.product_id
WHERE    [order].id IN ('{list of ~100 integers}')
         AND product.quantity_in_stock < order_line.quantity
GROUP BY product.name
ORDER BY order_count DESC,
         product.name

In SQL Compact, this generates the following query plan (click the image for the full size):

On the right of the query plan, you can see it joining the order and order_line tables (takes ~97% of the time). The problem is that it does this join before it filters out the orders, thereby reading all the records in the order_line table (which is obviously big!). This query runes noticeably slower with more data. Running the query on the full SQL Server results in it using the indexes as expected.

The solution to this, is to manually optimize the query yourself, instead of relying on the database engine to do it for you, by removing the LEFT JOINs and inverting the query, using INNER JOINs rather, to force the engine to filter rows out as early as possible (you’d usually write the query like this anyway, if you were doing it by hand, but remember, we’re using NHibernate to generate the query).

Here is the above query, optimized:

SELECT   product.name,
         COUNT([order].id) AS order_count
FROM     [order]
         INNER JOIN order_line
           ON order_line.order_id = [order].id
         INNER JOIN product
           ON product.id = order_line.product_id
WHERE    [order].id IN ('{list of ~100 integers}')
         AND product.quantity_in_stock < order_line.quantity
GROUP BY product.name
ORDER BY order_count DESC,
         product.name

and the corresponding query plan:

If you examine the query plan, you’ll notice that it’s filtering the rows up front, rather than joining and then filtering.

So, in conclusion, it doesn’t seem that SQL Compact optimizes LEFT JOINs properly, and you have to optimize the query yourself to avoid table scans and poor performance.

One important fact to note is that we’re using NHibernate to perform the querying, and its generating sql with lots of LEFT JOINs in it (nothing wrong with that, most databases optimize this quite well). The number of joins are determined by what base object you choose to query on, and how many values from other objects you’re pulling back.

We’ve since replaced the generated queries that show this performance problem with hard-coded named queries, written with inner joins as described above.

The first option to avoid this problem is to not actually create test methods until you’re ready to implement them.

I know I like to sometimes lay out a list of tests I want to implement before actually writing them. If you want to do this, then I suggest you create the methods, but ensure that the test is ignored (in nUnit, put an [Ignore] attribute onto the method), or that it fails (in nUnit, Assert.Fail). This is especially important if you’re going to commit to your source control system your partially complete tests.

Doing it this way will ensure that

1. You don’t forget that you need to implement the test.
2. That someone doesn’t think that something is being tested when in actual fact it isn’t.

• Delicious
• Google Reader (shared items)

Ideally, I don’t want to have 2 lists on the side, especially when most items I both bookmark in Delicious and share in Google Reader.

Enter Yahoo Pipes. Its an incredibly easy-to-use and powerful tool. You can see the pipe I’ve created here: http://pipes.yahoo.com/pipes/pipe.info?_id=wAcslJ753RGkTFabBR50VA. If you click the “View” Source” link you can see how its put together.

The general idea is to:

1. add multiple sources (fetch feed)
2. union them together into one feed
3. Ensure they are unique, based on the id
4. Sort them by the publish date
5. Output the result

The end result is one rss feed that contains the unique list of links and is sorted properly. I just then point the WordPress RSS widget at that feed, and it works.

You can do a lot more fancy things, but what I’ve described above is the simplest.