Luke Quinane

Windows App Store

2009-11-04T22:22:00.001+11:00

I'm developing applications for PC and the iPhone at the moment. It looks like the iPhone application will be the first to market for two reasons: it is a simpler application (which always helps), and there is less hassle to get it out the door. The main reason there is less hassle developing an iPhone app (ignore Apple's approval process) is because most common things are managed for the developer.

The iPhone App Store

I can't say I'm a big fan of owning hardware that you have little or no control over but I do love both my iPhone and my Xbox 360. These controlled environments provide several benefits to the consumer:

People feel safe using their credit cards to buy software. I don't think people are very keen to use their credit card at any old place - they don't want to be ripped off by paying for something that doesn't arrive and they don't want their credit card details stolen for other unauthorized uses.
There is no fear of malware. Having all the software up for sale screened by a third party prevents malware from creeping in. Even if it does get in the vendor can release an update to disable it remotely.

There are also many benefits for the developer:

No need for a complex installer. This is a huge pain on Windows and Linux (I'm not sure about OSX) and having a simple standard means of deploying the application is great.
Distribution handled. The developer doesn't need to find or pay for hosting applications even free ones.
Payment handled. The payments from the customer are handled automatically.
Licensing handled. There is no need for a custom activating and licensing mechanism.
Error reports handled. On Windows there is winqual but it requires a $99 certificate, avoid that cost requires a custom solution.
Single payment. The app store vendor takes a cut of sales so there is no separate cost for a winqual certificate, signing certificate, hosting, etc.

Windows App Store

An app store for Windows would address several key problems for desktop development:

Remove the need for custom installers.
Automatic updates and a single update notification. It seems like almost everyone does a poor job of updates and on my PC I have separate Adobe, Java, Apple and Google processes checking for updates.
Distribution. No need to find hosting.
Unified "feel" for applications. Consumers would have a single spot they felt safe using their credit card and knowing they weren't downloading malware. A great example of this on Windows is Steam.
Mandated quality standards. This would hopefully move towards removing crapplets on PCs.
A easy & safe way to make payments.

Microsoft already have "Windows Marketplace" which apparently supports third-party titles however a quick look at the site fails to show any such applications. I think at a minimum they should re-brand whatever app store they make perhaps using Bing. "Bing App Store"?

Perhaps Valve will moving into distributing applications as well as games...

Secure Updates?

2009-09-30T23:20:00.001+10:00

Currently I'm looking into secure automatic updates for a .net program I'm developing. I've asked on stackoverflow for people to review my approach to the update process here. In the process I've come across some interesting articles on automated updates and I thought I'd review how other applications go about it.

Java

Downloads http://java.sun.com/update/1.6.0/map-1.6.0.xml which has details of various updates:

<java-update-map version="1.0">
   <mapping>
      <version>1.6.0-rc-b98</version>
      <url>http://javadl-esd.sun.com/update/1.6.0/au-descriptor-1.6.0_15-b71.xml</url>
   </mapping>
   <mapping>
      <version>1.6.0</version>
      <url>http://javadl-esd.sun.com/update/1.6.0/au-descriptor-1.6.0_15-b71.xml</url>
   </mapping>

   ...

</java-update-map>

Then downloads the specific file given in the 'url' element (e.g: http://javadl-esd.sun.com/update/1.6.0/au-descriptor-1.6.0_15-b71.xml):

<java-update>

  <information version="1.0" xml:lang="en">
    <caption>Java Update - Update Available</caption>
    <title>Java Update Available</title>
    <description>Java 6 Update 15 is ready to install. Click the Install button to update Java now.  If you wish to update Java later, click the Later button. To get a FREE copy of OpenOffice.org, the global standard in free, Microsoft compatible office productivity software, just click the More Information link below.</description>
    <moreinfo>http://java.com/infourl</moreinfo>
    <AlertTitle>Java Update Available</AlertTitle>
    <AlertText>A new version of Java is ready to be installed.</AlertText>
    <moreinfotxt>More information...</moreinfotxt>
    <url>http://javadl-alt.sun.com/u/ESD6/JSCDL/jre/6u15-b71/jre/jre-6u15-windows-i586-iftw.exe</url>
    <version>1.6.0_15-b03</version>
    <post-status>https://sjremetrics.java.com/b/ss//6</post-status>
    <cntry-lookup>http://jal.sun.com/webapps/installstat/CountryLookup</cntry-lookup>
    <predownload></predownload>
    <options>/installmethod=jau SP1OFF=1 SP2OFF=1 SP3OFF=1 SP5OFF=1 SP6OFF=1 SP7OFF=1 SP8OFF=1 SP10OFF=1 MSDIR=ms4 NEWMSTB=1 SPWEB=http://javadl-esd.sun.com/update/1.6.0/sp-1.6.0_15-b71</options>
    <urlinfo>6068ce6c957932593d20059bebab0dfc8b056ac3</urlinfo>
  </information>

  ...

</java-update>

It does download a file from VeriSign which may be a check for the downloaded binary, e.g. http://ocsp.verisign.net/MFEwTzBNMEswSTAJBgUrDgMCGgUABBRODEXefhs%2FUZFum2o8YfzOFwceMwQUkz5j3yJ0BOBkhDHd2yOfDq%2B2TZMCEA89qsgV9niZmSI6gIO0S%2FU%3D

Paint.net

Downloads http://getpaint.net/updates/versions.5.600.x86.en.txt which looks like this:

; 3.xx manifest

DownloadPageUrl=http://www.getpaint.net/download.html

StableVersions=3.36.3158.38068
BetaVersions=3.50.3550.40197

3.36.3158.38068_Name=Paint.NET v3.36
3.36.3158.38068_NetFxVersion=2.0.50727
3.36.3158.38068_InfoUrl=http://www.getpaint.net/roadmap.html#v3_0
3.36.3158.38068_ZipUrlList=http://www.getpaint.net/updates/zip/Paint.NET.3.36.zip
3.36.3158.38068_FullZipUrlList=http://www.getpaint.net/updates/zip/Paint.NET.3.36.zip

3.50.3550.40197_Name=Paint.NET v3.5 Beta 1 (Build 3550)
3.50.3550.40197_NetFxVersion=3.5.1
3.50.3550.40197_InfoUrl=http://paintdotnet.forumer.com/viewtopic.php?f=46&t=31684
3.50.3550.40197_ZipUrlList=http://www.getpaint.net/files/zip/preview/Paint.NET.3.5.Beta.3550.Update.zip,http://www.dotpdn.com/files/Paint.NET.3.5.Beta.3550.Update.zip
3.50.3550.40197_FullZipUrlList=http://www.getpaint.net/files/zip/preview/Paint.NET.3.5.Beta.3550.Install.zip,http://www.dotpdn.com/files/Paint.NET.3.5.Beta.3550.Install.zip

~~This manifest file and the associated binaries don't appear to be signed in any way~~. Update: Rick Brewster has commented to say that the downloaded binary is signed and the signature is verified.

Skype

Checks for updates by querying: http://204.9.163.158/ui/0/4.0.0.227./en/getnewestversion?ver=4.0.0.227&uhash=143c299d62b6baac1f30936f72dd4683f

4.1.0.166

Downloads the update file: http://163-158.static.quiettouch.com/ui/0/4.0.0.227./en/upgrade
Unless the downloaded binary's signature is checked this would be vulnerable to a man in the middle attack.

Other

iTunes posts a big blob of data back to Apple on startup checking for updates - I didn't look into this much.
Google Chrome also posts back information when checking for updates. My version of Chrome was up to date so I didn't see the update process in action.
Firefox was already in the process of downloading a new version so I had already missed the file download negotiation.

Summary

It seems like everyone is doing automatic updates differently (no surprises there). It also looks like there is plenty of scope for man in the middle and spoofing attacks if the downloaded binaries aren't signed or don't have their signatures. It doesn't seem like many people are checking their manifest files before downloading binaries which could lead to Safari style "carpet bombing" where malicious binaries are downloaded onto the system.

Roomba Review

2009-09-29T13:56:00.001+10:00

Last Christmas we got a Roomba 530 (you can see a similar model here). A Roomba is a robotic vacuum cleaner, however, it mostly collects rubbish (dirt, garbage, hair, etc) using it's system of brushes. Since it is battery powered it doesn't have a very strong vacuum action. Having had Roomba in our house for almost a year I thought I give a quick review of it's pros and cons.

Firstly before I go into the details of the actual Roomba unit there are a few important background details:

I live in a small unit that has all hardwood floors.
I also have two cats who generate a lot of cat hair.

This combination is almost perfect for an automated cleaner: small amounts of dust and cat hair need cleaning every few days, there is no carpet to soak up dirt.

Pros

Roomba is awesome!!!11

It is quite cool to have an automated robot to go and clean your floors while you sit back and watch. And it is very mesmerising at first the way Roomba negotiates chair legs, people and other obstacles. Its a cliché but I do wonder if the next generation will even know what a "manual" vacuum cleaner is.

Roomba is small

In a small unit space is important, Roomba takes up a fraction of the space of a traditional vacuum cleaner.

Roomba cleans where you can't

To paraphrase the iRobot promotional video, 'Roomba cleans under stuff'. I am constantly surprised at how much gunk it is able to find underneath furniture like our couch (out of sight out of mind I guess).

Fire and forget

You can put Roomba on and go out; it will happily clean and re-dock itself once its done. However it practice you need to clear the floor of cables, etc. Otherwise you will come back to find Roomba has eaten the cord of your curtains and is hopelessly stuck.

Quiet

Roomba is noisy, but compared to a regular vacuum cleaner it is very quiet. My cats would run away when we started to put together the old vacuum cleaner but with Roomba they are happy to hang around and watch it with disdain.

Cons

Maintenance, maintenance, maintenance

The main problem with Roomba is that while it cleans your house without much supervision you need to clean it. After every clean you need to empty its pathetically small bin and after most cleans you need to clean the main brushes. Every three or four cleans I have to also remove hair from it's wheels and in the bearings of the main brushes. Occasionally you have to unscrew the 'flicking' brush to remove some long string that its eaten. Finally after about 8 months I notices that Roomba was no longer driving around in a straight line and I had to clean its sensors with a cotton bud.

Hair on the brushes	Hair in the bearings
Hair cleaned off the brushes	Gunk from the sensors

Fire and forget?

Fire and forget is sometimes like fire and get stuck: unless you prepare your floor removing any big particles (or are insanely neat) Roomba will eat a rubber band, pen, string, cable or similar and get is main brushes stuck. Usually I put Roomba on and then start to clear things that it might get caught on.

Dark furniture

Roomba uses several sensors to work out if it is going to bump into things before it actually triggers its big bumper. Unfortunately these sensors don't pick up dark furniture very well and as a result Roomba bangs into things (quite loudly). Short of putting reflector strips at Roomba height you can always move the furniture or just ignore it.

Summary

Roomba is awesome and I love it but it does require maintenance. Having seen it in action I feel that it works best on hard floors but it seemed to work well on carpet when I tried it at my parent's place. The main advantages are being able to put it on and leave it and also that it cleans under furniture. It does have some annoying quirks but nothing that has bothers me too much.

Epic Meltdown

2009-09-25T23:06:00.001+10:00

August Links

2009-08-24T22:40:00.001+10:00

Utility

Online text case converter

VM Ware

Automating VMWare Tasks in C# with the VIX API

VMWareTasks: C# VixCOM

Misc

How to write a disassembler

iPhone Reaching Critial Mass

2009-08-18T13:44:00.001+10:00

It seems to me that the iPhone is beginning to reach critical mass. And by critical mass I mean that the iPhone is reaching the status of the iPod or Nokia 5110: everyone has one. Part of my reasoning is that everyone I know seems to be talking about their iPhone or planning to get one. (Of course my sample size is very small and probably biased, but hey).

Travelling to work and back home everyday it seems like heaps of people already have them. Since it is becoming more popular is also seems to becoming less flashy and pretentious. Admittedly I live and work in inner Sydney so the sample set may be biased again.

And what alternative is there? In the same way that PC makers are struggling to design and build PCs that match Apple's sleek designs no one seems to be making a viable iPhone competitor. Also the AppStore is a very attractive place to put an application since consumers are quite comfortable using it to purchase software. Which in turn means all the developers are looking at developing for the iPhone.

It all adds up to one thing: critical mass and unfortunately lock-in. And the scary thing is that everyone seems comfortable that Apple won't abuse its position and become an evil monopoly.

Microsoft

Since it seems like Microsoft has lost the lead in the mobile space I think the only way out of the Apple-opoly is to bring their awesome set of developer tools to the iPhone. The lure of Visual Studio and C# / VB.net shouldn't be under estimated. If developers could easily make cross platform applications that ran on both the iPhone and Windows Mobile/Blackberry/Android it would be a big step towards breaking that lock-in.

Who's working towards bringing the C# and the CLR to the iPhone? Novell.

It seems like a strange move but I'm very keen to try it out...

TWAIN Dot Net

2009-06-20T22:02:00.001+10:00

I just released the first version of TwainDotNet, a TWAIN library for the .net framework. It is an open source, MIT licensed work built on top of Thomas Scheidegger's article from The Code Project: .NET TWAIN image scanner.

I had two main reasons for starting the project on code.google.com. Firstly code attached to an article provides no way for people to collaborate. There is no issue tracking, not place to feed back changes, etc. Secondly, I wanted to change the code to work with WPF and feed those changes back to everyone else!

So anyway the code is now up there and an initial binary available for download. I've picked the MIT license since it seems to fit the spirit of Thomas's original public domain dedication but with limitation of liability. And hopefully now there will be a bit more collaboration! :)

On a side note I've tried out Mercurial for the first time. The python based TortoiseHg is not quite as polished at TortoiseSvn but it is still very easy to use. I will seriously be thinking about moving my other Subversion projects over to Mercurial at a point in the future...

Exchange Wedding Anniversary?

2009-06-01T19:10:00.001+10:00

I saw this in the Microsoft "Exchange Server Protocols Master Property List Specification":

2.932 PidTagWeddingAnniversary

Canonical name: PidTagWeddingAnniversary
Property ID: 0x3A41
Data type: PtypTime, 0x0040
Area: MapiMailUser
References: [MS-OXOABK], [MS-OXOCNTC]
Alternate names: PR_WEDDING_ANNIVERSARY

Its quite funny how much detail you can put onto a contact in Exchange!

June Links - ASP.net Deployment & Misc

2009-06-01T18:59:00.001+10:00

ASP.net

HTML Sanitisation

NSsh Progress

2009-05-22T23:59:00.001+10:00

I've made some more progress with NSsh recently. Password based authentication is now basically complete. The next hurdle is getting public key authentication working - which is not only extremely useful but required by the SSH specification.

Surprisingly the main hurdle is not the task of validating the key that the user sends over for authentication (there are plenty of open source examples for that). Instead the Windows API is causing problems, specifically creating a process for the user.

To create a process for a user a token is required. This token can easily be created using Windows API calls that take a password. However without a password the alternatives are less attractive. If the machine is on a domain it is possible to perform Kerberos authentication using a constructor on the WindowsIdentity class. But, the only other alternative is to use the CreateToken / NtCreateToken functions. As one person puts it:

"It's possible, although it requires you to do a lot of code."

Hmmm, that doesn't sound good.

Export Processing

2009-05-22T23:42:00.001+10:00

Here is a screen shot of Process Explorer's system information dialog while my export processing code is running. The glitches in the IO graph are the write-cache thread flushing its data in the background. Anyway I think it looks nice :-)

Smart Pointers != Managed Automatic Memory Management

2009-04-14T21:03:00.001+10:00

I posted an answer on a stackoverflow question about C++ and garbage collection. My answer wasn't ideal and caused a bit of discussion in the comments so I thought I would just sum things up here.

Manual (Traditional) Memory Management

Manual memory management places the onus of memory management on the programmer. It is the style of memory management available to C and C++ developers. Here are some of the attributes of manual memory management:

Slow free-list allocation. To allocate memory in this style of memory management the runtime library consults a free list to find a free block. When allocating lots of blocks this can be slower than bump-pointer allocation.
No locality benefits. Blocks allocated sequentially won't usually be allocated next to each other. Later when some blocks are freed the remaining blocks a fixed in place and can't be moved.
Data is harder to share between modules. Module writers must manually decide who is responsible for freeing data and when it can be freed.
Invalid and dangling pointers are possible. Blocks can be freed multiple times. Every allocation must be checked. Etc, etc, etc..

Automatic Dynamic Memory Management

This is usually referred to as garbage collection and is the type of memory management used in Java, C#, Python, Ruby, etc, etc. Here are some memory management strategies possible with automatic GC:

Fast bump pointer allocation. Objects can be allocated using a bump pointer and then promoted to a free-list manage space only if they live. Alternatively a copying or compacting GC may always allocate with a bump pointer.
Locality benefits. Since objects can be moved and references updated objects can be allocated and later moved to improve locality. This results in better CPU cache usage.
No pointer problems, easy to share data between modules, etc, etc.

Smart Pointers

Some people argue that the answer to the problems of manual memory management is using smart pointers. They are certainly better than using nothing however they only provide the most primitive implementation of reference counting:

Unbounded cost on decrement. If the root of a large data structure is decremented to zero there is an unbounded cost to free all the data. (I personally find this property amusing since this problem is often an argument against GC)
Manual cycle collection. To prevent cyclic data structures from leaking memory the programmer must manually break any potential structures by replacing part of the cycle with a weak smart pointer. This is simply another source of potential defects.

First Class Reference Counting

Reference counting garbage collection has several advantages over smart pointers (beyond the base GC advantages).

Changes to an object reference count are ignored for stack and register references. Instead when a GC is triggered these objects are retained by collecting a root set.
Changes to the reference count can be deferred and processed in batches. This results in higher throughput.
It is possible to coalesce changes to the reference count. This makes it possible to ignore most changes to an objects reference count improving RC performance for frequently mutated references.
Incremental cycle detection cost. It is possible to process cycle detection tasks in bounded chunks at GC time.

Summary

Automatic memory management has several technical advantages over manual memory management. When you factor in the reduced cost of development and maintenance when using garbage collection you should be in front.

For details of a high performance reference counting GC implementation look at this paper.

Net Censorship in Australia

2009-03-25T21:08:00.001+11:00

Current the planned Australian Internet filter and the Australian Communications and Media Authority (ACMA) are causing quite a bit of controversy. The ACMA's Internet blacklist was leaked and there were some disturbing entries on the list. Among the entries were a Queensland dentist's website, a tour operator, and after disclosing the contents, wikileaks has been added to the list.

Transparency

There are two problems with the current blacklist as it stands. Firstly, a site that links to a blacklisted or illicit site is also blacklisted. The prime example of this is wikileaks.org which was blacklisted after disclosing the ACMA's blacklist. Media outlets have also been threatened with an $11,000 a day fine for linking to banned sites. This is insanity:

What if you link to a site then at a later date it posts offending material?
What if a site is placed onto the blacklist after you've linked to it?
What if a user posts a black listed link onto a forum you control?
The link itself doesn't contain any illicit material.

It seems like a violation of free speech to prevent someone from discussing illicit material, how is linking to it any different? Of course the linking issue it the key issue being played out in music sharing cases such as the current trial for "The Pirate Bay".

Secondly, the list is secret and thus there is no accountability. The article on ComputerWorld claims that the decision to add a site to the list can be made by a single bureaucrat. Without the list being disclosed how can there be any accountability for government. They may argue that disclosing the list will provide a central point for people to look for the offending illicit material (child pornography, etc). However:

People know how to get to illicit material anyway. Pedophile networks can be extremely sophisticated and secure.
If the governments filter is put in place all those sites will be blocked anyway.

Leave it to the Experts

There is already a lot of evidence that the Internet filtering plan will not work out (including ACMA's own reports). Instead of entering into an unaccountable Internet censorship setup, I feel that we should direct the funding into our state and federal police bodies. They are already doing a great job but could use the extra support. In NSW the police have just recently been given a boost to their online powers but only with a warrant - already much more accountable than a secret list.

Interesting Links

2009-03-17T22:58:00.001+11:00

SOLID principles as motivational pictures - an excellent succinct way to explain the SOLID principles.
ohloh - a site that provides stats on open source projects.
Parallel extensions for .net - a library for writing parallel applications in .net. Daniel Moth explains some of the details (improved thread pool, tasks, parallel for loops).

Java Exceptions

2009-03-08T23:34:00.002+11:00

Checked exceptions... they cause me pain. At first glance they seem to provide a way for design by contract to include exceptions. However they only increase complexity and have limited usefulness for design by contract.

SNR

Firstly, they increase the "signal-to-noise ratio" for the code. The SNR for code determines how clear your code is (someone else coined that term - but try Google for it!). Anders Hejlsberg also talks about imperative vs declarative programming which is a similar concept. Anyway consider the following code snippets:

Update UI from non UI-thread in Java:

try
{
   // Run the update code on the Swing thread
   SwingUtilities.invokeAndWait(new Runnable()
   {
       @Override
       public void run()
       {
           try
           {
               // Update UI value from the file system data
               FileUtility f = new FileUtility();
               uiComponent.setValue(f.readSomething());
           }
           catch (IOException e)
           {
               throw new IllegalStateException("Error performing file operation.", e);
           }
       }
   }
}
catch (InterruptedException ex)
{
   throw new IllegalStateException("Interrupted updating UI", ex);
}
catch (InvocationTargetException ex)
{
   throw new IllegalStateException("Invocation target exception updating UI", ex");
}

Update UI from non UI-thread in C#:

private void UpdateValue()
{
   // Ensure the update happens on the UI thread
   if (InvokeRequired)
   {
       Invoke(new MethodInvoker(UpdateValue));
   }
   else
   {
       // Update UI value from the file system data
       FileUtility f = new FileUtility();
       uiComponent.Value = f.ReadSomething();
   }
}

The UI threading code in the C# example still adds some extra noise so I like to use Postsharp to reduce it down to this:

[UIMethod]
private void UpdateValue()
{
   // Update UI value from the file system data
   FileUtility f = new FileUtility();
   uiComponent.Value = f.ReadSomething();
}

Which seems a lot clearer to me. When you start to do more and more UI work in Swing checked exceptions start to become really annoying and useless.

Jail Break

To implement even the most basic of implementations, such as Java's List interface, checked exceptions as a tool for design by contract fall down. Consider a list that is backed by a database or a filesystem or any other implementation that throws a checked exception. The only possible implementation is to catch the checked exception and rethrow it as an unchecked exception:

public void clear()
{
   try
   {
       backingImplementation.clear();
   }
   catch (CheckedBackingImplException ex)
   {
       throw new IllegalStateException("Error clearing underlying list.", ex);
   }
}

And now you have to ask what is the point of all that code? The checked exceptions just add noise, the exception has been caught but not handled and design by contract (in terms of checked exceptions) has broken down.

Service, s'il vous plaît?

When the cost of checked exceptions starts to kick in (wrapping in unchecked exceptions, millions of checked exceptions) you can see code like this in response:

try
{
   // Perform some action
}
catch (Exception e)
{
   log.warn("Unexpected exception processing action", e);
}

The problem with this code is well documented in the .net world. Basically if you can't handle the exception then step aside.

Summary

In non-trivial situations all of the above problems compound making debugging and maintaining code harder. They add noise to the code and the usefulness as a design by contract mechanism is questionable.

Really, in Java the divide between checked exceptions and runtime exceptions should have been a red flag: some exceptions are are unexpected. Exceptions are for exceptional circumstances and you don't always want to deal with exceptional circumstances at all levels in your code.

Java Suppress Finalizer

2009-03-03T19:29:00.005+11:00

A colleague pointed my to Dr Heinz M. Kabutz's article on Java finalizers. The article compares the performance of objects that have a trivial (empty) finalizer vs those that have a very simple finalizer. The basic outcome is that the running time for the non-trivial case is orders of magnitude larger than the trivial case.

I was quite shocked when I saw this since the simple finalizer given in the example is similar to what is proposed in .net IDisposable pattern. I was thinking "Holy Smoke, Batman! How can it really be that bad??". The answer is in Jack Shirazi's article: basically the finalizer causes the object to live through the nursery collection and thus puts much more pressure on the mature space collector.

.net?

I decided to implement the test in C# quickly and see how it performed. Here is the code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace ConsoleApplication1
{
public class ConditionalFinalizer
{
    private static readonly bool DEBUG = false;

    // Should be volatile as it is accessed from multiple threads.
    // Thanks to Anton Muhin for pointing that out.
    private volatile bool resourceClosed;
    private readonly int id;

    public ConditionalFinalizer(int id)
    {
        this.id = id;
        resourceClosed = false;
    }

    ~ConditionalFinalizer()
    {
        if (DEBUG)
        {
            if (!resourceClosed)
            {
                Console.Error.WriteLine("You forgot to close the resource with id " + id);
            }
            resourceClosed = true;
        }
    }

    public void close()
    {
        resourceClosed = true;
        GC.SuppressFinalize(this);
    }
}

class Program
{
    static void Main(string[] args)
    {
        // Allow the JIT to warm up
        for (int i = 0; i < 10 * 1000 * 1000; i++)
        {
            ConditionalFinalizer cf = new ConditionalFinalizer(i);
            if (i % (1000 * 1000) != 0)
            {
                cf.close();
            }
        }


        DateTime start = DateTime.Now;

        for (int i = 0; i < 10 * 1000 * 1000; i++)
        {
            ConditionalFinalizer cf = new ConditionalFinalizer(i);
            if (i % (1000 * 1000) != 0)
            {
                cf.close();
            }
        }

        TimeSpan time = DateTime.Now - start;
        Console.WriteLine("time = " + time.TotalMilliseconds);

        Console.ReadKey();
    }
}
}

The most important change is the added call to "GC.SuppressFinalize". This test case is the specific reason for this call - basically it removes the object from the finalization queue thus allowing it to be collected in the nursery collection. I also added an initial loop to allow the JIT to compile the classes in use. Here are the results:

Calling "GC.SuppressFinalize" dramatically improves the performance for .net and my initial results line up with Dr Kabutz's. Java really does a neat optimisation for the trivial finalizer case! Unfortunately it also gets smashed in the non-trivial case...

Java

Dr Kabutz's code for showing the classes pending finalization goes 90% of the way towards providing something similar to "GC.SuppressFinalize", I decided to add that last little bit:

import java.lang.ref.Reference;
import java.lang.reflect.Field;

public class FinalizeHelper {

static FinalizeHelper finalizeHelper = new FinalizeHelper();

private final Class<?> finalizerClazz;
private final Object lock;
private final Field unfinalizedField;
private final Field nextField;
private final Field prevField;
private final Field referentField;

public FinalizeHelper() {
    try {
        finalizerClazz = Class.forName("java.lang.ref.Finalizer");

        // we need to lock on this field to avoid racing conditions
        Field lockField = finalizerClazz.getDeclaredField("lock");
        lockField.setAccessible(true);
        lock = lockField.get(null);

        // the start into the linked list of finalizers
        unfinalizedField = finalizerClazz.getDeclaredField("unfinalized");
        unfinalizedField.setAccessible(true);

        // the next element in the linked list
        nextField = finalizerClazz.getDeclaredField("next");
        nextField.setAccessible(true);

        // the prev element in the linked list
        prevField = finalizerClazz.getDeclaredField("prev");
        prevField.setAccessible(true);

        // the object that the finalizer is defined on
        referentField = Reference.class.getDeclaredField("referent");
        referentField.setAccessible(true);

    } catch (RuntimeException e) {
        throw e;
    } catch (Exception e) {
        throw new IllegalStateException("Could not create FinalizeHelper", e);
    }
}

private void suppress(Object instance) {
    try {
        synchronized (lock) {
            // Get the start of the un-finalized list
            Object current = unfinalizedField.get(null);
            Object previous = null;

            while (current != null) {
                Object value = referentField.get(current);

                // Check if this entry refers to the instance we are interested in
                if (value == instance) {
                    // Unlink the current entry from the queue
                    Object next = nextField.get(current);                    
                    if (previous == null) {
                        unfinalizedField.set(null, next);
                        prevField.set(next, null);
                    } else {
                        nextField.set(previous, next);
                        prevField.set(next, previous);
                    }
                    break;
                }

                // Move to the next entry
                previous = current;
                current = nextField.get(current);
            }
        }
    } catch (IllegalAccessException e) {
        throw new IllegalStateException(e);
    }
}

public static void suppressFinalize(Object instance) {
    finalizeHelper.suppress(instance);
}
}

Here is the updated test files:

public class ConditionalFinalizer {
private static final boolean DEBUG = false;

// Should be volatile as it is accessed from multiple threads.
// Thanks to Anton Muhin for pointing that out.
private volatile boolean resourceClosed;
private final int id;

public ConditionalFinalizer(int id) {
    this.id = id;
    resourceClosed = false;
}

protected void finalize() throws Throwable {
    if (DEBUG) {
        if (!resourceClosed) {
            System.err.println("You forgot to close the resource with id " + id);
        }
        resourceClosed = true;
        super.finalize();
    }
}

public void close() {
    resourceClosed = true;
    FinalizeHelper.suppressFinalize(this);
}
}

and:

public class ConditionalFinalizerTest1 {
public static void main(String[] args) throws InterruptedException {
    long time = System.currentTimeMillis();
    for (int i = 0; i < 10 * 1000 * 1000; i++) {
        ConditionalFinalizer cf = new ConditionalFinalizer(i);
        if (i % (1000 * 1000) != 0) {
            cf.close();
        }
    }
    time = System.currentTimeMillis() - time;
    System.out.println("time = " + time);
}
}

And here are the updated results:

Suppressing the finalizer for those objects significantly increases the performance of the test. The difference in performance between .net and Java is probably because I'm using reflection to scan the finalizer list in the Java case.

Summary

There is a definite benefit to suppressing finalizers for object that get manually disposed, its just a pity that there is no native "GC.SuppressFinalize" method for Java. The IDiposable pattern is very useful for situations where you want to be able to:

Manually dispose of an item when you know it is no longer needed.
Be able to run clean up code to safely bring down connections, close files, etc.
Be able to rely on the finalizer to catch situation where it is not known when an object should be cleaned up or to catch mistakes that mean the clean code is not explicitly called.

Joel on SOLID

2009-02-02T20:33:00.001+11:00

After listening to Jeff and Joel's Stack Overflow podcast today I felt like Joel had slightly missed the point of the SOLID principles. I can see how it is easy to miss the point of SOLID - an acronym of acronyms! But for me "SOLID" adds to design patterns and anti-patterns because it helps me to describe, more formally, when some code doesn't seem right - when code has a "code smell" to be a bit of a hippy about it.

Joel goes on to say that he doesn't think the principles apply well to real life situations, however, I think they apply perfectly. They are all about taking a pragmatic but disciplined approach. Take the current work I'm doing as an example: I was asked to change our product's export function and make it parallel where possible. To start with it was a single class with a few thousand lines of code. It had a "unit" test to go with it that covers about 70% of the entire code base when run and takes about five minutes to complete. To even think about how it might be made parallel I had to do some serious refactoring.

Anyway here is my interpretation of the principles and how I used them when refactoring our export code:

Single responsibility principle: Read "keep it simple stupid". We literally had an Export class that did the whole export. It exported the raw data, PDF-ed if required, TIFF-ed if required, etc, etc. I changed the code so that the export class creates a WorkQueue class that determines the items to process; processors for processing items at each step; a report writer; and much more. Since parts of the system were no longer tightly coupled I could add tests for each part which ran faster and were more maintainable. Here is a simplified before and after:
Open/closed principle: extending a module shouldn't break stuff. To be honest I don't consider this principle much during my day to day work, perhaps that is due to the type of product I'm working on.
Liskov substitution principle: have a good reason for using a base class. I feel that this strongly tied in with the SRP. For example our Exporter class inherited from BaseExporter (we have multiple types of export) and common "export" code was placed here. Sometimes a better solution is to place common code in a supporting class instead of inheriting. The acid test for inheritance should be am I ever going to use the base class independently of the subclasses? The LSP is about avoiding broken inheritance hierarchies. Uncle Bob gives the example of a set of number types which is excellent. He says inheritance is not "is a" and in the numbering example he points out that while integers are a subset of real numbers they have no commonality in terms of storage: an integer might be stored with a single 32 bit value where a real number has a mantissa and an exponent.
Interface segregation principle: again read "keep it simple stupid" but also "you aren't going to need it". This is simply a principle to avoid creating fat interfaces. The point is to define a service in terms of the clients that are going to use it. This is especially important in Java where events an delegates aren't available. For example if you have an ExportListener interface that has itemProcessedNotification and itemCompletedNotification a client will have to implement both listeners even if it doesn't care about both. Fat interfaces reduce signal to noise ratio - it obscures what you are trying to do.
Dependency inversion principle: don't tightly couple things. If everything depends on concrete implementation classes, if there are global static singletons, etc, things will be hard to modify and test. This is especially important in C# where methods are sealed by default (in Java you can mock most classes and mock out their non-final methods). In my export code I change things so that the Exporter class only depended on abstractions such as the ExportProcessor interface. This decoupled the components and made testing easier.

Java Inner Classes and SOLID

2008-12-10T14:04:00.005+11:00

I was dealing with some legacy code at work and came across some Java inner classes. Looking at those inner classes I realised that they had a bad feel to them; they break some of the SOLID principles.

The main problem lies with non-static inner classes. In the case of a static inner class why not just move the code out into a separate file? Some people argue that inner classes help to keep related things close together but it seems to go against the spirit of the SOLID principles. And other things aside, they completely break encapsulation.

Single Responsibility Principle

The single responsibility principle is about keeping things simple and manageable. However, inner classes add a highly (sometimes very highly) coupled added responsibility to a class. By adding the inner class complexity increases and testing is harder. In the case of a static inner class the two classes may be testable independently. However non-static inner classes can't be tested as an individual unit.

Dependency Inversion Principle

Again non-static inner classes break this principle: they are tightly coupled and depend on each other's implementation. As I mentioned above this makes testing the two classes in isolation impossible.

Open/Close, Liskov Substitution and Interface Segregation Principles

These principles relate more to the implementation specifics of classes. While the relationship between an inner class an its enclosing class doesn't explicitly break the principles, it must make it harder to write an outer class that complies with the principles.

Anonymous Inner Classes

In most cases anonymous inner classes fall into a slightly different category. Especially in Swing, anonymous inner classes are an important feature required to create clean UI code. However they are very much the exception to the norm. These sorts of small event handling classes are really just a poor substitute for language level events and delegates (such as those in the .net framework).

Summary

The more I consider Java inner classes the more I am inclined to avoid them. Looking at them in terms of the SOLID principles helps to outline why they can seem to complicate a class and related class design.

OSX and the Malware Economy

2008-12-07T23:02:00.005+11:00

One thing that annoys me is when people make silly statements about IT security. After the recent activity on the Apple forums regarding Mac anti-virus software, I found some people referencing this old article claiming Mac users are safe from malware. The main part of the article that stands out to me is this quote:

As Windows users continue to run their adware, spyware, and virus removal programs, Macintosh enthusiasts have enjoyed - literally - zero viruses. Adware and spyware are nonexistant as Mac users surf the Web without issue. OS X's UNIX shell is a secure system that keeps your computer safe, and Apple has built in a firewall for added security.

The first statement is certainly accurate, while there is an enormous amount of malware around for Windows systems there is very little around for Macs (although not zero). The second and third sentences are the sort of broad statements that are risky to make.

If you follow Robert Hensing's blog like I do, you'll be aware that security in OS X is not as good as some Mac users make it out to be. The links below are obviously slightly biased since Robert works for Microsoft however I feel he gives a fairly balanced point of view. Here are some examples:

The Mac is the first to fall in the pwn2own contest.
OS X vulnerabilities increasing instead of decreasing.
Vista, XP security compared to Redhat, Ubuntu and OS X.

So if all that stuff is true why is there only a handful of malware out there for OS X?

Platforms, platforms, platforms

So you want to start an cybercrime shop creating and distributing malware, what things do you need to consider?

The first thing to sort out is probably an attack vector. Typically this is a vulnerability in some software; favourites include the browser, browser plugins, network services (such as SMB on Windows) or by just tricking the end user into installing your software.

If you are using a software vulnerability such as a buffer overflow you will need to write some "shellcode" to get things started. The point of the shellcode is to provide a means to get the payload down onto the target system. It typically does this by opening a network connection to download the extra data. The tricky thing about shellcode is that it depends on the architecture of the system and probably also the operating system (e.g. if you need to open a socket to a remote host to get the payload).

Once you've executed your shellcode and downloaded your payload the possibilities open up. From here you can either write your payload to disk so you can restart once the computer or browser starts up again. Alternatively you can just hang around for the current session. Again the payload depends on the system architecture and the operating system. What the malware can do all depends on what privileges it scored when the shellcode executed. This is why IE 8 and Chrome both run with very limited privileges under Windows Vista and why Vista uses low privileges and UAC by default.

What this means is to put down a successful piece of malware you depend on a number of things: vulnerable software (or user error), operating system and architecture. And even then you may only exploit a low privilege user or process. A quick look at metasploit reveals something unsurprising: these are all solved problems for OS X - if you want to exploit an unpatched Mac, you can.

Its a numbers game

There are a few things that motivate malware writers but increasingly the trend seems to be financial motivation. Naturally malware is going to focus on the platform with the largest market share but other factors come into play which alter the concentration of malware.

Firstly, you need a reason to put malware onto a device. It sounds silly but there is literally no point developing malware for a device that won't be used to store or enter valuable information such as personal details, banking details, account details. The best example of this is the mobile non-smartphone market: Nokia sold 200 million of its Nokia 1100 model phones but its WAP 1.1 browser is not exactly the thing that most people do their Internet banking with.

The size of the PC market (and by PC I mean all PCs not just WinTel PCs) is quite large, some estimates from last year put it at the 1.3 billion mark, with Windows taking about 90% of that. As I mentioned above, a large user base is important but the Windows platform offers more that than - it offers a standard platform. For example, a piece of malware that targets Windows NT will be able to run on Windows NT4, Windows 2000, Windows XP, Windows Server 2003, Windows Vista and Windows Server 2008. It also has a fairly standard set of applications to attack: Office, Flash, Acrobat Reader, etc. Furthermore it has loads and loads of badly written third-party crapplets that all add to attack surface. Finally, you have heaps of people running Vista without UAC and Windows XP as the administrator. Microsoft can do all the threat modelling in the world and write the tightest code out there but if some Joe User clicks install when they visit a malware site all is lost. It all adds up to a malware bonanza.

While the Mac market share is growing it is still quite small in comparison (estimates currently range from 3% to 16%). OS X on desktop has only been around since 2001 and in 2006 the system architecture changed from PPC to Intel. These changes along with a less stable (and possibly less crappy) API makes it less attractive for exploit, shellcode and malware writers.

It seems to me like you need a sort of critical mass before security researchers and malware writers will start to put serious effort towards looking into a piece of software or platform. I also think that better security practices in recent times, such as workstation firewalls and automatic updates (OS X come with both by default), combined with a smaller market share make Macs a less viable malware target.

There's more than one way to skin a cat

So you've started up your cybercrime shop and decided initially to target the WinTel platform but now you're hungry for some extra cash. It turns out that there is more than one way to make money in the cybercrime economy and malware is only one of the options. Criminals are increasing using scams, phishing and spam to target end users and often this can be done in a platform independent way - Mac users are no exception.

A Mac user can visit a malicious site and be unaffected but a Windows user may be infected with malware. However if enough of a target develops the malware may evolve to target things differently.

A mobile focus

Mobile sales are increasing and the sales of mobile "smartphones" is increasing too. To me, the key difference between a mobile phone and a smartphone is that you are more likely to store personal information in the smartphone and also more likely to use it for banking and other sensitive transactions. And this is a point that has not gone unnoticed; the DC 09 Black Hat conference will be focusing on embedded devices among others.

Summary

The moral of this story (and evident by this post) is that it is easy to throw stones. OS X has certainly had its share of security issues and Windows certainly has a strong malware industry. The point I would like to make is that perhaps Apple hasn't needed to step up their software process to the next level but things can change quickly. I also think Microsoft has stepped up to the plate after their previous security woes and also that people wine about UAC too much (if it pops up lots I think you're doing something wrong ... ). Finally, cybercrime is something that is starting to affect everyone.

IKVM Benchmarking

2008-11-14T23:34:00.009+11:00

I'm interested in what options are available for migrating a high performance Java application over to the .net runtime. IKVM is one obvious pathway so I decided to run some quick benchmarks to gauge how well it performs.

Method

I ran the DaCapo benchmark suite (2006-10-MR2) on my dual core PC and also a 16 core server. I allowed the execution time of the tests to converge (-converge) to help rule out startup costs and just used the default workload size. Since IVKM will have no heap restrictions I made the Java heap as large as possible. This turned out to be 1499Mb under the 32bit JVM and I limited the 64bit JVM to 2500Mb since I didn't want it to start page faulting on my box. As it turned out this was heaps [sic] of memory.

I took the fastest run time from the tests considering it as the "least disturbed" run. The graphs below are normalized to a percentage of the fastest runtime for each test so a smaller value is better. Several tests failed to run under IKVM which was a bit unfortunate - chart, eclipse and jython.

Results

Firstly lusearch is missing from the following results. This is because I'm quite interested in this benchmark in particular and it performed so badly that it warrants further investigation...

Dual Core Machine

I really should have the exact details of my machine but I don't have them handy. It is basically a dual core AMD 64 with 4Gb of RAM.

As you can see the results are quite mixed. Pmd and Xalan perform quite poorly, however looking at the threading information and the memory stats nothing really jumps out. Xalan is one of the multithreaded tests however so is hsqldb in which IKVM fairs quite well.

16 Core Machine

Again I don't have the exact details of this box but it is a quad-quad-core with 32Gb of RAM. The extra cores seem to exaggerate the dual core results further. Interestingly, on hsqldb (one of the multithreaded tests) IKVM 32 bit outperforms the Java 64 bit VM and the 32 bit client. The other strange result is on pmd in which the 32 bit Java client is the fastest to execute - I suspect this is a copy paste issue :-(

Summary

Some very mixed results and some questionable test results :) I guess some more work is needed.

Luke Quinane

Windows App Store

The iPhone App Store

Windows App Store

Secure Updates?

Java

Paint.net

Skype

Other

Summary

Roomba Review

Pros

Cons

Summary

Epic Meltdown

August Links

Utility

VM Ware

Misc

iPhone Reaching Critial Mass

Microsoft

TWAIN Dot Net

Exchange Wedding Anniversary?

June Links - ASP.net Deployment & Misc

ASP.net

HTML Sanitisation

NSsh Progress

Export Processing

Smart Pointers != Managed Automatic Memory Management

Manual (Traditional) Memory Management

Automatic Dynamic Memory Management

Smart Pointers

First Class Reference Counting

Summary

Net Censorship in Australia

Transparency

Leave it to the Experts

Further Reading

Interesting Links

Java Exceptions

SNR

Jail Break

Service, s'il vous plaît?

Summary

Java Suppress Finalizer

.net?

Java

Summary

Joel on SOLID

Java Inner Classes and SOLID

Single Responsibility Principle

Dependency Inversion Principle

Open/Close, Liskov Substitution and Interface Segregation Principles

Anonymous Inner Classes

Summary

OSX and the Malware Economy

Platforms, platforms, platforms

Its a numbers game

There's more than one way to skin a cat

A mobile focus

Summary

IKVM Benchmarking

Method

Results

Dual Core Machine

16 Core Machine

Summary