I’m in the middle of a project developing a timesheet/rostering application, which uses a lot of Javascript. I have a series of web services that interact with jQuery. For reasons beyond the scope of this post, the web service returns a JSON string, which needs to be parsed into a JSON object, and then further processing is done on the object.

I had a bug where Safari was choking on the JSON string returned by my webservice, yet Firefox and IE were happy parsing it. As it turns out, my JSON string had a Javascript keyword in it; ‘break’.

Here is my JSON string (well, an example):

{employee: 'Joe', worked: true, periods: [{start: '09:00 AM', finish: '17:00 PM'}], break: {paid: true, hours: '1'}}

Safari was choking on the break in the above JSON string.

When I wrapped the attribute in single quotes, like so:

{employee: 'Joe', worked: true, periods: [{start: '09:00 AM', finish: '17:00 PM'}], 'break': {paid: true, hours: '1'}}

it was happy.

You can replicate this by using the below code snippet to parse out your JSON string quickly in Safari’s debug window:

var json = "{employee: 'Joe', worked: true, periods: [{start: '09:00 AM', finish: '17:00 PM'}], break: {paid: true, hours: '1'}}";
var obj = (new Function("return " + json))();

which should fail. If you add the single quotes around break though, it should work just fine.

This also fails for other Javascript keywords:

(new Function("return " + "{if: true}"))();

fails. If you wrap if in single quotes, it works.

I think the take away lesson is to always wrap your object attributes in a JSON string in single quotes, otherwise Safari will choke on it.

I got the following exception today:

If a data source does not return ICollection and cannot return the total row count, it cannot be used by the GridView to implement server-side paging.

If you’re trying to return an IEnumerable from your select method, try changing it to a List or an array. That should clear the problem right up.

As much as this site is currently a programming blog, it’s also my electronic representation. The website shares my name, and is representative of me. There’s more to me than just programming. For a while I’ve tried blogging or writing about non-programming related topics on another domain, however I don’t feel this is fair to myself.

I guess the best way to describe myself is an aspiring polymath. I have an interest in a broad range of topics, notably programming/computer science, mathematics and art. This website doesn’t really represent that. I can’t maintain multiple websites to separate various areas of my life – I’m a single person with multiple facets, and I should have a single website, with multiple facets.

Additionally, I find myself with less and less time to devote to this kind of writing, as it slowly makes it’s way down to the bottom of my priority list. Currently I’m trying to rekindle my math education by going back to basics, learn Punjabi so I can impress the fiance’s parents, do several programming projects at once and get back into doing art regularly, and writing is a very far last.

So, all that lead up to say that I’m going dark, and will be overhauling the site sometime in the future. I’m going to change the format of the site so that it more accurately represents me as a whole, is better organised and has a nice format that allows me to post quick, smaller posts that can be filtered out when subscribing to RSS feed. I may or may not stick with WordPress – I think I’ll probably start hacking away at Enki and do some funky stuff with wildcard subdomains or something.

All older posts will remain, as a few of my posts are rather popular (by more than 50% compared to the rest of the site) and I suspect they’re really helping people out.

I’ve recently stumbled on an interesting AI competition (before it got big with the video – I’ve been writing this post for about a month), based off of an open source Java project called Inifine Mario, called Mario AI Competition 2009. The competition is to create an AI agent that can navigate randomly generated Mario levels, pitting each agent against each other. The first round of this competition has already closed, however there is another round that closes September 3.

I won’t be attending the conferences nor will I be submitting agents to the competition. I will however work on a few agents, as this is probably the most fun application of AI that I’ve seen in a while, and it’s a small enough environment to enable me to get very familiar with it very quickly, and hence get into making solutions very quickly.

On the surface, this seems a trivial, niche problem. An agent that can navigate a video game level doesn’t really have a lot of diverse applications. However, it’s a great way to begin practicing some fundamental concepts of AI. The parameters of the competition don’t limit the techniques you can use to create the agent. In the video below, Robin Baumgarten is using A* path finding techniques.

However, A* is just the start, and is a fairly basic approach (though it works pretty well by the demos), and there are many more options, including machine learning and neural networks. Unfortunately, I won’t be covering any of that yet. The goal of this post is to get myself, and anyone else interested, up to speed on the environment, and the basics of creating an agent. I will be creating a very simple reactive agent that has no planning, and is quite flawed. However, you should be able to walk away with a firm grasp of how the environment works and what tools you have at your disposal.

Before I go on, a disclaimer is necessary: I haven’t worked with Java since my university days, so about 6 years or so. I will post some code, and that code will probably suck. You have been warned.

The Game Environment

The details on the code available from the competition website are fairly vague, so I’m posting this to perhaps help some people who are interested in getting into it, and also for my reference to help while working on it. Firstly, obviously, you’re going to have to download the code available on the competition website. The code is relatively stable at this point, however it was constantly being updated with fixes for bugs posted on the Google Group for the competition. These changes would be posted on the group, so I’d check it regularly, or grab it from Google Code with svn, and just update your repo regularly. Once you’ve downloaded it, open it in your IDE/editor of choice, and roll up your sleeves.

Firstly, I’m going to reference classes by their fully qualified class names; ie: ch.idsia.scenarios.Play, which references Play.java located in the src/ch/idsia/scenarios folder. This way you can play along at home.

According to the online documentation, you can fire up a run by typing

java ch.idsia.scenarios.Play

from within the “classes” directory. This will start a copy of the Infinite Mario game with a human player. To start with a pre written agent, type

java ch.idsia.scenarios.Play ch.idsia.ai.agents.ai.ForwardJumpingAgent

to start with an agent that will run forward and jump as soon as it can.

The ch.idsia.scenarios.Play class is a simple class that sets up the agent, whether it’s human controlled or automated, and then it sets some play environment variables and runs the application. If an argument is supplied, it will attempt to instantiate an instance of the class and pass it through to the play environment.

I won’t be covering the the details of the simulation code, as it’s all pretty logically separated and you don’t really need to know how to do that. Information is passed through to the agent through an implementation of the interface ch.idsia.mario.environments.Environment. This argument to the getAction function provides the agent with a serialised view of the terrain and the location of enemies, along with information about Mario. Basically, any information you can get by looking at the scene.

This information is provided in the form of a byte matrix, through several functions. observation.getLevelSceneObservation() will return a matrix with details of the terrain. It will also tell you what the type of the terrain is, such as a pipe or a one way platform. This detail is found in ch.idsia.mario.engine.LevelScene. You can see the number codes for the different types of terrain. It’s broken down into gaps, hard borders, half borders and flower pots.

The observation.getEnemiesObservation() is very similar to observation.getLevelSceneObservation(), except it addresses enemy locations. Again, you’ll also get the type of the enemy from the matrix in the form of an integer code, which you can find in ch.idsia.mario.engine.sprites.Sprite.

I’ve written a debugging function that will output both the enemy and terrain matricies to give you a code view of what’s going on. It’s interesting and enlightening to watch the code view of the simulation next to the simulation itself, and it will help us write code to fit our design. The function is as follows:

byte[][] levelState = observation.getLevelSceneObservation();
byte[][] enemyState = observation.getEnemiesObservation();

//Debug information
for (int y = 0; y < 22; y++)
{
    for (int x = 0; x < 22; x++)
    {
        //Check for enemies first
        if (enemyState[y][x] > 0)
        {
            String enemyValue = Integer.toString(enemyState[y][x]);
            int padding = 3 - enemyValue.length();
            if (enemyState[y][x] == 1)
                System.out.print("mmm");
            else if (padding == 2)
                System.out.print(" " + enemyValue + " ");
            else if (padding == 1)
                System.out.print(" " + enemyValue);
            else
                System.out.print(enemyValue);
        }
        else
        {
            //Check for terrain next
            String terrainValue = Integer.toString(levelState[y][x]);
            int padding = 3 - terrainValue.length();
            if (padding == 2)
                System.out.print(" " + terrainValue + " ");
            else if (padding == 1)
                System.out.print(" " + terrainValue);
            else
                System.out.print(terrainValue);
        }
        System.out.print("|");
    }
    System.out.print("\n");
}
System.out.print("---------------\n");

I put the above code into a function in the agent itself, and called the function from the getAction function of the agent, purely because I know this is going to get called once per tick. You can put it wherever you want to put it.

Creating a Dumb Agent

As I mentioned at the start of this post, I’m just going to start with creating a very simple, dumb agent that will attempt to traverse the level, and even then not get it right all the time for reasons I will cover. On the surface, Mario’s tasks for navigating a stage without enemies is pretty straightforward. All he needs to do is to get himself around the level and jump over pits.

However, sometimes Mario can find himself falling into a gap, or over-jumping his mark and jumping into a hole. So basically, we need to check if there’s something in Mario’s path that needs to be jumped over, or if there’s a gap that we need to jump over. But, we can’t just jump – we also need to check if jumping is going to put is into a gap and kill Mario, and if Mario runs off of where ever he is, he might fall into a hole.

This would be fairly straight forward if we had a robust method of predicting where Mario’s jumps will take him, but unfortunately we don’t officially get this information.
There is a movement algorithm in ch.idsia.mario.engine.sprites.Mario, which starts at the move method, however, not only is this not in the spirit of the competition, but more importantly, it’s very hard to figure out, for me at least. I’ve not been able to reliably modify the movement code to act as a predictor of Mario’s path if you plug-in how long to hold each key for.

Hence, at this stage, we’ll just be making Mario go right and try and jump over obstacles. Seems fairly simple.

You’ll see in the provided agents that the function getAction is the heart of the agent, and an instance of the Environment interface discussed before. This function is the work horse of your agent, as it will tell the simulation what the agent is going to do.

Although the observation.getLevelSceneObservation() and observation.getEnemiesObservation() methods return the full view of the level, all the features are jumbled together. Later on, we’ll need to know what’s a cannon and what’s a pipe, and what’s raised ground, and maybe even what’s a block we can punch. We’ll need to know this because cannons shoot Bullet Bills and enemy flowers live in pipes, and these can appear at any time, and if Mario is near by, he’s going to get hurt.

I’ve created the following function to categorize the environment into pipes, cannons and pits:

private Boolean[] pits;
private Vector cannons;
private Vector pipes;
private void processTerrain(byte[][] state)
{
    //Init features
    pits = new boolean[22];
    cannons = new Vector();
    pipes = new Vector();

    //Init pits
    for (int i = 0; i < 22; i++)
    {
        pits[i] = true;
    }

    for (int y = 21; y >= 0; y--)
    {
        for (int x = 0; x < 22; x++)
        {
            //Look for gaps
            pits[x] = pits[x] && state[y][x] == 0;

            //Look for cannons and pipes
            if (state[y][x] == 20)
            {
                boolean pipeFound = false;
                //Check if x,y is in pipes
                int pipeLength = pipes.size();
                for (int pi = 0; pi < pipeLength; pi++)
                {
                    int[] coords = (int[])pipes.elementAt(pi);
                    if (coords != null && coords[0] == x && coords[1] == y)
                    {
                        //Pipe found, skip it
                        pipeFound = true;
                    }
                }
                if (!pipeFound)
                {
                    //Peek ahead to determine if it's a pipe or a cannon
                    if(x < 21 && state[y][x + 1] == 20)
                    {
                        //It's a pipe, add it and the next one to the pipe array
                        pipes.add(new int[]{x + 1, y});
                    }
                    else
                    {
                        //It's a cannon
                        cannons.add(new int[]{x, y});
                    }
                }
            }
        }
    }
}

The three variables are defined as private members of the agent, and calling this function will populate them with tile locations in the case of pipes and cannons, and tile columns in the case of pits. You don’t have to do this with such a simple agent – you can simply use similar techniques to parse over the whole level state.

I use this information to determine how far away obstacles are. At first I was going to make the distance between obstacles affect the jumping distance and height, however without an accurate way to determine where jumps are going to land, this can actually cause more harm than good. So I ignore it, and just use a fairly blunt approach.

Firstly, another helper function:

private int setDeltaForObstacles(Vector tiles)
{
    int length = tiles.size();
    int buffer = 0;
    int localDeltaX = 0;
    //Loop through every tile
    for (int counter = 0; counter < length; counter++)
    {
        //Get coords of tile
        int[] coords = (int[])tiles.elementAt(counter);
        //Calculate the delta, then check to see if it's good enough
        localDeltaX = coords[0] - 11;
        if (action[Mario.KEY_RIGHT] && coords[0] > 11)
        {
            //If the tile is after mario, and we're heading right, delta is good
            break;
        }
        else if (action[Mario.KEY_LEFT] && coords[0] < 11)
        {
            //If the tile is before mario and we're heading left, add to buffer
            if (coords[0] > buffer)
                buffer = coords[0];
        }
        else if (action[Mario.KEY_LEFT])
        {
            //If the tile is after mario, but we're heading left, get the last tile loc from the buffer
            localDeltaX = buffer - 11;
            break;
        }
    }
    return localDeltaX;
}

This function takes a Vector and will return the closes tile to Mario, depending on his direction. It’s a very coarse way to determine the closes obstacle to Mario. Bear in mind that it doesn’t recognise when obstacles are put close together, thus doesn’t optimise Mario’s jumping.

The goal of the agent is to jump over the closest obstacle. That’s it. Considering our obstacles are now in three different datastructures (cannons in one, pipes in another, and pits and raised terrain in the original byte array), we need to store the delta (distance between Mario and the closes obstacle) and update it only when we’ve found a smaller delta. Also, I want to keep the delta of any pits separate, as I’m going to have different rules for pits than other obstacles (because pits are the only thing that can kill Mario at this stage).

To this end, I create two integer variables at the top of the getAction method. They are intitialized to 11 because this is where Mario is:

int deltaX = 11;
int deltaJump = 11;

Here is the code to extract the delta of the nearest obstacle, and the delta of the nearest pit. Note that levelState is the results from a call to observation.getLevelSceneObservation:

//Analyse pipes
int pipeDelta = setDeltaForObstacles(pipes);
if (pipeDelta < deltaX && pipeDelta > 0)
    deltaX = pipeDelta;
//Analyse cannons
int cannonDelta = setDeltaForObstacles(cannons);
if (cannonDelta < deltaX && cannonDelta > 0)
    deltaX = cannonDelta;

//Analyse the gaps
boolean gap = false;
for(int y = 0; y < 22; y++)
{
    if (pits[y])
    {
        if (action[Mario.KEY_RIGHT] && y > 11)
        {
            int localDelta = y - 11;
            if (localDelta < deltaJump)
                deltaJump = localDelta;
        }
        else if (action[Mario.KEY_LEFT] && y < 11)
        {
            int localDelta = 11 - y;
            if (localDelta < deltaJump)
                deltaJump = localDelta;
        }
        gap = true;
    }
}

//Analyse ground above Mario
boolean raisedGround = false;
if (action[Mario.KEY_RIGHT])
{
    for (int x = 12; x < 22; x++)
    {
        for (int y = 12; y >= 0; y--)
        {
            if (levelState[y][x] == -10)
            {
                int localDelta = x - 11;
                if (localDelta < deltaX)
                    deltaX = localDelta;
                raisedGround = true;
                break;
            }
        }
        if (raisedGround)
            break;
    }
}

Once we have the deltas, we can use this information to decide what to do. I mentioned above treating obstacle deltas different from jump deltas. This is an attempt to make Mario take smaller jumps when he can see a pit is after an obstacle. This is controlled by two instance members that keep track of jump height and jump length, called jumpHeight and jumpLength respectively. This is very hit and miss. Notice that Mario does not respond to anything more than 4 tiles away, and tiles that are behind him. This is to prevent him from going jump crazy:

jumpHeight = 7;
jumpLength = 14;// * (deltaX / 11);
if (((pipes.size() > 0 || cannons.size() > 0 || raisedGround) && deltaX >= 0 && deltaX < 5))
{
    if (observation.mayMarioJump() && !action[Mario.KEY_JUMP])
        action[Mario.KEY_JUMP] = true;
}

//If mario is jumping, there must be an obstacle in the way
if (action[Mario.KEY_JUMP])
{
    if (gap && deltaJump >= 3)
    {
        jumpHeight = 7;
        jumpLength = 1;
    }
}
//No obstacles, but there is a gap
else if (gap && deltaJump >= 0 && deltaJump <= 3)
{
    if (observation.mayMarioJump() && !action[Mario.KEY_JUMP])
        action[Mario.KEY_JUMP] = true;
}
maintainJump();

Finally, the maintainJump method is to ensure we’re holding down the jump key for the optimal amount of time. It is also coded to provide rudimentary control over jump height and length, however this isn’t being taken advantage of.

private void maintainJump()
{
    if(action[Mario.KEY_JUMP] && (jumpCount > jumpHeight))
    {
        action[Mario.KEY_JUMP] = false;
        jumpCount = 0;
    }
    // otherwise you're in the middle of jump, increment jumpCount
    else if (action[Mario.KEY_JUMP])
    {
        jumpCount++;
    }
    if (jumping)
    {
        if (action[Mario.KEY_RIGHT] && (rightCount > jumpLength))
        {
            action[Mario.KEY_RIGHT] = false;
            rightCount = 0;
        }
        else if (action[Mario.KEY_RIGHT])
        }
    }
}

The complete agent is at the bottom of this post for you to download and include in your project. Simply put it in the path src/timgittos, configure the ch.idsia.scenarios.Play class to have no enemies, with options.setPauseWorld(true), and run any level you want (passable result with difficulty 3). Then build the project and run it with java ch.idsia.scenarios.Play timgittos.BlogAgent.

As you will observe, the agent will make Mario go towards the right as fast as possible, jumping over obstacles as it meets them. For the most part, it works well enough. However, there are quite a few fatal flaws, and I will explain why it fails in this way.

Most obviously, the agent frequently falls down gaps. It doesn’t fall down all gaps – which would indicate that it’s not a failure in the gap detection, but rather something else. In fact, what’s going on is that the gaps that Mario tends to fall down are those that enter his field of consideration (4 tiles ahead of him) while he’s in the air. The agent makes no effort to check whether it’s jump will land Mario in a pit (since it can’t predict where Mario will land), and my previous efforts at trying to make the agent force Mario to the left when it detects he’s going to go down a pit have failed.

Mario will also fall down pits if there are multiple pits in a row. Again, this is because the agent doesn’t exert much control over the size and length of Mario’s jumps ,and so can’t pin point that it wants to land in the middle of the two pits, so it tends to overshoot the jump and fall down the second pit.

Most annoyingly, the agent will sometimes get stuck. If it reaches an obstacle, such as a cannon or a bit of wall or a pipe and jumps at the wrong angle, it will enter a weird wall jump loop, where it jumps, gets stuck on the obstacle, jumps off, jumps again, and gets stuck again. This will loop until there is no time. This is a side effect of the environment allowing wall jumps, which is cool when you can get the agent to make Mario jump out of a pit he’s falling into, but annoying when he gets stuck.

In any case, this agent is far from complete, and just serves to demonstrate how the simulation environment works, and how to interact with it. Once you’re clear how to work with the simulation, you can concentrate on building a useful agent, either by starting from scratch (like you would if you were going to use a machine learning approach), or by building on top of this base (for a determinate approach like A*).

Download: Mario AI Blog agent (.java)

When working with JQuery and AJAX enabled WCF services, I recently encountered the following error, through Firebug and Fiddler:

The maximum string content length quota (8192) has been exceeded while reading XML data.
This quota may be increased by changing the
MaxStringContentLength property on the XmlDictionaryReaderQuotas
object used when creating the XML reader.

This is caused by WCF webHttpBinding default having fairly draconion limitations on XML message length and depth. This thread on MSDN helped diagnose the problem, however it didn’t do much to help me solve my particular problem. Ultimately, I stumbled upon the solution quite accidentally.

Firstly, as per the MSDN thread, you need to add the custom binding configuration underneath the configuration section:

<system.serviceModel>
  <bindings>
    <webHttpBinding>
      <binding name="bindingConfiguration">
        <readerQuotas maxDepth="32" maxStringContentLength="2048000" maxArrayLength="16384" maxBytesPerRead="4096" maxNameTableCharCount="16384" />
      </binding>
    </webHttpBinding>
  </bindings>
  ..
</system.serviceModel>

At first I thought this was a custom binding, and tried to replace the service endpoint binding to the custom binding defined above, however this was causing an “System.ServiceModel.ServiceActivationException” exception.

The above binding declaration is actually a configuration for the webHttpBinding. As such, you need to add the BindingConfiguration property onto your service endpoint, along side your binding declaration. This is as follows:

<service behaviorConfiguration="Your.Service.Behavior"
      name="Your.Service.Name">
    <endpoint address="" behaviorConfiguration="Your.Behavior.Configuration"
     binding="webHttpBinding" bindingConfiguration="bindingConfiguration" contract="Your.Service.Contract" />
</service>

This will allow you to send larger messages to your WCF services via AJAX, and get on with creating applications.

Due to the recommendations in the comments of my Ruby CMS article, I have decided to start playing with the Camping Ruby microframework backed by CouchDB. Getting Camping up and running was easy enough, and I managed to get some trivial functionality going. Camping uses SQLite for it’s database, which is fine if I wanted to massage my data into a relational form. However, I have models with optional attributes, and rather than making nullable database types, I want to leverage CouchDB’s schema-less, RESTful document storage. Each persisted object in my application will be available on a unique url, so CouchDB is a nice fit for what I want to do.

The Great Installation

I found this great article about getting Camping talking to CouchDB, and it seemed easy enough, so I decided to try it. Since I already had Camping working, all I needed was to get CouchDB installed on my Macbook. As much as I’m not scared of compiling applications from source, I’ve had issues in the past with OS X and compiling, so I decide to find a simple package or disk image to install. I found CouchDBX, however checking the requirements showed that it supports Leopard only.

With a little more research, I found out I had to use Macports to install CouchDB. Macports relies on XCode Tools which come available on the DVDs that came with OS X. If you haven’t previously installed XCode Tools, then you should install them before you install Macports. Something that is buried in a support ticket, however, is that Macports targets XCode tools 2.5. My XCode tools was something like 2.4.8. While I did manage to get Macports installed, the installation for CouchDB fails when it tries to build tk. If you, like me, have an older XCode Tools than 2.5, you’ll need to download 2.5, which you can find deep inside the Apple website. This does require an Apple Developer Connection account to get, so if you don’t have one, you’ll need to create one.

Installing CouchDB is as simple as issuing

sudo port install couchdb

and Macports will take care of the rest. CouchDB lists it’s dependencies as Spidermonkey, Erlang and a few others, however each of those has dependencies, and each of those dependencies has dependencies, and it ends up taking quite a long time and downloading a fair few packages. The upside is that if you didn’t have Erlang previously installed, you do now. Erlang is something I’ve been wanting to take a look at.

Start The Server Before You Leap

While the article I linked to above about getting Camping and CouchDB talking is good, it’s not exactly the most verbose explanation especially if you haven’t read the documentation for CouchDB and prefer to just dive in. For those who are still a little unsure like I was about how exactly to do this, I’ll outline my approach below.

Before you get started writing any code at all, you need to start your CouchDB server. This is something that escaped me for the longest time, and when I finally twigged, it gave me a bit of grief. To start CouchDB, issue the following command:

sudo couchdb

If you try this without escalating privilege, it will return an error “{“init terminating in do_boot”,{{badmatch,{error,shutdown}},[{couch_server_sup,start_server,1},{erl_eval,do_apply,5},{erl_eval,exprs,5}, {init,start_it,1},{init,start_em,1}]}}”, which searching for will bring you to the CouchDB wiki page for error messages. This page will tell you the problem is an unavailable port. For me, at least, this is not true. CouchDB runs on port 5984 which as far as I know is not used for any system services or other software. The reason I was getting this error was that I wasn’t running it with enough privilege.

Once you get CouchDB started, you’ll notice it blocks the terminal. I don’t like to have too many windows open at once, so I’d prefer to have it run as a background process. Fortunately CouchDB will let you do that with a command line switch option, along with changing the location of the pid file. I’ll leave figuring out how you want to start it up to you.

Once you have CouchDB started, you’ll have access to a web-based administration panel called Futon. Futon is available to you at http://localhost:5984/_utils/, assuming you’re running CouchDB on your local host. The Futon utility will allow you to create databases and insert documents in preparation to test your connectivity with Camping.

Camping Time

There were two Ruby gems I had considered when deciding how to connect Ruby to CouchDB. I should point out that due to the RESTful nature of CouchDB, you strictly don’t need any Ruby gems and could roll your own fairly easily, if you wanted to. I didn’t want that level of control, personally. The two gems I considered were CouchRest and RelaxDB. CouchRest is a simple Ruby wrapper around the CouchDB REST API, whereas RelaxDB is more abstracted, bringing ActiveRecord-like functionality to CouchDB. While before I would have chosen RelaxDB, I’m intentionally trying to work a little closer to the raw APIs here, so I chose CouchRest.

You can install CouchRest through RubyGems:

sudo gem install couchrest

however I found that the gem version wouldn’t work and kept throwing errors when I started my Camping application. To overcome this, I just cloned it from the GitHun repository and placed it in my Camping app directory and referenced it that way.

The first thing to do would be to create a normal Camping sample application, if you haven’t done already. For the purposes of this post, I’m going to use the “skeletal Camping blog” application used in the documentation, and edit that to work with CouchDB. I figure this will give everyone a relatively common base from which to start. Import CouchRest just below where the application imports Camping:

require 'camping'
require 'couchrest' # or 'couchrest/couchrest' or similar if you've cloned from Github

Given that the purpose of this article is to demonstrate connecting to CouchDB, and not the design of a framework around it, we can safely ignore defining a model for now, and rely on CouchDB as our model. So if you’re following along from the example application, you can safely delete the classes from Blog::Models module. We’re not going to delete the whole module, because of the create method.

The Camping create method that is run when the server starts up your Camping app. From the documentation, “This is a good place to check for database tables and create those tables to save users of your application from needing to manually set them up.” Instead, we’re going to use this method to set up CouchRest in our application:

module Blog::Models
  def Blog.create
    db_url = 'http://localhost:5984/'
    storage = CouchRest.database("#{db_url}blog")
    Blog::Controllers::Index.set_storage(storage)
  end
end

Next, because we’re now calling a new method on the controller class, we need to modify that. Change the Blog::Controllers::Index class to the following:

module Blog::Controllers
  class Index < R '/(\w+)'
    def Index.set_storage(storage)
      @@storage = storage
    end
    def get(id)
      @post = @@storage.get('posts/' + id)
      render :index
    end
  end
end

What I’ve done here is given the controller class a static reference to my CouchDB database (from the create method). Then the controller was changed to respond to a regex, which is passed into the get method. We use the id passed in to retrieve a document from CouchDB, which we assign as an instance variable.

Lastly, we need to modify the Index function view to pull data from the CouchDB database:

def index
  h1 @post[:title]
end

This will output the title of the test document created at the start of the article.

If you haven’t already, start up Futon and create a database for the purposes of this demo. Call the database “blog”, and fill it with a test document with the id of “posts/test”, and at least a “title” property with the value “Test Post”, and any other properties you wish.

Start the Camping application with camping blog.rb, and point a browser to http://localhost:3301/test, and you should see “Test Post” in a header tag, rendered out to the browser, which means that Camping has successfully communicated with CouchDB.

I didn’t cover putting data into CouchDB, however you would do it in a similar vein, writing a post method inside your routes to create data and save it to CouchDB. From here, personally, I’m going to create some wrapper classes for requests and models and build up something which is a little DRYer, however as far as a demo, this is good enough.

Firstly, I tried to ask Wolfram Alpha how it worked, and it just gave me a puzzled look. So I can only guess, and guess I shall.

The Wolfram Alpha website lays it all out on the table, with regards to what it’s all about. Wolfram Alpha’s goal “is to accept completely free-form input, and to serve as a knowledge engine” to answer search queries entered. From the very limited playing I’ve done, it seems pretty cool.

Following my move from Perth to Texas, I typed in “Austin to Perth” into Wolfram Alpha just to see what it told me. I got back map, with a line drawn between the cities. I got the distance in miles between the two and the flight time. I got a side by side comparison of the local times, populations and approximate elevations.

Google however, delivered me what I was actually looking for, sort of. Maybe that’s just a difference in expectations and thinking.¹

It’s clear that Wolfram Alpha is pretty cool, and will do some awesome things with the data before it returns it back to you, but it doesn’t really help me figure out how it works. My thoughts are that Wolfram Alpha combines a large database of indexed knowledge in the form of a knowledge base and a powerful logical language, the language being Stephen Wolfram’s own Mathematica, naturally. As he describes it, Wolfram Alpha is the “killer-app” for Mathematica.

First, lets discuss the vast store of data available to Wolfram Alpha, most likely in the form of a knowledge base. A knowledge base is an information representation scheme designed to allow information to be operated on using logic. Information is stored in “sentences” in a knowledge representation language. The knowledge base allows logical agents, in this case the Wolfram Alpha application, to receive queries, and then search and analyse the knowledge base, performing logical inferences and information compounding, then collating it and returning the results. The logical agent does all the heavy lifting, but the very structure of the knowledge base helps the agent.

The heart of the logical agent is it’s knowledge representation language. It allows the agent to “reason” through sentences and make inferences in order to derive new facts. Proposition logic and First-order logic are both knowledge representation languages. Knowledge representation languages use syntax and semantics to define knowledge, where syntax outlines the structure of a language, and the semantics are the meaning.

Logic is a very mathematical concept, and mathematics can be seen as a kind of logic that operates on numbers. Mathematics has a syntax, that defines the infix notation of logical operators on operands, and the semantics are what those operators do. 1 + 1 = 2 is an example of a complex logical language, where the syntax defines that the + operator acts on each 1, and the = operator defines the result of the previous operation. The semantics define that the value of 1 + 1 is 2.

Mathematica, the heart of Wolfram Alpha, is a computational language, according to it’s Wikipedia page. It’s built on top of the Symbolic Manipulation Program, designed by both Chris A. Cole and Stephen Wolfram. The Symbolic Manipulation Program is a “computer algebra system”, which facilitates symbolic computation, allowing computers to operate on symbols and manipulate expressions instead of operating on their values, which sounds a heck of a lot like something that could be used to create a logic that can operate on a knowledge base.

Going back to logical agents and knowledge bases, there are a number of ways logical agents can extract and infer information from knowledge bases. The topic of knowledge based logical agents is a pretty vast one, and I couldn’t possibly explain it completely in a single post, even if I knew more than just a fraction of it, which I don’t. But I can summarize what I do understand, and how it applies to Wolfram Alpha.

As referred to often in the post so far, inference or entailment plays a large part of how a logical agent can extract “new” information from a knowledge base. Inference is deeply rooted in formal logic, and techniques have been developed to allow automated software processes to apply it to a knowledge base.

Logical entailment is the idea that a sentence “logically follows” another sentence. If a given sentence a is true, and the truth of sentence b depends on the truth of a, then b must be true, and also that if b is true, then logically a must also be true. This is useful if the knowledge base doesn’t contain the fact that one of a or b is true, but it does contain definition that the truth of b depends on the truth of a.

The knowledge base is designed to be a model of reality, and holds sentences about the world intended to describe it to a level of detail. Sentences can be as atomic as facts, such as “the sky is blue”, or can be logical sentences such as “if the sun is in the sky, it is not night”, which of course has exceptions (such as in certain seasons in the polar regions), which also need to be modeled in the knowledge base.

This is relevant because, with a finite model represented by the knowledge base, there are a finite number of entailment and inferences that can be made on the model, which means that a logical agent such as Wolfram Alpha can reduce the world to a finite search space, and can run a search over this space looking for information that follows on from the set of facts in the knowledge base.

Search techniques for searching the knowledge base are similar as search techniques in other AI based problems, such as constraint satisfaction problems and touring problems. These searching algorithms include backtracking and local search methods, such as hill climbing.

When searching this knowledge base, the logical agent needs to abide by a series of rules, known as inference rules, in order to guarantee the the knowledge it entails and infers is valid.

Firstly, the logical agent needs to be aware of the equivalence of two logical sentences. If two sentences are both true under the same model, such as the Wolfram Alpha knowledge base, then they are logically equivalent, and can be used to format facts in a different way to aid inference.

Secondly, the logical agent needs to be aware of the concept of validity. A sentence is valid if it is true in every model, which essentially means it’s a tautology and is always true. These tautologies are useful because they aid in validating inferences, where a infers b if a implies b is a tautology, where implication is such that b is true if and only if a is true. This can get a little confusing, and if you can, I’d suggest you read Chapter 7 in AI:AMA
.

Thirdly, it needs to ensure the sentence is satisfiable, which means it’s true in at least one model, which is basically just searching for inferences and determining that, for a given search space state, the inference being suggested is true.

The logical agent and inference/entailment from a knowledge base can be observed in Wolfram Alpha with a fairly simple query: “red + yellow“. In this query, the logical agent would probably hit the knowledge base and tell it that both red and yellow are true, meaning that we want to find information about those. It would then search for all sentences that are entailed or logical structures that mention the fact red or yellow, such as a sentence “orange is a mix of red and yellow”. It would then tell the knowledge base that this fact had been entailed, and tell it that orange is true. Then, when it finds the sentence “blue is complementary to orange”, it can add that to the record set. Obviously there will be other sentences in the knowledge base, and the agent will have to use a filtering algorithm to determine which sentences and facts are relevant to the query, otherwise it might return results that the user is not interested in.

To successfully parse the search query, it would also need a measure of natural language processing, so that it can convert a human-readable query into a machine readable query. This, combined with a large, extensive knowledge base, a fast logical language such as Mathematica and AI techniques for inference and entailment of knowledge mean that Wolfram Alpha can answer a wide range of questions that a simple indexed and categorised search engine couldn’t.

However that’s just my best guess. And keep in mind who I am: I’m just a young web developer with an interest in AI. I don’t actively participate in AI research, and I have no formal training. My guesses and musings are probably widely off the mark, and if anyone has any other ideas or corrections, clarifications or discussions, I’d love to hear them in the comments.

1. If you mistype a search query into Wolfram Alpha, curiously it won’t suggest a correction. Considering how relatively easy this task is (just use it’s Levenshtein Distance), I find it odd that it doesn’t support this.

]]>

I’ve been spending the last week settling down here in Austin, and the week before that preparing for the trip. I’m still suffering from some jet lag, and catching up on the last 8 months with my girlfriend, who I hadn’t seen since August 2008. I haven’t had the time nor the concentration required to write a blog post, and I haven’t touched my Google Reader account, which I can see has over 1,600 unread articles.

However, I do have some interesting posts coming up. I’m planning a response to an AI article that was linked to me in a comment, a more in depth look at the importance of searching by solving the 8-Queens problems in Python using some different search techniques, and a look at creating some graceful degradation control adapters for ASP.NET server controls that don’t work without Javascript.

So, don’t unsubscribe from my feed in your readers, the blog isn’t abandoned, life has just taken priority over it. When things settle down, I’ll start writing articles again.

I’ve been thinking a lot lately about solving problems. I read a number of programming aggregators and startup blogs, and the term “solve a problem” keeps nagging at me. To be successful, at programming and business, you need to solve a problem. I’d love to eventually run a development company, but I don’t know what problem to solve. So, I consider the problem my current employer is solving.

I currently work for a web development company that is doing fairly well. We have a suite of products that we sell to our clients, and sell services related to those products. They’re doing well, so obviously, they must be solving a problem. So, from “client” point of view, the problem we solve is that of letting people who don’t know anything about websites build and run complex websites.

We have a CMS product that allows non web geeks manage websites. This includes writing new pages and editing content on existing pages, publishing news items, distributing these to an email list, managing uploaded files and inserting modules of dynamic content to their hearts desire. The modules we offer include contact forms, random image rotators, resume upload controls, auto-generating bread crumbs and menus, and so on. These modules aim to reduce the complexity of managing dynamic sites, so that ordinary people can do it.

On top of that, we do custom applications built on top of that CMS site. Clients have a site they run themselves with the CMS, but some also have dedicated applications with a separate interface. These are separate because they’re custom jobs. They provide functionality that is not available in the CMS (yet), or functionality that would be too complex to maintain using the CMS. So again, we’re reducing the complexity of running a custom application.

So, on first looks, we build websites and applications that non-geeks can use. But looking further, we manage complexity. We reduce the complexity for the client, so that the client can do difficult things with ease, and don’t have to know how it’s done, only that they can do it.

On the other hand, good programming is about managing complexity. That is, trying to keep your code as free from complexity as you can get away with while still doing the job. Complex code is buggy and unmaintainable.

Complexity exists, and it is immutable. Once you’ve simplified a task to the point that the only steps involved are those that accomplish that task, and that task cannot be accomplished without those steps, you have your complexity, and it isn’t going anywhere. The task cannot be made more simple, because to do that we’d have to leave out steps, which would not accomplish the task.

What we do from then on is delegate complexity, shift it around and make it other people’s problem. Take for example, the task of creating and publishing a blog post. Our software has tagging, categories, post titles and post content. Creating a post involves writing a post, giving it a title, adding tags, selecting categories and publishing it.

Breaking it down, we need to create a body, create a title, create all categories, create all tags, link the tags to the post, link the post to the categories and publish it. Each link has a display name and a URL slug, and each category has a display name and a URL slug. URL slugs cannot contain certain characters, however their display names can.

Now, we can delegate this complexity to the user, which will result in simple code. The user will be responsible for creating tags and categories, and giving them URL slugs that fit our rules, while the application will just take the data and store it, and retrieve it when asked. The user will be responsible for assigning tags to the post and the categories, and the application just stores it. This is obviously an unacceptable level of complexity for the user.

We could also delegate all the complexity to the application, making the user experience simple. The user just types in the tags they want, picks the categories, and the application will link the tags if they exist, create them if they don’t, automatically strip unwanted characters from URL slugs and hide a lot of the complexity. In more complicated cases, this will result in very complex code that will be bug prone.

So the question is, where do we draw the line? What balance of code complexity and user interface complexity is successful? I guess you could say “make it 50/50″ and think that’s good enough, but I don’t agree. I think the balance is very contextual, and often leans in favour of code complexity. In simple examples like above, taking on all the complexity of the task in the code is a trivial thing and easily done. For nuclear reactor control software, things might not be straight forward.

This is the key to a successful software product/business, I think. Finding the right balance between complexity. If you put too much complexity on the user, user’s won’t be able to use it, and your software won’t sell, and the whole thing will bomb. If you put too much complexity on the software, it will be buggy and difficult to maintain, and will eat up development time and money, and possibly even end up costing more than it earns, and then it will bomb. That’s not even including the more personal aspects of programmer job satisfaction and sanity.

Success is finding the sweet spot when balancing complexity.

The problem is that the jQuery Context menu overzealously unbinds all click events from the document. This means that it’s going to break all your other Javascript that relies on clicking. Checking the documentation for jQuery’s unbind function, you can see that it accepts a function as a second parameter. This is the function that will be unbound if you pass that in as a parameter.
So, I fixed the jQuery Context Menu by changing the following:

1. In the main contextMenu function, near the end, just above the return statement, I added the following:

   // External click event for document
   function onDocumentClick(e) {
         var menu = $('#' + o.menu);
         $(document).unbind('click', onDocumentClick).unbind('keypress');
         $(menu).fadeOut(o.outSpeed);
         return false;
   }
   

2. Next, I change the function that assigns the click listener to the document to assign the defined function, instead of an anonymous function. So, change:

   // Hide bindings
   setTimeout( function() { // Delay for Mozilla
         $(document).click( function() {
               $(document).unbind('click').unbind('keypress');
               $(menu).fadeOut(o.outSpeed);
               return false;
         });
   }, 0);
   

   to

   // Hide bindings
   setTimeout(function() { // Delay for Mozilla
         $(document).click(onDocumentClick);
   }, 0);
   

3. Lastly, I remove all instances of:

   $(document).unbind('click');
   

That’s it. That fixed the problem for me. You’ll notice I don’t bother with the keypress bindings. That’s because I’m not using them, so I’m not bothered. I’m assuming the fix for that will be a similar strategy, for those who are bothered.

I’ve uploaded my version for those who can’t get it working using the tips in this post, but remember the copyright remains with Cory S.N. LaViska over at A Beautiful Site.
jquery.contextMenu.js