JRuby

We'll install Ruby along side of JRuby, but we will stick to Ubunutu's currently supported 1.8 as this is compatiple with JRuby. This way we can develop an application that is deployable on JRuby and Ruby giving a bit of flexibility.

sudo apt-get install jruby ruby-full rubygems

Then we will install Rails 3. To do this we will first need to update RubyGems.

sudo jruby -S gem update --system
sudo jruby -S gem install rails

JRuby and Rails

The hardest part about setting up a Rails app for use with JRuby is the database gems are different. Thankfully the JRuby guys have made this real easy by providing a template.

jruby -S rails new myapp -m http://jruby.org/rails3.rb -d mysql

Inspecting myapp/Gemfile shows that there are two platforms blocks with different gems for ruby and jruby. I recommend changing the jruby block to:

gem 'activerecord-jdbc-adapter'
gem 'activerecord-jdbcmysql-adapter'
gem 'jdbc-mysql'
gem 'jruby-openssl'
gem 'jruby-rack'

So now we can go ahead and install everything needed by running from the root of myapp:

sudo jruby -S bundle install

Now, with all the imagination that it entails, we'll create the standard Post example and set up the databases.

First update conf/database.yml to reflect your settings. You will also need to update all instances of adatapter: mysql to:

adapter: <%= defined?(JRUBY_VERSION) ? 'jdbcmysql' : 'mysql' %>

This puts a conditional in so that if your app is deployed on JRuby it will use the jdbc drivers. I had some major issues using jruby -S rake to do migrations, so this allowed me to use the regular rake to do my migrations.

Then we will create a basic Post scaffold and set up the databases for our Posts.

jruby -S rails generate scaffold Post name:string title:string content:text
jruby -S rake db:create:all
rake db:migrate
rake db:migrate RAILS_ENV="production"

We can then test the development environment by running

jruby -S rails server

At http://localhost:3000 you should have something similar to:

Note that the Ruby version 1.8.7 (java) which indicates that it is running on JRuby, and that the Database Adapter is jdbcmysql. You can make sure that the database is working by playing around at http://localhost:3000/posts

Deploying

With JRuby we can package up the Rails app as a war file so that it can be deployed to all the regular containers in the same manner as a 'real' Java application. Warbler is a simple gem that will do all the magic and hard work. From the root of of your Ruby application

sudo jruby -S warble

This will create a file myapp.war in the root of myapp. Note that warbler will set you config to production automatically, which is why we did the produciton migration earlier. Assuming you want to deploy to a standard Tomcat 6 on Ubuntu you can run:

sudo cp myapp.war /var/lib/tomcat6/webapps/
sudo chmod 777 myapp.war
sudo /etc/init.d/tomcat6 restart

It seems that the default home page used for the developement server is not reproduced in the production configuration, so you may get a routing error at http://localhost:8080/myapp but you should now have a working scaffold at http://localhost:8080/myapp/posts. You can verify that warbler automatically used your production settings by manually checking the myapp_production database after entering a few Posts in your applicaiton.

Resources

Most of this was adapted from Deploy a Rails 3, Sqlite3 application in Tomcat using JRuby by Greg Morneo.

We'll progress through these in increasing difficulty. Starting with an infinite loop, through to your standard for loop.

The Grammar

First of all we will define the grammar for all out types of loops.

//Infinite loop
LOOP        { L = getLbl;
              emitLbl L }
<block>
END     { Emit(jmp L }

//Post coditional
DO          { L = getLbl
              emitLbl L }
<block>
UNTIL
<condition> { Emit(JNE L) }

//For Loop
FOR <ident> = <expr1> TO <expr2> <block> END

These translate into the follow data type

    data Statement = Statement String 
                  | Branch Condition Block
                  | Branch2 Condition Block Block
                  | While Condition Block
                  | Loop Block
                  | DoUntil Block Condition
                  | For Statement Expression Block
                  deriving (Show)

Infinite Loops

An infinite loop by itself is pretty useless, but combined with a break statement it can be pretty handy for when you don't really know how long you need to be going through your loop. A typical example of an infinite loop is a game loop or an event loop, that essentially loops while true until a particular state, event or message terminates the program.

The paser is very simple. We just drop the keywords and pass the block onto the Loop data constructor.

    -- |Parses loop..end statments
    loop :: Parser Statement
    loop = accept "loop" <-+> block <+-> accept "end" >>> Loop

The assembly to create an infinite loop is just an unconditional jump back to the start of the loop. This means we will require a label at the start of the block and a simple jmp after our block.

    emitStatement (Loop b) = do
        startLbl <- getLbl
        block <- emitBlock' b
        let jmp = emitLn ("jmp " ++ startLbl)
        return ((emitLbl startLbl) ++ block ++ jmp)

Post Conditonal Loop

We already have a preconditional loop with while, so now we will add it's cousin the do...until. We use the keyword until to avoid some issues with the while keyword. This loop is useful where you would like something to happen at least once.

On the face of it, this is just while with the condition and block reversed. That's exactly how we will treat it. The recogniser is the same as while with minor order adjustments, as is the emitter.

    -- |Parses do..until statments
    dountil :: Parser Statement
    dountil = accept "do" <-+> block <+-> accept "until" <+> condition +>> DoUntil

    emitStatement (DoUntil b cond) = do
        startLbl <- getLbl
        c <- emitCondition cond
        block <- emitBlock' b
        let jmp = emitLn ("je " ++ startLbl)
        return ((emitLbl startLbl) ++ block ++ c ++ jmp)

For loop

The for loop construct we are going to build here is not the c-style one with a condition and increment function. We are going to use a simpler Basic style construct that is typically used like:

FOR x=1 TO 10
    <block>

There are a few assumptions with this construct. It is assumed that the 'counter' is an integer that incements by one on each pass through the loop.

You may have noticed in the first section that I (intentionally) did not add any psuedo-syntax to the for loop syntax. If we were to try to directly translate a for loop into assembly it would be a bit tiresome and quite ugly. Instead, we will translate the source into an alternative construct.

// alternative for loop construct
<ident> = <expr1>
TEMP = <expr2>
WHILE <ident> <= TEMP
<block>
<ident> = <ident> + 1
END

So we will parse the construct, but rather than emit the assembly directly, we will actually just 're-write' the AST to this new construct, which we can then just emit with our existing code.

To do this we will need to remediate our expression parser and emitter so that they work in the larger context of our program parser.

We'll bring the Assign type into be a data constructor for Statement, add the assignment option to statement add a new pattern to emitStatment, and (for now) ignore the sections and just emit the text section.

    -- Lbach.Grammar.Basics
    data Statement = Statement String 
                  | Branch Condition Block
                  | Branch2 Condition Block Block
                  | While Condition Block
                  | Loop Block
                  | DoUntil Block Condition
                  | Assign Assignment
                  deriving (Show)

    data Assignment = Assignment String Expression 
                    deriving (Show)

    -- Lbach.Parser.Control
    statement = loop 
                <|> dountil
                <|> while 
                <|> ifelse 
                <|> ifthen 
                <|> assign
                <|> other

    -- Lbach.Parser.Expressions
    assign = token letters <+-> token (literal '=') <+> expr +>> Assign

    --Lbach.Emitter.Control
    emitStatement (Assign s expr) = do
        return $ emitText expr

You'll also need to remove emit2 from Lbach.Emitter and update some patterns in Lbach.Emitter.Expressions to get it compile.

Now that's out of the way, we can get down to business on the for loop parser. Let's revisit the grammar.

FOR = TO END or FOR TO END

This leads to a very natual data type and parser.

    -- |Parse a for loop
    forloop :: Parser Statement     
    forloop = accept "for" <-+> assign <+-> accept "to" <+> expr <+> block <+-> accept "end" >>> br
        where br ((a, e), b) = For a e b

As you can see I chose not to transform the AST at the parser level. Therefore we have to do it at the emitter level. This is surprisingly quite straight forward.

    emitStatement (For (Assign (Assignment s e1)) e2 b) = do
        let var1 = s
        let var2 = "temp"
        let increment = Assign (Assignment var1 (Add (Var var1) (Num 1)))
        line1 <- emitStatement $ Assign (Assignment s e1)
        line2 <- emitStatement $ Assign (Assignment var2 e2)
        rest <- emitStatement (While (Condition (var1 ++ "<=" ++ var2)) (b ++ [increment]))
        return $ line1 ++ line2 ++ rest

Testing some samples

*Main> p "loop block end end"
L0:
        <block> block
        jmp L0
        ret

*Main> p "do block until condition end end"
L0:
        <block> block
        <condition> condition
        je L0
        ret

*Main> p "for a=1 to 2 block end end"
        MOV eax, 1
        MOV [a], eax
        MOV eax, 2
        MOV [temp], eax
L0:
        <condition> a<=temp
        je L1
        <block> block
        jmp L0
L1:
        ret
*Main> p "for a=1 to 1000 if a b end d = 3+2*c end end"
        MOV eax, 1
        MOV [a], eax
        MOV eax, 1000
        MOV [temp], eax
L0:
        <condition> a<=temp
        je L1
        <condition> a
        jne L2
        <block> b
L2:
        MOV eax, 3
        PUSH eax
        MOV eax, 2
        PUSH eax
        MOV eax, [c]
        POP ebx
        MUL ebx
        POP ebx
        ADD eax, ebx
        MOV [d], eax
        MOV eax, [a]
        PUSH eax
        MOV eax, 1
        POP ebx
        ADD eax, ebx
        MOV [a], eax
        jmp L0
L1:
        ret

*Main>

It is a misnomer that functional programming cannot have state, rather it cannot have side effects, which is a very different ideal. If you need to maintain state in your application, and a compiler inherently does, then you need to continually pass this state from one function to the next. Conversely, all your functions that modify the state must all return the entire state.

The State Monad was created for instances exactly like this. It defines an explicit interface for passing state from one function to the next, and provides several functions for executing the functions using this state. I'm not going to provide yet another State Monad breakdown - for that I can recommend the following: the official wiki, a great explaination of the mechanics and most recently Learn you a Haskell.

Some things I learned the hard way

The State Monad and Record Syntax

I really struggled with using the State Monad in a situation where multiple peices of state are used. As the compiler is going to need many peices of state such as the label counter, break label, symbol table etc., I started looking for info on record syntax and State Monads. The best I could find was a few paragraphs in Real World Haskell which was pretty clear if brief.

Types for Functions that use the State Monad

Almost all the sample code and tutorials on using the state monad involved no parameters for the functions. All the types where of State s a and almost no examples showed how to pass in extra parameters. Thankfully Learn you a Haskell had some examples that used b -> State s a which lead me to figure out that functions using the State monad can have any type but final type of a function must be State s a and that the type of the function used with runState et.al. must be State s a only. Took me a while to figure this one out.

Random number examples

Annoyingly, 9 times out of 10, any example on the State Monad uses the random number example. This example is next to useless as it neatly ties up the original state in StdGen which hides the actual usage of State. Please try to come up with something original and worthwhile that explains the whole thing. Thanks to Learn you a Haskell, which uses a simple stack example, I was able to figure how and when the State is called.

Simple Example

Rather than try to retro fit the State Monad throughout our emitter in this article, which will be complex and tedious, I will use a simplified example.

First thing we do is create a data type to collect all out stateful information, and a new type using State and our new data type.

-- This will hold all of the data in our state
data EmitStuff = EmitStuff {
    lblCounter :: Int,
    lastLabel :: String 
    } deriving (Show)

-- It is common practice to make your own state type
type MyStateMonad = State EmitStuff 

lblCounter will hold the current count for creating numbered labels, whilst lastLabel will be the last generated label.

Next we will create a function that generates a new label. This will solely use the state to calculate the new label, and then update the state. Do notation makes this much clearer to read.

-- Generates a new label based on the current label counter
-- Stores the generated label, and updates the label counter
newLabel :: MyStateMonad String
newLabel = do 
    st <- get
    let l = "L" ++ show(lblCounter st)
    put st { lblCounter = lblCounter st + 1, lastLabel = l}
    return l

Next we'll create an example that takes an extra parameter. A function that generates a label with a comment.

-- Emits a label followed by a comment 
emitLabelAndComment :: String -> MyStateMonad String
emitLabelAndComment x = do 
    l <- newLabel
    return (l ++ ":  #" ++ x)

Let's take a moment to understand what this one is doing. You'll recall that the State Monad is a wrapper for the type s -> (a, s) which means that the above type signature expands into String -> s -> (String, s). This means that this function will also take the state as input, which is then passed onto newLabel through the State Monads internals. This is a key concept in turning your manual state management code into something that can be used with the state monad. Mike Vanier explains how we can turn the 5 different type of functions using state into one standard format: s -> (String, s).

The next example is calling multiple functions that use state from one function.

-- Simple example showing how we no longer need to manually thread the state when creating labels 
fakeIfStatement :: MyStateMonad String   
fakeIfStatement = do 
    falseBranch <- newLabel
    endLabel <- newLabel
    return ("if not true then \njump " ++ falseBranch ++ "\nDo some true stuff\njump " ++ endLabel ++ "\n" ++ falseBranch ++ ": #some stuff to do if false\n" ++ endLabel) 

Executing the State Monad

Finally we need to tie it all together. The state monad comes with three utility functions for executing the code.

1. runState returns the result and the state as a tuple
2. evalState returns the result and discards the state i.e. (result, _) = runState
3. execState returns the state only and discards the result i.e. (_, state) = runState

Each of these functions take a fucntion of type State s a and an intial state of type s. So first we will create our initial state.

initialState = EmitStuff { lblCounter = 0, lastLabel = "" }

We can then use any of the utility functions to execute our code against this state.

runState fakeIfStatement initialState 
-- or
evalState newLabel initialState

But this is not very useful as the state is reset everytime we apply it to a new function. We have already seen the answer in fakeIfStatement. We simply create one function that executes all the other functions for the duration of the state.

-- Tying it all together 
emitAll = evalState emit EmitStuff { lblCounter = 0, lastLabel = "" } 
    where emit = do
              a <- emitLabelAndComment "This is the first label"
              b <- fakeIfStatement 
              c <- emitLabelAndComment "This is the last label"
              return (a ++ "\n" ++  b ++ "\n" ++ c)

Now if you were to execute putStrLn & emitAll in ghci you will see labels nicely placed and incremented as expected.

You can download and play with all of the above from Github.

Implementing in LBaCH

Implementation into the existing Emitter.Control required a rather large rewrite (94% according to git), but was not terribly difficult. It now reads much clearer and looks a lot more like the BNF notation.

For now I have made the state entry point at the emitBlock level, but in future it may be moved up to encompass the entire emit process.

emitBlock :: Block -> String
emitBlock b = evalState e EmitterData { lblCounter = 0, lastLabel = "" }
    where e = do
              a <- emitBlock' b
              return a 

You can find all the gory details here.

I've been playing around with the Python starter package to get a feel for it. AI, whilst a field I am interested in, I have absolutely no experience with. I am fumbling and googleing my way through the first few attempts.

I intend on switching to Haskell once I get my feet under me, as there is a 1 second time limit for each turn, and as my strategies get more complex, the Python version is going to reach that limit pretty quick. Jasper Van der Jeugt has kindly put together a Haskell Starter Kit, so the ground work is laid down.

Whilst I won't reveal any of my strategies just yet (for fear of embarrassment), I will share my testing script.

The script runs your bot against all the example bots, and any other bots you put in the examples directory, on all maps and gives you a score. The score is simply the percentage of wins. Test runs are recorded in a history file, along with the git version so you can assess whether you are going forwards or backwards.

You can also specify how many maps to use if you want a shorter feedback loop.

import os, subprocess, sys

total_wins = 0
opponents = 0
maps = 100
mybot = "python MyBot.py"

history = "history.txt"
this_run = []

if len(sys.argv) > 1:
    maps = int(sys.argv[1])

import subprocess as sub
p = sub.Popen('git rev-parse HEAD',stdout=sub.PIPE,stderr=sub.PIPE)
output, errors = p.communicate()
print output
this_run.append("Bot version: " + output.strip())

for file in os.listdir("example_bots"):
    opponent_cmd = ""
    if file.endswith(".jar"):
        opponent_cmd = 'java -jar example_bots/' + file
    if file.endswith(".py"):
        opponent_cmd = 'python example_bots/' + file
    if opponent_cmd:
        opponents = opponents + 1
        player_1_counter=0
        player_2_counter=0
        for i in range(1, maps):
            cmd = ['java', '-jar', 'tools/PlayGame.jar', 'maps/map' + str(i) + '.txt', '1000', '1000', 'log.txt', '"' + opponent_cmd + '"', '"' + mybot + '"', '>tmp 2>&1']
            cmd2 = " ".join(cmd)
            os.system(cmd2)
            r = open('tmp').read()
            if "Player 1 Wins!" in r:
                player_1_counter = player_1_counter + 1
            if "Player 2 Wins!" in r:
                player_2_counter = player_2_counter + 1
                total_wins = total_wins + 1
        print file + " :: " + str(player_1_counter) + "/" + str(player_2_counter)
        this_run.append(file + " :: " + str(player_1_counter) + "/" + str(player_2_counter))
print "Done. Won " + str(total_wins) + " of " + str(opponents * maps)
print "Score of " + str((float(total_wins) / (opponents * maps)))
this_run.append("Done. Won " + str(total_wins) + " of " + str(opponents * maps))
this_run.append("Score of " + str((float(total_wins) / (opponents * maps))))

open(history, 'a').write("\n".join(this_run) + "\n\n")

Good luck.

The facilities were exemplary. The exhibitors were seperated from the presentation halls so that you could avoid the vendors between talks. The catering was plentiful. And I spent a fair amount of time with the Lego in the chill out area. Check out the Flickr Stream to see what it was all about.

But the purpose of the event was to learn something. Which I did. I learn't a lot actually. Like ASP.NET MVC2 has some killer features, I heard about autofac for the first time, and got very interested in CCR.

I'm not going to review each of the talks I attended, because that would be boring and ardious. Instead I will just share my highlight - Joel Pobar.

Joel's talk (or rant) on Philosophy of Software Quality with David Connors was quite critical about current standards and processes, yet I found it to be incrediable insightful. Their third pillar, Craftsmanship, is something that I have instinctually known to be critical for a good developer to become great, yet I was unable to label. It now makes perfect sense.

I also attended Joel's other talk, High performance, highly scalable applications on the .NET Framework. This was a little over my head, but Joel's enthusiasm and delivery style kept me interested and definately sparked an 'I should know more about this' light bulb.

If you get the chance to attend any of Joel's presentations, grab it with both hands. Even if you have no idea what the subject matter is, you will not be dissappointed.

1. Number found by running document.getElementsByClassName('topic').length against http://australia.msteched.com/topic/list

Why?

During a farm upgrade I needed a way to bring across all the profiles from the old SSP to the new SSP. Some of the data in the profiles was original content and was not in any of the profile source systems e.g. Skills etc. As I didn't want the users to have to re-enter all their information, this tool was born.

But why write something?

During my research I could not find anything that got the job done. I found ways to import/export the properties and a project to import profile data based on an XML file mapping, but didn't seem to fulfill the export option it promised. There was nothing that would let me do a simple export then import. So now there is.

Usage

To export all the profiles use {{spprofiledump.exe --url http://example.sharepoint.com --file dump.xml --export}}. This will create an XML file with all the profiles contained in the SSP. To import data into an SSP use {{spprofiledump.exe --url http://example.sharepoint.com --file dump.xml --import}}.

The XML File

The file is reasonably straight forward. A sample entry is probably the simplest way to explain it:

<ProfileDump>
  <User AccountName="DOMAIN\USER">
    <Property Name="Title" Type="string">Mr</Property>
    <Property Name="FirstName" Type="string">Geoff</Property>
    <Property Name="LastName" Type="string">Ford</Property>
    <Property Name="SPS-HireDate" Type="date">10/06/2003 12:00:00 AM</Property>
    <Property Name="SPS-Skills" Type="List">
        <Entry>SharePoint Development</Entry>
    <Property>
    ...
  </User>
<ProfileDump>

As you can see, dates and multi-valued properties are handled correctly.

Get It

You can get the source, and a pre-compiled binary at the SPProfileDump homepage. Please put all feedback, questions, issues etc. over there.

I tried all the usual stsadm commands. First I tried just an upgradesolution, then I tried forcing a redolyment as well. I even went as far as to delete the solution and re-add, but all to no avail. Everything I tried was met with a furiating 'meh' from the servers.

SharePoint Storage Model

SharePoint has an interesting concept of how, where and when to store certain files. Coming from a regular web developer background, I assumed that all the files on disk is what SharePoint would use for rendering the pages. Turns out I was wrong.

After a lot of hunting, I stumbled on this article from Mike Knowles which explains it quite clearly

...once Master Pages and Page Layouts have been customized then SharePoint does not allow solutions to overwrite them. Depending who you talk to or what article you read, or how late you end up working when vexed by this problem, you might consider this a feature or a bug.

In the SharePoint development world, you can't depend on updating files on the file system to affect any changes at all, because the 'files' might actually be database records.

My Solution

As all good SharePoint developers should, I was using a Feature to deploy my layouts. The plan was to use a FeatureReceiver to simulate the check-out, modification and check-in of the files in question. It works suprising well.

The code is quite straight forward. The only gotcha was to ensure that the file was able to be checked out and not customised. Because the files that you need to update are dependant on the Feature, I have not included my implementation of GetLayouts(). I used the Features.xml via properties.Definition.GetXmlDefinition() to get a list of all the ElementFile nodes, but you might prefer to traverse the file system.

public override void FeatureActivated(SPFeatureReceiverProperties properties)
{
    var files = GetLayouts();

    SPFile targetFile = null;
    FileStream fileStream;
    string targetFileName;
    try
    {
        using (SPWeb web = Extensions.GetWeb(properties))
        {
            foreach (string file in files)
            {
                targetFileName = file.Replace("\\", "/");
                targetFile = web.GetFile(targetFileName);
                if (targetFile.Exists == true)
                {
                    if (targetFile.CheckOutStatus != SPFile.SPCheckOutStatus.None)
                    {
                        targetFile.UndoCheckOut();
                    }
                    if (targetFile.CustomizedPageStatus == SPCustomizedPageStatus.Customized)
                    {
                        targetFile.RevertContentStream();
                    }

                    targetFile.CheckOut();
                    using (fileStream = File.OpenRead(properties.Definition.RootDirectory + "\\" + file))
                    {
                        targetFile = web.Files.Add(targetFileName, fileStream, true);
                        fileStream.Close();
                        targetFile.CheckIn("Feature Activation", SPCheckinType.MajorCheckIn);
                        Trace.WriteLine(targetFileName + " successfully updated");
                    }
                } else {
                    Trace.WriteLine(targetFileName + " does not exist");
                }
            }
        }
    }
    catch (Exception exception)
    {
        Trace.WriteLine(exception.Message);
    }
}

This could be written as an extension on SPFeatureReceiverProperties and you caould add some diffing so you don't create unneccessary versions of the files, but this simple implementation works sufficently.

Now when you update the layout files, repackage and redeploy the solution, you will need to do a activatefeature -force to ensure that your layouts are updated.

This review is of using Drupal 6 in a commercial environment for a medium to large website with over 600 pages, and a traffic profile of approx. 10,000 hits per hour with almost 90% bounce rate off the home page.

I will cover each of the stages of development: implementing the template, installing and configuring modules, importing the content, integrating other services and deployment.

Templating

The template system in Drupal is a powerful and flexible system that is an odd combination of template files and template functions. Some modules prefer functions over files and others vice versa, whilst some things are easier to do using functions but files are more easily maintained.

This duality can lead to a bit of confusion. It can be a bit of a hassle to figure which is best, what can be done where etc. The Theme Developer module was indispensable during this phase. We mostly used files with very rare exceptions, leaving template.php to almost exclusive house helper functions.

I personally liked the seperation of the template files to allow you to override only certain things, such as targetting the way a particular field is displayed. That is, until I couldn’t do what I wanted.

For example, I was trying to override an individual block template, whilst retaining the default block template. Apparently, to override a single block template you must override the default template. This took me a little while to figure out. I can understand why this is so, but I don’t think it was a good design decision.

The other thing I will recommend is to use a design partner that understands the complexity and separate components of Drupal. If your designer provides a design that you can’t quite mould into Drupal’s way of doing things, you will have to resort to horrible things like calling theme_blocks() within node.tpl.php.

Modules

As with any popular CMS, Drupal has an extensible architecture which they term modules. There are literally thousands of modules available on Drupal.org. However, as with any popular CMS, these are of varying degrees of completeness and robustness.

There is a bewildering array of modules available, some that seem to do almost nothing, to others that are so common that they have actually been flagged for inclusion in the core of Drupal 7 suc as CCK. Finding the module that does what you want can be a little bit of a game of find and seek. A good research environment, one that can be set up and torn down fast, is mandatory when trialing new modules.

Developing modules in Drupal is a pretty pleasant experience. While the core code is using an old ‘hooks’ paradigm, it works very well. Combined with the extensive code comments, excellent API site and one of the best developer handbooks I have seen in a CMS, you can quickly and easily find and understand the function you need.

Migrating the Old site

The site we re-implemented in Drupal has been around for several years and was a large, sprawling collection of HTML mixed with assets and code.

So the first order of the day was to rationalize and understand the existing content. This was probably the hardest part of the entire project, and was completely under estimated. Be warned, moving a site in a large, complex organization has a far reaching impact. We found out after our first release that a number of other teams, and even external sites, were depending on assets served on the site.

There are a couple of modules available for importing content into Drupal, but the method we used was Migrate and Table Wizard. We had to write a crawler that would dissect each page in the existing site, stripping styles, downloading assets and attempting to fix links, and dumped all of this into a database of content to import.

The crawler saved us a huge amount of time, but every page still required manual review to ensure that no inline styles slipped through and that it actually looked and functioned correctly.

Integrating with other things

The hardest part of the entire build was working with external data. As this site has been around for several years, a number of dynamic pages had been constructed using data from various sources using various techniques.

Drupal likes to share its information out. You can provide RSS and ATOM feeds and even provide web services for others to consume. But when it comes to Drupal using external data, you are limited.

There are a couple of modules that make integrating with specific services easy such as Flickr and Amazon. There are even a few modules that attempt to make it easy to use generic data such as Feeds and Data, but I found these a bit cumbersome and limited. I am keen to see them progress in the future.

So when it came to connecting to the web services that we had written to expose all the legacy applications to the new Drupal site we got a bit stuck. We ended up writing a very thin module that simply called a web service and passed the data onto a template for rendering. With Drupal’s ability to override template files, we were on our way.

Deployment

We went with established best practice for enterprise Drupal and separated out authoring environment from the production environment. We then used deployment scripts to automatically and painlessly deploy from one environment to the other, with only a change to settings.php.

This worked well with the glaring exception of the SWF Tools module. For some reason, whenever we deployed a page using SWF Tools filter, the flash was still referencing the previous environment. With a little digging, we were able to find a workaround, but the original issue remains.

Caching and Performance

Drupal is a complex machine and requires a fair bit of horsepower to keep it going. There will be untold misery if you neglect to enable at least the default page caching in production. Believe me – we've been there.

Caching in general is difficult to get right. It is a balance between fresh data and fast page serves. Unfortunately, Drupal’s built in page cache is based on a minimum lifetime. This means that the pages are stored in the cache forever until a page edit causes the cache to invalidate.

This might be fine for mostly static pages, but we had a few pages that we wanted to use (almost) live data on without the need for someone to edit the page. This fell afoul of the standard cache, because the page was pulled from the cache without updating the dynamic parts.

I don’t know how, or if it’s even possible, to specify that a portion of the page is not to be cached. I think we could have used blocks, but because this data was to be placed anywhere in the content with the use a filter, blocks were out of the question. For now we have simply excluded the important pages from being cached, but we can’t do this for too many pages or we start seeing 100% CPU utilization. And that’s not fun. We are investigating the use of Boost, which is based on a maximum cache time and looks to be a huge performance increase as well.

Conclusion

Drupal is not just a CMS, but as a CMS it functions exceptionally well. There are enough modules out there to do almost everything you need, yet if you do need to develop your own module you won’t be short on tutorials, samples and documentation.

The user interface can be a bit awkward, but by limiting the options that editors could use and providing a bit of training, they are more than comfortable doing their own edits.

My only real issue with Drupal is that the performance needs some serious work and caching could be a little more flexible and easier to work with dynamic pages.

Overall, it was a pleasant experience, especially when juxtaposed with other CMS implementations I have had the (unfortunate) pleasure of being involved in. I still think there is a lot for me to learn, especially around the capability of other modules such as Panels and Rules etc, as well as just understanding the 'Drupal way'.

With Drupal 7 very active and nearing release (they are up to Alpha 5 at the time of writing), things are looking very good for the Drupal eco-system.

The Program

First of all we will create some 'holding' functions. Let's define these first.

<program>::=<block> END
<block>::=[<statement>]*

What this says is that a program is a block followed by the end keyword, and that a block is any number of statemtents.

As usual this requires a type, a parser and an emitter.

Parsing a Program

The types are very simple.

    data Program = Program Block deriving (Show)
    type Block = [Statement]
    data Statement = Statement String deriving (Show)

Because multi-character keywords are pretty simple using our techniques, we will skip the single character keyword versions. I will be using a function called accept which is a simple parser that reads the letters and then compares it to the given keyword. I've also created a letters' that fails on an empty string. I'll leave the definition of these as an exercise.

    program :: Parser Program
    program = block <+-> accept "end" >>> Program 

    block :: Parser Block
    block = iter statement

    statement :: Parser Statement
    statement =  other

    -- | Accepts any string except 'end'
    other :: Parser Block
    other = (token letters') <=> (/="end") >>> Statement

With these parsers you can write really cool programs like something end or two statements end. Don't worry, it gets more exciting. You might also want to update your parse function to use program instead of assign.

Emitting a Program

Our emitter must now take a Program, and iterate over the Statements in it's Block. For now we will be outputting a place holder <block> for our other statements.

    type State = Int

    emit :: Program -> String
    emit (Program b) = emitBlock b ++ emitLn "ret"

    emitBlock :: Block -> String
    emitBlock b = result
        where (s, result) = emitBlock' 0 b

    emitBlock' :: State -> [Statement] -> (State, String)
    emitBlock' s [] = (s, "")
    emitBlock' s (b:bs) = (s', first ++ rest)
        where (s1, first) = emitStatement s b
              (s', rest) = emitBlock' s1 bs

    emitStatement s (Statement b) = (s, emitLn (" " ++ b))

In case you are wondering why all s and State, this is how we will track the label counter, and other stateful items like the symbol table in the future. I originally had emitBlock using fold' and map in one line, but it caused complications when it came time to thread the label counter through the emit functions.

So now we can parse and emit multi-statement programs. Those statements might not do anything yet, but we'll get there.

If-Then Statements

The first control structure we will look at is the basic if statement. This takes a condition, and if the condition is true, it will execute the body of the if statment.

The hardest part of this section is deciding what our syntax will be. To keep things simple, we will stick to Crenshaws recommended syntax of

IF <condition> <block> ENDIF

By having an explicit block terminator (ENDIF) we avoid the ambiguity of 'is this still part of the if body'.

Parsing If-Then

We create a new type constructor, which I called Branch, which will hold the condition and the body of the if statement, which is really just another block. The parser is very similar to the syntax definition:

    data Statement = Statement String 
                   | Branch Condition Block

    ifthen :: Parser Statement      
    ifthen = accept "if" <-+> condition <+> block <+-> accept "end" >>> buildBranch
        where buildBranch (c, b) = Branch c b

Because we use block as part of the function, we can parse multi-statement bodies, and even nested if statments such as:

*Lbach.Parser> ifthen "if cond partA partB end"
Just (Branch (Condition "cond") [Statement "partA",Statement "partB"],"")
*Lbach.Parser> ifthen "if condA if condB bodyB end bodyA end"
Just (Branch (Condition "condA") [Branch (Condition "condB") [Statement "bodyB"],Statement "bodyA"],"")

Gotta love the power of recursion.

Now to be able to use our new ifthen as part of a program we need to update statement to include it, using our good friend the alternate combinator <|>:

    statement :: Parser Statement
    statement =  ifthen <|> other

Emitting If-Then

An if statement needs to be translated into the following assembly:

<condition> ;calculate the condition
jne end     ;goto 'end' if the condition is NOT equal
<block>     
end:

We need to use a jne, which means jump if not equal, because we want to skip the body if the condition fails. We will continue to output the placeholder <block>. We will also use a placeholder for <condition> as we haven't covered relational statements yet.

When generating asm we can't call all our labels end so we have to create a function to generate a new label and we'll also create one to emit the label. For labels we will use L1, L2 etc, and to keep things nice we need to return the next label count.

    getLbl :: Int -> (String, Int)
    getLbl count = ("L" ++ (show count), count + 1)

    emitLbl :: String -> String
    emitLbl lbl = lbl ++ ":\n"

Unfortunately, the need to thread the label counter through the emitters complicates them a little, but they are reasonably simple. We have to be careful to pass the State around in the right order so that our labels come out correct.

    emitStatement :: State -> Statement -> (State, String)
    emitStatement s (Branch cond b1) = (s', c ++ jmp ++ block1 ++ end)
        where (s1, c)       = emitCondition s cond 
              jmp           = emitLn ("jne " ++ lbl)
              (s', block1)  = emitBlock' s2 b1
              end           = emitLbl lbl
              (lbl, s2)     = getLbl s1
    emitStatement s (Statement b) = (s, emitLn (" " ++ b))

    emitCondition s (Condition c) = (s, emitLn (" " ++ c))

The only tricky part is following the label counter through. We first send the label counter through the condition in case it needs to use labels. The counter is then used to get a new label for use in the jump and outputting the label to jump to. It then gets passed onto the body block which may use labels and therefore increment the counter. This final number is then returned to the caller. To see why we pass it in this order try parsing and emitting a nested if.

*Main> let k = emit . parse
*Main> putStr $ k "if condA if condB bodyB end bodyA end end"
        <condition> condA
        jne L0
        <condition> condB
        jne L1
        <block> bodyB
L1:
        <block> bodyA
L0:
        ret
*Main>

If-Else

Now that we have the basic concepts of creating branches, extending it to the if-else statement not so hard. Extending the definition of if to include the else statment looks like:

IF <condition> <block> [ ELSE <block>] ENDIF

Before we start, we need to modify other to ignore the else keyword. I chose to use a keyword list to ignore rather than test for individual keywords.

    
    keywords = ["if", "else", "end"]

    other :: Parser Statement
    other = (token letters') <=> (\x -> not $ any (==x) keywords) >>> Statement

Parsing If-Else

You could treat else as an optional part of ifthen but I chose to seperate them into seperate type constructors and functions.

    data Statement = Statement String 
                  | Branch Condition Block
                  | Branch2 Condition Block Block
                  deriving (Show)

    statement :: Parser Statement
    statement =  ifelse <|> ifthen <|> other

    ifelse :: Parser Statement      
    ifelse = accept "if" <-+> condition <+> block <+-> accept "else" <+> block <+-> accept "end" >>> br
        where br ((c, b1), b2) = Branch2 c b1 b2

As you can see, it is much the same as the basic if statement. The only difference is that we add a second Block to the type constructor to allow for the else body. Again, recusrion allows for nested if statements in nested if-else statements all the way down to turtles.

Emitting If-Else

The asm for and if-else is not much more complicated than the if version, but it does entail another jump.

<condition> ;calculate the condition
jne else    ;goto 'else' if the condition is NOT equal
<block>     ;the block to execute if the condition is true
jmp end     ;an unconditional jump to skip the else block
else:
<block>     ;the block to execute if the condition is false (the else block)
end:

This is where we really need to keep track of our label counter. Because we need two labels for an if-else, one for the jump to the else portion and another for the jump to the end, the threading gets kind of hairy. I'll just show it to you and let you figure out where it winds it's way through.

    emitStatement s (Branch2 cond b1 b2) = (s', c ++ jmpElse ++ block1 ++ jmpEnd ++ el ++ block2 ++ end)
        where (s1, c)       = emitCondition s cond 
              jmpElse       = emitLn ("jne " ++ lblElse)
              (s4, block1)  = emitBlock' s3 b1
              jmpEnd        = emitLn ("jmp " ++ lblEnd)
              el            = emitLbl lblElse
              (s', block2)  = emitBlock' s4 b2
              end           = emitLbl lblEnd
              (lblElse, s2) = getLbl s1
              (lblEnd, s3)  = getLbl s2

As I said, with 4 intermediate counters, the threading does get a bit hairy. I took a quick look at the State Monad which looks like a nice way to simplify this issue, but it was a bit too much to introduce just yet. I might retro-fit the State Monad as a seperate article outside this series.

While

Third time round we should be able to fly through this. The definition of while is

WHILE  <condition> <block> END

Which is very similar to the if definition. The difference is in the generated asm.

Parsing While

By now we don't really need an explaination for this.

    data Statement = Statement String 
                  | Branch Condition Block
                  | Branch2 Condition Block Block
                  | While Condition Block
                  deriving (Show)

    statement :: Parser Statement
    statement =  while <|> ifelse <|> ifthen <|> other

    while :: Parser Statement
    while = accept "while" <-+> condition <+> block <+-> accept "end" >>> buildWhile
            where buildWhile (c, b) = While c b

No surprises there.

Emitting While

The asm for a while statement is different from an if in that we need a start label to come back to at the end of the loop. We jump to the end label if the condition is true where as in an if we were jump if false.

The desired asm looks like:

L1: 
<condition>
je L2
<block>
jmp L1
L2:

Once again, the lable counter threading is tedious, but other than that the function is the same as we've seen before and is dictated by the asm structure.

    emitStatement s (While c b) = (s', start ++ cond ++ jmp ++ block ++ loop ++ end)
        where start = emitLbl startLbl
              (s1, cond) = emitCondition s c
              jmp = emitLn ("je " ++ endLbl)
              (s', block) = emitBlock' s3 b
              loop = emitLn ("jmp " ++ startLbl)
              end = emitLbl endLbl
              (endLbl, s2) = getLbl s1
              (startLbl, s3) = getLbl s2

Conclusions

For those that haven't been following along, you can get all the code from github. Let's try some complicated samples.

C:\cradle\code\src>runhaskell Lbach.hs "while a if b c else d end e end f end"
L1:
        <condition> a
        je L0
        <condition> b
        jne L2
        <block> c
        jmp L3
L2:
        <block> d
L3:
        <block> e
        jmp L1
L0:
        <block> f
        ret
C:\cradle\code\src>runhaskell Lbach.hs "if a while b c d e end f g else h i if j k end end end"
        <condition> a
        jne L0
L3:
        <condition> b
        je L2
        <block> c
        <block> d
        <block> e
        jmp L3
L2:
        <block> f
        <block> g
        jmp L1
L0:
        <block> h
        <block> i
        <condition> j
        jne L4
        <block> k
L4:
L1:
        ret

Apparently, with only and if and a while statement, what we have is capable of producing almost any program. I wouldn't like to test that theory though. In the next article we will be creating a few more control structures such as for loops and a break statement.

Setting up our compiler to use Cabal

Unfortunately the mkcabal package, which automates the creation of a basic cabal enabled Haskell package, won't install under OS X Snow Leopard yet, so we are going to have to do this by hand. Don't worry - it's not that hard.

Create the following as lbach.cabal.

Name:                lbach
Version:             0.1
Description:         Toy compiler
License:             MIT
License-file:        LICENSE
Author:              Geoff Ford
Maintainer:          g_ford@hotmail.com
Build-Type:          Simple
Cabal-Version:       >=1.2
Executable lbach
  Main-is:           Lbach.hs
  Build-Depends:     base >= 3,haskell98
  hs-source-dirs:    src

The parameters in this file should be pretty self explanatory, with the main ones to keep an eye being the dependencies and the entry file. One thing to note is the hs-source-dirs parameter. This says that all the code can be found under the src directory, and the Main-is parameter will therefore look for src/Lbach.hs.

You'll also need to create a file called LICENSE and create setup.hs as:

    import Distribution.Simple
    main = defaultMain

Once this is done, move lbach.hs to src\Lbach.hs and then we need to modify it so that it will run from the command line. For now, we'll just make it so it tries to parse the string passed in as the command line argument. Add the following at the top of your file in place of import Char:

    module Main
    where

    import Char
    import System.Environment

    main :: IO ()
    main = getArgs >>= print . parse . head

This simply creates a module called Main and a function called main, which all Haskell programs must have as their entry point. Before we test the package, we should test that the program works.

~/Projects/compilers/cradle/code/src> runhaskell Lbach.hs "test=1+2 / counter"
Assign "test" (Add (Num 1) (Div (Num 2) (Var "counter")))

Awesome. Now let's test if cabal can package it up and install it for us. We'll only install it for ourselves just to be on the safe side.

~/Projects/compilers/cradle/code> cabal install --prefix=$HOME --user
Resolving dependencies...
Configuring lbach-0.1...
Warning: 'license: MIT' is not a recognised license.
Preprocessing executables for lbach-0.1...
Building lbach-0.1...
Installing executable(s) in /Users/geoff/bin 
~/Projects/compilers/cradle/code> ~/bin/lbach "test=1+2 / counter"
Assign "test" (Add (Num 1) (Div (Num 2) (Var "counter")))

Modularisation

To keep our program source files from running to unmanageable lengths we will need to break them up into separate files. This is also a good time to think about how we are going to split the compiler up into modules.

We immediately know that there are three basic modules - the Parser, the Emitter and the Grammar. I have also separated the expression parsers from the general parsers. I'll also prefix everything with an Lbach namespace so as not to get in the way of any existing namespaces.

With Haskell, modules are expected to be in files and folders that reflect their name. For example the module My.Great.Module should be in the file My/Great/Module.hs. This gives the following file structure.

src
  |- Lbach.hs
  |- Lbach
    |- Parser.hs
    |- Parser
      |- Expressions.hs
    |- Grammar.hs
      | Basics.hs
    | Emitter.hs

I've updated the emitter to handle our changes to Expression and the addition of Assign.

The reason we have all the code under a src directory is because of the common practice of also having a test directory at the same level. This keeps the 'real' code and the test code nicely separated.

What goes where is a matter of personal taste. I won't describe in detail how I broke the compiler up, but you can always check it out at github.

Don't forget to check that cabal can still package it up and install it.