alpinegizmo

One last ride around the Alster

2009-09-29T00:00:00+02:00

After almost exactly a year, my time in Hamburg is at an end. I'll miss many things here -- pretty good Mexican food, really good Thai food, and the great bunch of Ruby developers at Xing.

This afternoon the weather was great and I decided to take one last ride around the Alster on my kickbike, which you can see above.

When friends see my scooter for the first time, they often look a bit puzzled and ask "Why?". Here are some reasons why I like this better than a bicycle:

After a long ride, my muscles are sore, but my joints are fine, nothing is numb, and my butt isn't sore. I have arthritis in my left foot, and for the past couple of years, long walks would result in days of pain. At first I was concerned that propelling myself on the kickbike would also hurt my foot, but on the contrary, it seems to have helped. My theory is that I've strengthened the muscles in my foot by using it this way; all I know for sure is that my foot doesn't hurt any more.
It works more of my body: not just my legs, but also the lower back, lower abs, and even the arms a bit.
It's simpler, lighter, and cheaper than a comparable bicycle. No pedals, greasy chain, or derailleur to deal with.
Because I can readily put my feet on the ground, I feel it's a bit safer when riding in a densely pedestrian urban environment.
And it's fun.

On my way home, I stopped at Qrito for one last burrito.

Just brilliant

2009-03-31T00:00:00+02:00

Another brilliant list from McSweeney's: Email addresses it would be really annoying to give out over the phone.

Wow

2009-03-24T00:00:00+01:00

With some of these youtube videos, even if parts are fake (and I have no reason to think this isn't legit), enormous imagination and effort were required. This video gets the alpinegizmo seal of approval.

Kings

2009-03-23T00:00:00+01:00

If you want one new show to watch now that Battlestar Galactica has ended, make it NBC's new show, Kings. I haven't been this excited by imaginative television in years. In the US you can catch it on hulu. The ratings weren't good, by the way, so let's all cross our fingers and hope NBC gives this show a chance to breathe.

If you prefer to wait and watch a whole season of something at once, take a look at the first season of Damages, also on hulu. Both Glenn Close and Ted Danson play characters who will stop at nothing to get what they want -- and of course they are in conflict, and can't both win. Ted Danson is driven by greed and ambition; Glenn Close wants to bring justice to those ruined by Ted Danson. She's on the good side of this battle, but she's got more darkness in her than a black hole. And I've left out the appealing main character -- Glenn Close's young associate -- will she succumb and become just as twisted as her mentor?

Rails on XML, Part 7: XML serialization (a gem)

2009-03-01T00:00:00+01:00

[Part of the Rails on XML series.]

An essential part of working with XML and Ruby is being able to serialize ruby objects to XML. There are several approaches out there for doing this; the one that appeals to me most is the one implemented by Rails in ActiveSupport. It provides a nice to_xml method for your ActiveRecord objects, and deals with arrays and hashes that contain ActiveRecord models. For example, here's what you might get from a really simple blog engine in response to Page.find(:first).to_xml:

<?xml version="1.0" encoding="UTF-8"?>
<page>
  <allow-comments type="boolean" nil="true"></allow-comments>
  <body>blah blah blah</body>
  <id type="integer">1</id>
  <title>my first post</title>
  <created-at type="datetime">2008-01-08T17:52:54+01:00</created-at>
  <updated-at type="datetime">2008-11-26T17:01:51+01:00</updated-at>
  <user-id type="integer">1</user-id>
</page>

There may be more information here than we will need for XSLT rendering (eg it probably doesn't matter that id is an integer), but I like the thoroughness of this approach, and the way they handle arrays and hashes is quite nice.

For our purposes we need to be able to serialize to XML anything we might put in a controller instance variable. The xml_serialization gem adds support for integers, symbols, and strings, and arrays and hashes that contain them.

For example, {:numbers => [1, 2], :strings => ['one', 'two']}.to_xml

<?xml version="1.0" encoding="UTF-8"?>
<hash>
  <numbers type="array">
    <number>1</number>
    <number>2</number>
  </numbers>
  <strings type="array">
    <string>one</string>
    <string>two</string>
  </strings>
</hash>

We also cover the important case where you already have XML (eg from a database or web service) that you want to pass through unmolested.

@data = RawXML.new '<tag>content</tag>'

In this case, @data.to_xml returns the same string that went in, i.e. '<tag>content</tag>'.

Rails on XML, Part 6: The software

2009-02-28T00:00:00+01:00

[Part of the Rails on XML series.]

I am pleased to report that Mirai España has agreed to release as free software the code we wrote for using XSLT views in Ruby on Rails applications. I have packaged this software as two separate components: the xml_serialization gem for serializing ruby objects to XML, and the xslt_render rails plugin.

At this point documentation is lacking. However, there isn't that much code and the tests do illustrate how the pieces are intended to be used, so the adventurous might be able to drop this into an app and get it working. I do plan future posts that will explain more of the details, which I'm sure will help. However, I know a few of you are interested in seeing the code now, so I don't want to delay in releasing it.

Escape from magical thinking

2009-02-15T00:00:00+01:00

Europeans often ask me why I moved to Europe (interestingly enough, I can't recall an American ever asking me this). One reason was to escape from the magical thinking so many Americans engage in. And which groups are most likely to need escaping from? Pew has prepared this handy chart.

Are you my doctor?

2009-02-01T00:00:00+01:00

This week I was noticing one of those small, but possibly revealing cultural differences between France and the US. It has to do with what doctors wear at work. I was having a routine examination done, and the specialist I was there to see -- male, mid-40s -- was wearing jeans and a long-sleeved fantasy sport league polo shirt. No white lab coat or stethoscope or any other of the traditional trappings of the medical profession (traditional, that is, from an American perspective). At first when I saw him coming and going about the office, I wasn't sure if he was the doctor, or a technician there to repair something. If I wasn't used to the fact that french physicians dress extremely casually, it never would've even occurred to me that he could be the doctor.

I'm not complaining, mind you. Just curious about what else might be different, but less obvious.

2008 in figures

2009-01-24T00:00:00+01:00

Weeks at home: 30
Country points: 4
Airline flights: 36
Photos: 3222

Countries visited for the first time: Portugal, Russia, Latvia, Lithuania

Flight departures: Marseille (12), Madrid (11), Lyon (2), Hamburg (2), Nice (2), Riga (2), Amsterdam(1), Berlin (1), Paris (1), Porto (1), St Petersburg (1)

Note for travelers: calling 112 in Portugal is not guaranteed to get you someone who speaks English.

No ID required

2009-01-23T00:00:00+01:00

Today is a travel day for me, and I'm writing these posts while sitting in various airport terminals. First Hamburg, and now Frankfurt. What I find rather odd is that it is possible to travel within Europe without ever showing any ID. I used my credit card to get my boarding pass, and I showed the boarding pass at security and at the gate. At the gate there wasn't even a person involved, just a barcode scanner.

Mysql on an encrypted disk image

2009-01-23T00:00:00+01:00

Yesterday I decided it would be a good idea to modify the installation of mysql that I use on my laptop so that the data on the disk is encrypted. With google I found a few people looking for help with this, but no solutions, so I'll share what I figured out.

First some configuration details. I'm running mysql 5.0 on a macbook pro with leopard (OS X 10.5.6). I'm pretty sure the same approach I'm using will work across any similar setup, but your mileage may vary. Also, I installed mysql from mysql.com, but you should be able to easily adapt this technique to work with the version from macports.

After shutting down the mysql server, the first thing I did was to create an encrypted sparse disk image (using disk utility), move /usr/local/mysql/data to the new disk image, and create a symbolic link from /usr/local/mysql/data to the new, encrypted location. I expected that to pretty much take care of it (and if it had, I wouldn't be blogging about it). When I looked at why mysql would no longer start up, I found that the wrong user now owned the data files -- I owned everything, instead of the mysql user. Nothing chown can't fix, right? Wrong.

Turns out that the files and directories in mounted disk images are owned by whoever mounts them -- the ownership information stored in the filesystem within the disk image is ignored. Yikes! Fortunately, this is merely the default behavior, and can be switched off. If you bring up a mounted disk image in the finder's get info window, at the very bottom, inside the "Sharing & Permissions" section, you will find a checkbox labeled "Ignore ownership on this volume". It is checked by default; you want to turn this off.

You will notice that I gave everyone read and write permission to this volume. I'm not super happy about this, but so far I haven't found any other way to allow the mysql user to be able to reach inside and do what it needs to do. Of course, what this volume contains is a data directory that only the mysql user is able to access, so I don't feel too bad. Nevertheless, if someone finds a solution to this, let me know, ok?

The other problem I've found comes up when I want to unmount the mysql data disk image. I usually find the finder telling me I can't eject this volume because it is still in use. In these cases lsof is my friend, as in "lsof /Volumes/mysql-data". And the culprit is usually mds, aka spotlight. You can stop this by going into System Preferences > Spotlight > Privacy and adding the encrypted disk image volume to the list of things spotlight should not index. Unfortunately it seems that this preference is not sufficiently persistent -- I keep having to re-set it -- so I may resort to disabling spotlight entirely, or try spotless.

Of course, it is also necessary to mount the encrypted disk image before starting mysql, and to stop mysql before unmounting it. Kudos to anyone who goes to the trouble of cleanly automating these steps.

Rails on XML, Part 5: Rails partials with XSLT

2009-01-19T00:00:00+01:00

[Part of the Rails on XML series.]

Two really nice things flow from using XSLT for the views in the way I described last time, in Part 4.

First is that you get partials for free. To understand why this is so, you need to understand a little about how the XSLT language works. When you are writing a view in XSLT, you are describing a transformation from XML to HTML. You are saying, when you see this in the XML coming in, produce that in the HTML coming out. This means that setting up a controller action to re-render part of a page is staggeringly simple. All you have to do is provide a subset of the XML used to render the full page, and only the relevant portion of the HTML will be generated. You can use exactly the same XSLT view for both the full page and the partial.

The second thing you get is a much cleaner separation between your controllers and your views. Everything the views need will be, and must be, provided in the XML. And given the rendering pipeline we established, that means everything will come from the controller instance variables. Of course most everything in normal rails views comes from controller instance variables too, but it always possible to do something dirty in an rhtml view, like reach around to the models and include a bunch of business logic in your view. Since we have no ruby code in our views, these cheats are simply impossible. That may seem like a limitation, but it can be liberating to be able to say that there is a contract between the backend team and the frontend team whereby the backend team will produce XML that contains everything that’s needed, and the frontend team will produce HTML from that XML. We found it was a good thing to have a real boundary there.

Rails on XML, Part 4: Rendering views with XSLT

2009-01-18T00:00:00+01:00

[Part of the Rails on XML series.]

One of our project’s major goals was to use XSLT for the views, and to integrate an XSLT rendering path into Rails in as natural a way as possible. This meant that each typical controller action would have a corresponding .xsl file, rather than a .html.erb file.

An XSL stylesheet is a program that describes a transformation; in our case a transformation from XML to XHTML. In some ways the XSL transformation language (XSLT) is very awkward, but it is a programming language. For various reasons – because our client already had XSLT files for his site, because there were members of the team who knew XSLT and not Ruby, and because it just seemed cleaner – we decided to see how far we could get if we used pure XSLT for the Rails views, with no embedded Ruby.

We realized that if we didn’t use Ruby in these views, we would not have direct access to the controllers’ instance variables. We decided the most Rails-like approach we could think of would be to have the framework automatically make all controller instance variables available to the XSLTs, by serializing this data to XML, and then provide the resulting XML as the input to the XSLT view. At this point we decided it was time to dig in and read through the Rails ActionView code and figure out how to extend it to work this way.

We found it is pretty easy to add a new renderer to Rails, and we experimented with that approach for a while. Ultimately we found it better for our application to write a simple module that we include in our controllers. This module defines a render_through_xslt method that works like this:

module XSLTRender
  def apply_xslt(template_file, xml)
    # use xsltproc to perform the transformation
  end

  def instance_hash
    hash = instance_variables.inject({}) do |vars, var|
      key = var[1..-1]
      vars[key] = instance_variable_get(var)
      vars
    end

    hash['flash'] = flash
    hash
  end

  def render_through_xslt(options = {})
    template = find_template(options)
    page_xml = instance_hash.to_xml
    html = apply_xslt(template, page_xml)
    render :text => html
  end
end

The final version is a bit more complex, as we added support for rendering partials and doing internationalization.

Comments via disqus

2009-01-07T00:00:00+01:00

Last night I spent a couple of hours integrating disqus into this site. This is a web service that implements comments. My server is still just serving up static web pages, now with a bit of javascript included. Look at the source for this site if you want to see how it works on this end.

So far I'm really delighted with how this switch to jekyll + disqus has gone. I can't recommend this solution if you're not comfortable with dealing with HTML, CSS, and the command line, but if you are a programmer -- especially if you are a ruby programmer -- you may find the simplicity compelling. I am a little concerned in the long run about depending on a company with no visible business model for the comment functionality, but I guess I'm willing to take the risk. Maintaining a viable comment feature in the face of torrents of comment spam is too much work, so I'm happy to outsource the problem.

Rails on XML, Part 3: XML workflow

2009-01-06T00:00:00+01:00

[Part of the Rails on XML series.]

A bit of overly simplified, sample XML. Don't get hung up on the details; the schema and data are made up. I just want to give you something concrete to look at so I can raise a few issues for you to consider.

<property>
  <type>office space</type>
  <area>100</area>
  <rooms>
    <room size="60" />
    <room size="30" />
    <room size="10" type="bathroom" />
  </rooms>
  <owner>
    <name>John Q. Public</name>
    <contactinfo>
      <phones>
        <phone number="1.555.555.1212" type="voice" use="contact" />
        <phone number="1.666.555.1212" type="mobile" use="contact" />
      </phones>
    </contactinfo>
  </owner>
  <location>
    <position lat="40.5000801086426" lng="-3.37295293807983" />
    <address>
      <street>123 Main St</street>
      <postalcode>28805</postalcode>
    </address>
  </location>
</property>

Creating, editing, and validating the XML

If you are thinking of writing an XML-based web application, where is the XML going to come from? How will it be edited? Where will you implement the business logic to validate that the data being entered makes sense?

Changes to the structure/schema of the XML (migrations)

How will you manage the evolution of the XML schema over time -- in other words, what will take the place of Rails migrations? Will you validate XML schema changes against an XML DTD?

Will (some) users be able to make changes to the XML schema, or will this require programmer intervention?

Finding good solutions to the problems of validation and migration is a key challenge when considering Rails on XML. In our application we generated the XML from other sources, and thereby sidestepped these issues, at least for a while.

Rails on XML, Part 2: Data modeling

2009-01-04T00:00:00+01:00

[Part of the Rails on XML series.]

Among the first design decisions to be made with any web application are how to represent the data, and where to store it. The assumption behind this set of articles is that you are going to have XML data, probably in some sort of XML database. But why might someone come to this conclusion, and what are the alternatives?

For many applications, modeling the data for a relational database is pretty straightforward, especially after you've done it many times before. Customer records, addresses, phone numbers, orders, login-ids and passwords -- we all know what to do with this stuff. Moreover, there are good tools available, and there are lots of talented people around who know how to use them. Both MySQL and PostgreSQL have their weak points, but are nevertheless pretty well-behaved and a small team isn't going to waste much time fighting with either one of them. Rails, with ActiveRecord, migrations, and validations, provides a good foundation for keeping complex applications relatively sane.

By contrast, choosing an XML data representation is much riskier. The tools are less mature, and few there be that know how to use them. Don't do it unless you have a good reason.

One obvious domain for applying the XML web application toolchain is for working with an existing set of XML documents. I suspect this is a relatively rare case, however.

As a motivating example, imagine being asked to build a commercial real estate site that will have detailed listings for business properties for rent and for sale: farms, office space, retail space, restaurants, hardware stores, etc. New types of properties will be added to the site on a regular basis -- six months from now you may encounter the first listing for a marina, or a mine. The web site is expected to have rich editing and search interfaces, making it possible, for example, to search for bars for sale with seating for more than 50 customers, or for a 2-room office with an on-street entrance and its own bathroom.

In this case, modeling each property as an XML document seems like an interesting idea. Attributes that many or all of the properties have can be managed uniformly (eg listing agent, asking price, size in square-meters, taxes), and more unique characteristics (eg number of berths of various lengths in a marina) can also be represented, edited, and searched.

Once you have decided to use XML for at least some of your data, you have to decide if you are going to be a purist and use XML for everything, or not. The cleanliness of a pure approach appealed to us, but didn't seem practical. We already had a login system we liked, based on restful authentication and open-id; we planned to use attachment-fu for uploaded images; and in general we wanted to leave the door open for any and all Rails plugins. Our client had already chosen to use DB2, which supports hybrid use of both XML and SQL, and we decided to take advantage of that.

Another interesting choice that I've been playing with, but haven't used for anything real, would be to use one of the new document-oriented data stores, such as CouchDB. It is my belief -- but I can't completely justify it -- that if you are planning to exploit the hierarchical structure that an XML document allows you to use, then XQuery is going to win over what you can do with CouchDB.

Rails on XML, Part 1: Background

2009-01-03T00:00:00+01:00

[Part of the Rails on XML series.]

I'm sure the first question in many readers' minds is "Why bother?". The intersection between the XML community and the Ruby community is very small. A rubyist might use XML to implement a RESTful web service, but that's going far enough, thank you very much. However, if you've been paying attention to the pure XML workflow for creating web applications, it has some interesting ideas, and some exuberant followers. I'm not sure I'm tapped into the most forceful propaganda for this way of thinking, but you can sample it here, in this quote from 2006:

A profound change is likely about to shake up your world if you’re a web developer, one that I suspect will make the recent efforts in the AJAX space pale in comparison as far as its effect. Very quietly, over the last few weeks, the Mozilla team has been upgrading their XForms capabilities ...

In point of fact, you can create some incredibly rich “web experience” applications that are mixed XHTML and XForms, and can do so in a remarkably short amount of time. These are not simply empty text fields that you lay out as you do with HTML input elements -- XForms bears about as much resemblance to HTML forms as a Bengal tiger has to a ferret. http://www.oreillynet.com/xml/blog/2006/03/why_xforms_matter_revisited.html

and in this quote from 2007:

[E]nlightened innovators ... have seen the real world benefits of dumping object middle-tier stacks and relational databases and going with a pure declarative approach to solving business problems based around XForms/REST and XQuery. These [are] the people that are going to lead innovations in application development. ...

At one of the XForms sessions a young Ruby advocate stated that he thought his Ruby code was "beautiful" but he did not himself think that XForms code was "beautiful". Most of the people in the audience agreed that XForms was beautiful but like any new language, it takes getting used to. ...

XForms/REST/XQuery to me is indeed beautiful. Its beauty comes from its ability to quickly map real-world requirements into working systems with high fidelity. No other system in the world approaches its elegant architecture. http://datadictionary.blogspot.com/2007/12/impressions-of-xml-2007-in-boston.html

To put these remarks in context, I need to backup and give an outline of the entire XML web application toolchain.

XML database: your application data lives here, in XML
XQuery: database query language, it takes the place of SQL
XSLT: transforms XML to HTML (can be applied in the DB, your app server, or in the browser)
XForms: forms that post XML back to the database

Note that if you buy into all of this, you don't really need an application server any more, be it implemented in Rails, or not. (If you want to play with some examples of this pure XML web app architecture, I think the sample apps distributed with eXist, an open source, native XML database, are the best place to start.)

Of course it is now 2009, and we've yet to see the XForms/REST/XQuery approach gain traction in the way you might've expected, if you bought into what was being said a couple of years ago. Yes, in some ways this XML architecture for web application development is truly elegant -- but some pieces of the puzzle are rather hideous.

I'll have more to say about what's good, and bad, in future postings. I'll also talk about various approaches we explored for combining these tools with Rails.

Rails on XML, the Series

2009-01-02T00:00:00+01:00

For much of 2008 my partner Larry Baltz and I were exploring how to build Ruby on Rails applications on top of an XML database. Our client was convinced that some of their data was inherently better suited to an XML representation, and wanted to see if we could craft a productive development environment from a marriage of these technologies. The journey has been quite interesting, revealing both pitfalls and pleasant surprises. Hopefully this account will save others some grief, and also point out some interesting directions for future work.

In the series of postings to follow I will cover various aspects of this journey.

Happy New Year!

2009-01-01T00:00:00+01:00

I'm starting off the New Year right -- not simply with a long overdue blog posting, but also a whole new blogging infrastructure. For a while I've been using Typo, but I've come to realize it is way too bloated for my simple needs. So now I am using jeykll, which generates the static site you see here from a simple git repository. Take a look if you want to see how it works. So I far I think it's a big improvement -- for you, because the site is so much faster, and for me, because it is much simpler and easier to maintain.

If you are following the feed for my blog, I recommend you update your subscription. So as not to lose readers who've been using Typo's RSS feed, I added some rudimentary RSS support to what I found in jekyll, but the atom feed is more robust.

I am painfully aware that I've spent several hours playing with the code for this blog today, and only a few minutes writing. This may say something about which activity I enjoy more, but hopefully I will sustain a more regular posting schedule in 2009.

(im)pureXML

2008-03-01T00:00:00+01:00

Rather than producing a native XML database like eXist or MarkLogic, IBM has chosen to add XML features to DB2, creating an interesting -- and convoluted -- hybrid that IBM has branded "pureXML".

If you've drunk the XML database koolaid, you may be thinking about how going down this path is going to (1) give you a toolchain where web app development becomes a much more declarative process, with better separation of concerns; (2) give you more flexibility than a relational database; (3) fit more naturally with a web-services model; and/or (4) get you moving toward the semantic web.

For me, one of the attractive prospects would be to clean up the process of generating HTML. If the data coming out of your database is already XML, then its not much of a leap to use an XSLT to transform that XML into HTML. Traditional view templating schemes (such as erb) can be a mess, so maybe this world of XML databases offers a cleaner solution.

Architecturally you've got three choices of where to apply an XSL transform -- in the database, in your application server, or in the browser. Each of these has advantages and disadvantages, but as a first pass we've been exploring having DB2 do the job. The results have been somewhat disappointing.

To get DB2 to apply an XSL stylesheet to some XML, you have to use an SQL function called XSLTRANSFORM. The XSL stylesheet has to be provided as the result of an SQL expression, which generally means that it is stored in an XML column in the database. The XML document to be transformed might also come from an XML column, or it might be the result of an XQUERY embedded in the SQL you've been forced to use as the overall structure for this operation. Given this structure it's not surprising that xsl:include declarations are not supported, but this is definitely problematic. Also disappointing is that stylesheets incorporating XQUERY statements are not supported.

A simple example in which the XML to be transformed is coming from the xmldata column in the employees table:

SELECT XSLTRANSFORM(xmldata 
  USING (SELECT xsldata FROM xslts WHERE name = 'employee.xsl')) AS html 
FROM employees WHERE id = 833038373

It's a little more fun when the source of the XML is an embedded xquery:

SELECT XSLTRANSFORM (
 (SELECT XMLCAST(XMLQUERY('$data//Phones' PASSING xmldata AS "data") AS XML) 
   FROM employees WHERE id = 833038373) 
 USING (SELECT xsldata FROM xslts WHERE name = 'phones.xsl')) AS html 
FROM employees WHERE id = 833038373

It may look a bit strange that the WHERE id = 833038373 clause occurs twice. The second instance could be pretty much anything that returns one row. While many SQL databases will accept a simple "select 1", DB2 insists on a FROM clause, such as "select 1 from employees fetch first 1 rows only". The only really important part of the last line above (i.e. "FROM employees WHERE id = 833038373") is that it return exactly one row.

XSLTRANSFORM isn't the only case where DB2's pureXML forces you to use an unholy mixture of SQL and XQUERY. As IBM explains, "With plain XQuery you can not exploit full-text search capabilities provided by the DB2 Net Search Extender (NSE). You need to involve SQL for full-text search." This means doing something like this:

    xquery 
    <Employees> { 
        for $e in db2-fn:sqlquery(
          "SELECT xmldata FROM EMPLOYEES.EMPLOYEES 
           WHERE CONTAINS(xmldata, 
             'SECTION(""/Employee/Info/Address/LocationInfo"") ""madrid"" ') > 0"
        ) 
        let $ei := $e/Employee/Info 
        return 
          <EmployeeInfo>
            { $ei/Name, <Departments> {$ei/DepartmentInfo/Name} </Departments> } 
          </EmployeeInfo>
    } </Employees>

For clarity, this example has been broken across multiple lines. Don't expect DB2 to be happy with input this easy to read.

This whole business of XQUERY embedded in SQL, and vice versa, can get amazingly messy. We've spent quite a bit of time figuring out how to do quoting and casting in order to keep both sides happy. And although DB2 supports SQL sub-queries within XQUERY, so far we've found it impossible to figure out how to quote SQL sub-queries within an XQUERY that in turn is embedded in SQL, which is necessary if you want to use XSLTRANSFORM on the result of an xquery like the one above.

Resources

7 weeks on DB2

2008-03-01T00:00:00+01:00

For a new project we've been using IBM's DB2 database and its pureXML features. This has been my first serious exposure to the brave new world of building web applications on an XML platform. Now that I've been doing battle with DB2 for seven weeks, I thought I'd give a progress report. In subsequent posts I'll try to share what we're learning about how well this all works.

In the past I've used SQLite, MySQL and PostgreSQL (usually underneath Rails) for web development. Compared to these open source databases, my first impression of DB2 has been -- to be blunt -- that it sucks. Now I'm aware that proponents of DB2 (see Zen And The Art Of Ruby Programming, for example) will tell you that because Rails is database agnostic (with a mysql bias) it doesn't show off the strengths of DB2's advanced database features. Apparently, DB2's strengths include "utmost speed, pureXML, compression, replication, high availability, [and] affordable 24/7 support." That may be true. I feel a bit like I woke up one morning, and found an 18-wheeler in my garage. It may be extremely capable and powerful, but so far, I'm just feeling proud that I've managed to get it down the driveway and onto the street without hitting anything or killing anyone. I still miss all the things that made driving my car a comfortable experience.

IBM, if you're listening, there are a few simple things you could do to make the transition to DB2 less painful:

Put some resources into improving the usability of the tools. For starters, put readline support into the command line processor, and modify the command editor in db2cc to wrap lines within a fixed-width textarea. (Yes, I'm aware of the history and edit features in CLP.) Figuring out how to do really complex queries with the current tools has been a seriously frustrating experience.
Get rid of silly limits. For example, the shell interface truncates XML results to 4000 bytes. What's the point of offering a shell-based command line processor if you can't use it in shell scripts?
Find some way to make routine administrative tasks, including backup and restore, managing text and xml indexes, and doing server restarts, less of a problem. We're spending way too much time fighting with the basics.
The ibm_db ruby on rails adapter is half-finished. Please get us to the point where "rake test" works.

We're getting past these issues by building in-house tools to smooth over the rough edges. Why bother? pureXML is the answer. In the future I hope to let you know if it proved worth the bother.

The E-111 is dead, long live the E-111

2008-02-01T00:00:00+01:00

Just to clarify for folks ... the old E-111 form has been replaced by the new European Health Insurance Card. Since Jan 2006 the E-111 is invalid, and you need to get one of these cards, which look like this:

I think this is actually easier to deal with, but, now that there's a better system in place, it would appear that the hospitals are insisting on folks having these cards.

A medico-linguistic adventure

2008-01-31T00:00:00+01:00

This is the fifth morning I have woken up in a hospital in Madrid. Unless something unforeseen changes the plan, I will be leaving the hospital sometime today, and going home to France tomorrow.

This has been a rather interesting experience, both medically and linguistically. I've learned that it is surprisingly difficult for me to brush my teeth with my left hand; that Spanish nurses are, as a whole, very warm and friendly (and some are very pretty); and that self-medicating minor problems for decades might be a bad idea. I've also used these few days as an intensive language learning experience, and I'm quite pleased with myself on that account.

Perhaps I shouldn't jump to too many conclusions from a single data point, but the socialized health care system in Spain would appear to be working quite well. Admittedly I am in the hospital for central Madrid, near the royal palace, but the level of care provided here and the professionalism of the staff are excellent.

Having a medical emergency in a country where you don't speak the language adds an extra level of scariness to the whole experience. Thanks to my knowledge of French I can understand some fraction of what people say, if they are speaking slowly and clearly, but when I arrived here at the hospital (by ambulance) I could communicate very, very little in Spanish. Fortunately, right after arranging for the ambulance I had the presence of mind to call someone I know locally who was able to quickly join me and translate for the EMTs and ride with me in ambulance. It was reassuring to have Minoru there during that initial period of engagement with the Spanish medical system.

Initially most everyone I encountered spoke either no English, or a teeny bit (and no French, either) -- but in the final tally what I've seen is that many (perhaps most) of the doctors here speak English, some extraordinarily well, but almost no one else in the system does (a few of the nurses speak French pretty well). Meanwhile I've learned enough Spanish to say things like "my arms hurt," "my head is congested," and "I need something for my fever."

A few thoughts for others who might find themselves hospitalized while traveling in a foreign country:

You are going to need your passport. The ambulance drivers insisted on it, and waited for someone to fetch it from my hotel room (the whole adventure started in the hotel, so that didn't take long). I have no idea what would've happened had I not had it handy, but I gather it would've been complicated. The hospital also insisted on seeing my passport, presumably in order to later deal with extracting payment from someone. I don't normally keep my passport with me when I'm traveling, and I probably still won't, but it's something to think about.
If you are a european resident, travel with a valid european health insurance card (model E-111). I didn't have one; I just had the French Carte Vitale (the social security system card), and this caused some concern at the hospital here for a few days until my wife was able to get the French to produce some appropriate paperwork. If I had been in and out of the hospital over the weekend, I think they might've tried to force me to pay them somehow.
If you regularly take any drugs (even things like Tylenol), learn their non-branded names (eg Tylenol is paracetamol). You can't expect foreign doctors to know all of our US marketing jargon -- things like Sudafed and Pepcid-AC may just leave them scratching their heads.
Try to get a roommate who speaks a little English. Despite being hard of hearing and having somewhat broken English, Eusebio was really helpful to me on several occasions. Given how few adults in Spain speak English, I give the hospital credit for finding me a Spenglish speaker to be my roommate.
At least in France and Spain, the hospitals don't provide anything for the patients to wear. Patients' families provide them with clean underwear and pajamas. If you are traveling alone I don't know how you'd solve this problem. "Put on clean underwear every day, in case you wind up in the hospital, and put a few extra pairs and some PJs in your laptop bag in case you have to stay a few days. And maybe your toothbrush and some slippers." Not very practical. Fortunately I had a friend who could bring me the things I needed from my hotel room. Thanks, Larry.
The Lonely Planet Spanish Phrasebook is terrific. I have several other books for learning Spanish here with me that I've used to good effect, but that alone would've been enough to get me past the worst of the language barrier.
And my final piece of advice for travelers: try to get sick in a country where the people are naturally outgoing, like Spain or Italy.

And the bad? In a couple of ways the experience has been diabolical. What put me in the hospital was problems with my esophagus, and my digestion, following a history of problems with choking on things (well, ok, not "things", but food and pills). So after 3 days of a liquid diet what am I served for my first solid meal? A terrifying piece of very bony fish. And now part of my prescription involves some horse pills that there's no way I can bring myself to swallow. I'm grinding them up and hoping that's not going to cause me some other problem. Besides the fish (which was pretty good, I'll have to admit), the rest of the food was only scary because of massive quantities of indigestible grease -- just what I'd serve if I ran the gastro-enterology ward of a major hospital.

Update later the same day: yes, I have indeed left the hospital. And many thanks to all those who've sent their well-wishes.

Invalid certificates and GMail notifier

2008-01-26T00:00:00+01:00

I've been using Google Mail as my primary email solution for some time now (I do send a copy of my business email to another server, just in case). One of the things I like about Google Mail is their notifier for OS X. But until yesterday I've been using a version from 2005, because all newer versions have had a big problem: at least for me, modern versions of the gmail notifier pop-up a dialog every 10 minutes promping me to signin again with my Google email address and password. This goes beyond annoying into the realm of totally unusable. (I also preferred the old version because it was small and simple. The newer versions use a lot more memory to provide features I don't want.)

Yesterday my old, cherished gmail notifier stopped working. After going all day with a silent gray M in my mac's menubar, I decided to try the latest version, which, sure enough, gave me trouble. Convinced that google had finally turned off some service the older version depends upon, I was motivated to find a solution -- which I seem to have found. If you're having this problem, here's what to do:

Open Safari.
Browse to https://google.com
Accept the invalid certificate that's causing the trouble. In my case the certificate for google.com came from google.es.
That's it.

Seems like google could make this a lot easier.

Cowboys and Waterfalls: Why research labs don't produce good software

2008-01-09T00:00:00+01:00

I spent the first twenty-plus years of my career working as a researcher in university and corporate research labs -- at Carnegie Mellon, Mitsubishi Electric (MERL), and Sun Labs. It has been my privilege to work on some great projects with some amazingly talented people. But I have to tell you that by and large, the quality of the software coming out of research labs isn't very high. I blame the cowboys and the waterfalls.

So what goes wrong, you ask?

Most researchers are pretty good programmers, in their domain of expertise. Some aspect of the software they create is likely to be really well done -- the efficiency of the algorithms, the correctness of the protocols, the usability. But other aspects of the code don't get as much care. Very few researchers make a significant effort to hone their general software development skills. It's common to go straight from graduate school into a research job, never having worked in a product group. Just to give you the flavor, it's not uncommon for cvs and svn to be considered too much trouble (rather than pains in the neck that ought to be replaced by git or mercurial or darcs -- but that's another topic entirely).

From what I've seen, most research lab directors will try to put some serious programming talent wherever it will do the most good, but corporate research labs often have considerable difficulty finding and keeping talented developers. Several factors contribute to this:

Vagueness. How a research project may impact products, who the customers will be, etc, may not be clearly understood. Developers who are used to getting written specs from the marketing department, or consultants who expect the client to provide a detailed requirements document often feel at sea in this environment. (The upside is that there is plenty of room for creativity.)
Low probability of success. For the researchers, success can come in many forms -- papers, patents, peer recognition. For the programmers who help them realize their ideas, success means seeing their code get used, either in an open source project or a product. The chance of this happening can be quite low.
Inappropriate structure and leadership. Corporate research labs are often part of a vast enterprise. Management of development resources in these research labs is usually left either to scientists who aren't particularly knowledgeable about industry-best practices in software engineering, or to managers rotated in from some other part of the company, whose experience is in a very different context.
Cowboys and waterfalls. The few really great coders in any research lab are likely to either be cowboys who can create amazing things overnight, but who lack the discipline to create code with lasting value -- or they may be believers in the "big upfront design" school of software development (aka the "waterfall" model). Management usually loves both the cowboys and the serious architects, because both appear to get things done, and are sometimes successful (though real long-term results are rare in most cases). Flashy demos, and grand schemes described in lengthy documents (and patents) can be impressive.

What would be better?

Agile methods of software development (and extreme programming practices) are an excellent match for a research environment, but I've seldom seen them applied in that setting in anything better than a haphazard manner. (I am assuming you have some familiarity with the basics of these methods of going about software development -- if not, see the resources at the end of this article for some pointers to further reading.)

Here are some reasons why agile methods are a natural fit for research-driven software development:

Constant flow of new ideas. It's part of the nature of research to be constantly adjusting your plan to fit with what you've just learned, and agile methods are all about adaptability. About ten years ago, while working as a researcher at Mitsubishi, I read in one of Tom Peters books that at Sony, the average time it takes to go from an idea to a prototype is one week. I was flabbergasted, but intrigued. Most research ideas fail -- your goal should be to fail faster!
Small teams. The conventional wisdom is that agile methods work well in small teams, but may be difficult to scale. Not really an issue in most research environments.
Variable scope is ok. When push comes to shove and your project threatens to be late, as a manager you have three rational choices: move back the deadline, reduce the scope of work, or reduce the quality (a death march isn't a rational choice).

And some reasons why there might be problems:

No customer (or plausible stand-in) to drive decision-making.
Lack of training and mentors.

Advice to research directors

Foster a climate of sustainable development. Occasional crunches to meet deadlines are natural, but don't let it get out of hand. Communicate that in general, choosing to keep quality high while reducing the scope of the work is the right choice. Putting in week after week at a punishing pace inevitably leads to poor quality.

For at least the key projects, find someone to actively participate in the role of the customer, be it a real customer, or more likely a product manager or a business development guru. Have them meet with the researchers and developers on a weekly basis to review progress and new ideas and plan the coming week or two.

Hire one or more experienced XP/agile practitioners into your lab to serve as mentors, with the goal of establishing these key practices: (1) pairing, (2) test-driven development, and (3) capturing ideas as "stories" with estimates. Working in short iterations is also important, but should come naturally. Be encouraging, but keep it relatively low-key and trust your staff to take advantage of the opportunity to adopt new ideas if they see them working.

When I look back on the projects I worked on as a researcher, the most regrettable aspect of the code I wrote is the lack of tests. If I could do one thing differently, I'd write tests.

Resources

If you're not familiar with agile methods or extreme programming (XP), here are some good places to start.

agilemanifesto.org
www.xprogramming.com/xpmag/whatisxp.htm
wikipedia on agile software development
wikipedia on extreme programming