Untyping

My Biased Guide to DEFUN 2009

2009-07-10T10:49:42Z

The DEFUN 2009 schedule has recently gone up. I haven’t seen many posts about it, so here’s my take on what looks good.

DEFUN is looking pretty good for Scheme, which is a pleasant surprise; it is well known that Scheme is not mainstream at ICFP. There are three awesome tutorials on the schedule, covering DSLs, web programming, and embedded programming respectively. I’m particularly excited by the DSL tutorial, given by Matthew Flatt and Eli Barzilay. It is my belief that PLT Scheme is the best platform available for developing DSLs, but not many people are aware of its full abilities. I hope this tutorial will go a way to getting the information out there.

Session A5(c) most intrigues me of the remaining tutorials. Ur/Web throws a whole pile of research into type systems at the problem of developing web applications. Through the power of dependent types Ur/Web claims to eliminate invalid HTML, invalid SQL, code-injection attacks, and bunch of other issues. I certainly want to better understand how it works, and perhaps see if we can apply some of the metaprogramming ideas to our own libraries.

If you’re interested in DEFUN you’ll probably also be interested in the CUFP 2009 schedule. I should be at both, as well as most of ICFP. If you’d like to chat, drop me an email.

Libraries update

2009-06-05T07:20:41Z

We produce a lot of open source code at Untyped, all of which is available from our Subversion repository. You can check out a read-only copy of our repository at any time. If you want to work on a branch without getting commit access to our repository, you might find git-svn or Mercurial’s Subversion integration useful. If you’d like to collaborate on development (and we’re open to all kinds of collaboration, including student/academic work) it is probably simplest to drop us an email (noel or dave at untyped dot com) to arrange things. There is a lot of stuff in there, so here’s a quick summary of the most interesting things we are currently working on.

Snooze is our flagship database abstraction layer, comparable to Hibernate or ActiveRecord. Snooze 2, which is out on PLaneT now, contains a robust query language and support for whole-model validation. Development on the version 3 of the library is underway, with emphasis on caching and inter-struct relationships.

Mirrors is our library for programmatic generation of XHTML and Javascript. It allows you to build blocks of code using a syntax similar to Scheme’s quoted lists. Rendering is done at compile-time as far as possible, so you get the convenience and compositional properties of quoted lists with the speed of PHP-style mechanisms. We intend to add support for CSS in a future release.

Delirium is our web UI testing library, similar to Selenium but with the expressive power of SchemeUnit. Versions 2 and 3, both on PLaneT, have equivalent feature sets: version 2 supports PLT 4.1.3 and earlier, while version 3 supports PLT 4.1.3 and upwards.

Smoke is our UI creation library (the partner library to Mirrors, pun definitely intended). While we have deployed this in a number of production applications, the interface is subject to constant tweaking so we haven’t published it to PLaneT. You can still get hold of the code from SVN and play with it, though!

Dispatch, our controller-to-URL mapping library, has been partly subsumed as its core features have been rolled into PLT’s web-server/dispatch library. There is still room for both systems, though, as the PLT library is built with simplicity in mind, whereas Dispatch was built to simplify web development. We plan on “rebooting” the Dispatch franchise with a version that wraps web-server/dispatch with some new features.

SchemeUnit, Noel’s excellent unit testing package, is as strong as ever. A version recently got rolled into PLT core, but you can still get the PLaneT package for the latest updates from Noel. Note that SchemeUnit’s code is hosted at Schematics.

Other points of interest:

Unlib is still going strong, with new shorthand require/provide syntaxes that can fetch stuff directly from SVN, a more humane version of keyword-apply, and some utilities to support dotted identifiers in our other libraries.
Excel, as its name suggests, is a library for creating Excel files in functional drawing style. It supports all the basics: formulae, inter-cell references, number formats, fonts, borders and fills, conditional formatting, and cell validation.
Autoplanet is a tool for deploying applications without having to worry about changing dependencies. It creates an application-local PLaneT cache and can be configured to download and install packages from PLaneT, SVN, or the local filesystem.

Flapjax: Second Batch

2009-04-17T15:53:52Z

Flapjax is the awesome functional reactive Javascript library from Brown PLT. We had a good experience with Flapjax some time ago, but in the interim it seemed that the project died. Turns out it was just hibernating. In the last few days Flapjax 2.0 has been released, along with a tech. report describing the system in more detail than the somewhat brief documentation.

To celebrate I coded up a small animation library for Flapjax. It’s hosted on Github, not our usual Subversion server as I wanted to gain a bit more experience with Git.

More State on the Web

2009-03-26T13:35:04Z

As a followup to The State of State on the Web I want to mention stateless servlets, a relatively new feature of the PLT web server that make continuations (even) more usable. Stateless servlets are essentially a kind of servlet with serializable continuations. A serialized continuation can then be stored on the hard disk, in the URL, in a cookie, or using any other mechanism you desire. This gets around the issue of memory consumption that is a concern with normal continuations. I don’t have a lot of experience with this kind of servlet, but Jay’s experience is that they are faster than normal servlets and the continuations are typically less than 100 bytes (and so can easily be encoded in a URL). Very nice!

The State of State on the Web

2009-03-20T13:25:25Z

There seems to be a miscomprehension that continuation based and RESTful web apps are mutually exclusive. Witness Nagare proudly proclaiming “no explicit URL routing / mapping … no global session object … no REST” as if continuation based frameworks were violently in opposition to these features. This is not the case. Fundamentally the issue is about managing state, and continuations, cookies, and friends are all approaches to solving the problem of encoding state over a stateless protocol. At Untyped we develop web apps that use a combination of continuations, RESTful URLs, and cookies for managing state and I believe this is the correct way to approach the problem. I hope this post will convince you of the merits of our approach.

Before looking at the tradeoffs of the different approaches I want to summarise continuations and their use in web applications. Simply put, the continuation of a program is what happens next. In the program (+ 5 (+ 2 1)) the continuation of (+ 2 1) is to evaluate (+ 5 []), where I’ve written [] to indicate the place where the value of (+ 2 1) goes¹. Now in Scheme we can capture a continuation, store it in a variable, and generally pass it around like any other value. This means we can effectively suspend a computation (by capturing a continuation) and then resume it at some time in the future (by invoking the continuation, which in Scheme appears as any a function application).

Now let’s look at what continuations do for web applications. A continuation-based framework associates a specific server state with a URL, which it does by capturing a continuation when a response is sent to a user. Everytime the user visits that URL they visit the same server state, invoking the captured continuation. As the user navigates around the site they build a history of server states that can be revisited using the back and forward buttons. This has several advantages. Firstly, if you don’t use mutation the back button will just work, because the user is just back to the same program state. Pretty neat. Furthermore, continuations give you procedure call semantics in your web app. Because a continuation is resumed when a URL is visited, to your program it appears as if the user’s request is the returned value of the function that sends your response. It’s as if you were using display and read on the web. This makes programming a lot simpler. For example, if you want to forward the user to a login page you just call the login page function, and it will return to the right place. No need to pass that page a URL to redirect the user to. This can be incredibly productive.

Now we’ve seen some of the advantages of continuations, we must consider the cases where the model falls down. There are two main issues: server load, and scope. Server load is simple. Every time you store a continuation on the server you use up some memory (RAM or disk space). At some point you have to reclaim that resource, so people may see “continuation expired” pages if they leave a long time between visits (though this is no worse that session expiry, which is quite common). Often a website has pages that are just displaying the results of simple queries to a database. These pages have no interesting state and using continuations in this case is wasteful of resources. Here RESTful approaches are appropriate, and we use them with, for example, the web server’s dispatchers.

Scope is another issue with continuation-based apps. Recall that continuation-based frameworks associate a particular URL, meaning a particular browser window (or tab), with a particular server state. There are some kinds of state that should be shared across all browser windows. Login information is a prevalent example. If I login to a site via one browser window, and then visit that site in another browser window I expect to already be logged in. This isn’t possible with continuations, as they are per window. Cookies, on the other hand, are per browser. So storing my login status in a cookie is the right thing to do.

In summary, RESTful approaches (URL routing, for example), cookies, and continuations are complementary and all have a place in web applications. Don’t think, for example, that is you use continuations you automatically reject everything RESTful! Finally, the Anton of Straaten addressed this issue from a different direction in his LL4 talk. Check it out for a different take on the problem.

1. Equivalently we could say the continuation of (+ 2 1) is (lambda (x) (+ 5 x)). This realisation is the key to continuation passing style, a program transformation useful in compilers and, perhaps surprisingly, AJAX web applications.

Untyped open source repository: open for business

2009-01-27T11:32:37Z

It’s been a long time coming, but we are proud to announce the launch of the Untyped open source repository!

This public Subversion repository houses the source code for our open source projects, including popular PLT Scheme packages such as Dispatch, Snooze and Mirrors as well as a few things you won’t find on PLaneT. It’s all free and open source, but please make sure you agree to our terms of use before you get stuck in.

You can check out any or all the projects using command line SVN. For example:

  svn co http://svn.untyped.com/mirrors/trunk mirrors-trunk

It’s no-frills at the moment - SVN and web browser support only - but we plan to prettify things in the future. See the README file for more information.

Weblog Updated

2009-01-03T16:09:09Z

In case you were wondering, we’ve moved our web presence to a new server (but still hosted by the most excellent Bytemark). The weblog was migrated/updated, and some things have moved around.

Our Feedburner RSS feed is the preferred way to keep up with the blog.

Questions on Scheme Web Development

2008-11-15T16:16:52Z

Ben Simon asks questions about web development using PLT Scheme. We answer!

[W]hat kind of server do I need to reliably run this puppy? Any Linux VM will do to start with. We use Bytemark. Amazon EC2 is another option. I recommend installing PLT from source; don’t rely on your distribution’s package to be up-to-date.
I wonder what kind of memory usage I’d want to plan for? It really depends on your application but as a guide we’ve run simple apps in 64MBs of memory.
I’d have to test out PostgreSQL or MySQL db support to make sure it was strong. PostgreSQL is solid, MySQL is not.
I’d have to sort out what the deployment cycle is like. Just copy over files and restart? Yes. Could I do hot deployment of some kind, by reloading scheme files (one of my favorite tricks in the book)? The web server does have some reloading functionality but we haven’t used it (no good reason; it just isn’t something we do).
What’s the best production web server arrangement. The PLT web server is solid, but we usually proxy through Apache so we can take advantage of Apache’s flexibility should we need it.

Recent changes in the PLT web server

2008-11-11T16:45:44Z

Jay McCarthy, maintainer of the PLT web server, has started blogging about improvements he is making to the web server. Start reading here and go back through the last six or so posts. It is great to see the web server getting more visibility.

Tests as todos

2008-11-05T21:44:14Z

Like most people I have a few projects on the go at once. To efficiently switch between them I must be able to quickly pick up where I left off. In my programming projects I’ve been using failing tests as reminders to myself. This fits in nicely with my programming workflow, and enables me to make progress before I’ve recalled all the details of the project I’m working on. Here’s how it works:

In my programming workflow I cycle between writing tests, writing code, and running tests (this is just test driven development). When I’m about to stop working on a project I write some failing tests, which act as a specification for what I should do next. At this point in time I’ve been working on the project for a while so I have recalled its structure and I’m in a good position to make this decision.

When I pick up a project after a break I enter straight into my normal workflow and run my tests. I inspect the failing tests and start implementing the functionality they specify. At this point in time I don’t even have to remember why I’m implementing this; the tests provide enough detail that I can just start coding. As I do so I invariably recall more details of the project. By the time I’ve finished the feature I’m ready to go at full speed.

This technique allows me to “hide” the time it takes to recall the project details; I still get useful work done in this period. It’s quite a simple idea and no doubt some of you are already using it, but if you haven’t tried it, give it a shot.