Javarants

Mustache is logic-less but the logic has to go somewhere

admin — Sat, 09 Mar 2013 23:21:05 +0000

As the developer of mustache.java I get a lot of feature requests to break the mustache language and add real logic to the templates. In virtually every case the reason the developer wanted the functionality was because they were using mustache with a template backed with a model without an interposed view. Let me give you a common example. Developer has a user object with a few fields:

class User {
  String username;
  long createdAt;
}

And they want to show when the user joined in their template:

{{#user}}{{username}} joined on {{createdAt}}{{/user}}

They look at this and think, “I need to be able to format longs into dates in the template”. If you were in .erb or .jsp you might create a helper to do that and call it directly from the template. For mustache, someone will typically suggest changing the language to add filters so you might write the template like this:

{{#user}}{{username}} joined on {{createdAt | date}}{{/user}}

Then there would be a bunch of date formatters you could use, you might even go so far as to ask to parametrize it in the template because it isn’t flexible enough with just a name:

{{#user}}
  {{username}} joined on {{createdAt | date("DD/mm/YYYY")}}
{{/user}}

And you start going down this road where more and more of the view logic is going directly into your template making it hard to internationalize, harder to share between mustache implementations and generally making it harder to maintain and test. However, this isn’t the way you should use mustache. The best way is to add an additional layer of logic in your chosen host language that is responsible for the view. So for this case, what you would like end up with is something like this view:

class UserView {
  String username;
  String joined_date;
  UserView(User user) {
    username = user.username;
    joined_date = new SimpleDateFormat("DD/mm/YYYY")
        .format(new Date(user.createdAt));
  }
}

with a template much like the first one:

{{#user}}{{username}} joined on {{joined_date}}{{/user}}

This allows you in the future to add special internationalization, user specific date formats, maybe a relative timestamp, etc. All without ever changing the template. It is also easier to test since you can independently check that the formatting of the date is correct without having to execute it within the context of the template text.

This strategy worked great at Bagcheck where there was only myself and a designer and it works great at Twitter where we have tons of engineers and designers working in the code base. It has also made it possible to move between rendering on the client in Javascript to rendering on the server in Ruby to rendering on the server in Scala all with the same set of templates and basically standard mustache implementations.

What is new and interesting in JDK 7?

Sam Pullara — Mon, 10 Sep 2012 02:12:03 +0000

I wanted to document the most interesting features of JDK 7 here, for my own gratification and to make a place where you can contribute your great finds:

Invokedynamic bytecode for doing runtime binding of callsites
Named regular expression groups
Try with resources / AutoCloseable
MethodHandles
BufferPoolMXBean — where did all my memory go?
SocketOption, ProtocolFamily and NetworkInterface
Async IO — including filesystem notifications and a new network stack
ForkJoinPool
TransferQueue
Precise exception rethrowing, catching more than one exception type
Switch on String
Much better GZIP support
Global logger and other log management stuff
ThreadLocalRandom
Bitset.valueOf() for arrays
Paths for manipulating file paths
Files for copying files and streams, etc. Swiss army knife of stuff
File attributes, ACLs, etc.
Long.compare — probably caused a lot of bugs
Tons of Locale / Character updates

Things I found that are old but new to me:

Scanner – using all regexes, probably horribly inefficient
PriorityQueue, Deque
Atomic*FieldUpdaters

Am I missing anything?

New features and extensions in Mustache.java

Sam Pullara — Sun, 09 Oct 2011 23:29:31 +0000

Almost a year and a half ago I talked about my implementation of mustache, which I call mustache.java. Since then, we used it to develop Bagcheck and it has also been adopted for a variety of use cases by Netflix, Twitter and others. In my post I mention that for all intents and purposes, the template engine was ‘finished’ and I would say that for the most part, that was accurate. However, there have been a couple of things that have been done since then around optimization and also some interesting extensions that I would like to share with people to get feedback.

First though, I will shortly recap the biggest differentiating feature of this implementation over other implementations:

Concurrent, streaming template evaluation

Mustache.java exectutes the template in such a way that partials, loops and even replacement (if you use futures) do not block the further evaluation of the template. This allows you to quickly launch all your I/O (or CPU) bound work for the whole template in context during a quick evaluation and then the engine can do things like flush to the underlying writer incrementally as work completes giving you great client-side and server-side concurrency. You can also use this to defer sections of the page to write them out later, a la Big Pipe. This has been invaluable to easily maintaining Bagcheck’s low-latency design. The downside to this is that you need to write your Java code in a functional style, I suggest that you use Google collections to do it right. The design of this has remained unchanged since the initial release though recently I added the ability for the template backing code writer to take control of concurrency in order to reduce context switches and garbage creation. This can get you a 5x improvement in template evaluation performance at the cost of developer time and expertise without losing the ability to do concurrent evaluation when necessary.

Since the first release, the handlebar server has been included to allow you to easily view mustache templates fully rendered in the browser with only JSON data as your backing code. This lets you quickly iterate on design without having to have the full stack running with enough information for someone to figure what needs to be implemented on the backend when new things are added. Sometimes though it can be painful to get realistic data for new features using production data. In those cases, I had always imagined a method to take a template and rendered content and give you back data would generate that same content in combination with the template. That functionality is now in the master branch (since 0.6-SNAPSHOT). Given any Mustache instance you can now call unexecute(content) on that instance and if it isn’t ambiguous or in some other way irreversible you will get back a Scope object that contains the data. Using toJSON() on that object will provide mock data that can be used with handlebar. Here is an example:

We then take the template and the output and unexecute them using the new method on Mustache objects:

This produces the following JSON (after pretty printing):

What it really does is turn the Mustache language into a look ahead grammar that can be used to read data from content, though that content has to be stringently authored. Another use case that seems reasonable to me is for people to check that their complicated backend code is combining in the right way with their template. Something like an integration test for template processing without having to hard code the non-template interpolated text into your test. Would love to get feedback on this feature as it is still unclear whether it will be useful to a wide audience.

The most recent feature added is support for Twitter-style functions (_ vs just #) and i18n support. Long ago I had added support for # functions: you can return a Google Guava Function from the value callback and after the engine executes the enclosed template it will call your function to transform the value. This works for simple i18n and caching but fails for more complicated i18n applications. In those cases, you may want to also change the template itself which means you will need a callback before the underlying template is executed. If you need that behavior, you instead return an instance of TemplateFunction which gets called back before evaluation and then compiles the template text that you return from the function invocation. This is most often used to change the order of words or phrases that include things like user names. Here is a simple example from the tests:

The biggest change since the release though is the addition of the MustacheBuilder. In 0.6 I will be deprecating MustacheCompiler as not only does the Builder not require the use of the Java compiler, but it also compiles almost instantly and performs better than the Java compiled versions of the templates. The Builder also gives us a lot more runtime flexibility that I am going to take advantage of over time for a variety of optimizations that are in the queue. I may return to investigate using ASM for compilation though in order to take advantage of JDK 7 and invokedynamic as that becomes the standard JVM in deployment.

You may not be familiar with some of the things that you can enabled while it is running to make debugging and profiling easier. There are 3 system properties that can be turned on with little documentation:

mustache.debug: output a warning when a callback is absent or null
mustache.trace: create a multithreaded trace, use MustacheTrace.setUniqueId() in each thread, output with MustacheTrace.toASCII()
mustache.profile: time all callbacks, output the top 10 total + average with Scope.report() and then reset the profiler

Now is the time for input. Here are some things that I have on my TODO list:

Full JDK7 invokedynamic support with ASM generated classes
Extensible templates a la Django templates
Native Scala support based on Twitter’s util-core
Generate scaffolding in a variety of languages for backing code with verification

Would love to hear from users and prospects. You can either comment here or join the dicsussions on the Mustache.java group.

Using Closures, Method Handles and Extension Methods in Java 8 (JSR-335)

Sam Pullara — Sun, 23 Jan 2011 01:57:53 +0000

Update: A bunch of code from hotspot was just integrated into the bsd-port and some of it breaks the build. Use my distribution at the end of this blog post if you want to test the features.

In December JSR-335 was approved by the JCP and is now starting. I’ve joined as member of the expert group, but like with most recent JSRs all the real discussion will take place on the public mailing list. A lot of work has already been done in OpenJDK to support the features of this JSR which will ultimately be included in Java 8 if it completes successfully. Here is the current state of the JSR. Though JDK 8 is when official support would ship, you can already build and execute the draft features of JSR-335 by using the regular OpenJDK 7 build along with some compile-time and run-time tools. Before I go into the guts of how to build them, here are some examples of what you will be able to do:

Create closures that are converted to Single Abstract Method (SAM) implementations when used in context
Create interfaces that specify default behavior for methods that are unimplemented by the implementing class
Reference methods in source and then convert those references to SAM implementations when used in context

Building OpenJDK 7 on Mac OS X 10.6 (based on Darwin10Build):

Install mecurial:
sudo port install mercurial
Install hg-forest:
sudo port install hg-forest
Install soylatte to /usr/local/soylatte16-i386-1.0.3
Clone bsd-port:
hg fclone http://hg.openjdk.java.net/bsd-port/bsd-port
export OPENJDK_ROOT=`pwd`
Grab build script:
cd bsd-port; curl -O "https://gist.github.com/raw/617451/d4862a41b07a1196d5149efc1c40406f8bb77dc1/update.sh"
mkdir ALT_COMPILER_PATH;cd ALT_COMPILER_PATH; ln -s /usr/bin .SOURCE;ln -s .SOURCE/g++-4.0 g++; ln -s .SOURCE/gcc-4.0 gcc;cd ..
unset JAVA_HOME
Edit hotspot/make/bsd/makefiles/defs.make and change ifeq ($(ARCH), amd64) to ifeq($(ARCH), x86_64)
Update sources and build (on a fast machine will take 15m):
sh update.sh

Now you should have a fully working OpenJDK 7 64-bit distribution:
macpro:bsd-port sam$ ./build/bsd-amd64/j2sdk-image/bin/java -version openjdk version "1.7.0-internal" OpenJDK Runtime Environment (build 1.7.0-internal-sam_2011_01_22_16_40-b00) OpenJDK 64-Bit Server VM (build 20.0-b03, mixed mode)
Now we want to build the Lambda compiler and runtime support:

Grab the source:
cd $OPENJDK_ROOT; hg clone http://hg.openjdk.java.net/lambda/lambda/langtools
Build the tools:
cd $OPENJDK_ROOT/langtools; ant -Dboot.java.home=../bsd-port/build/bsd-amd64/j2sdk-image -f make/build.xml

OpenJDK 7 doesn’t actually support everything that we need to run code generated for Lambda though. We need to also build a java agent that will modify the bytecodes in classes as they are loaded to run on the older VM for some features. Here is how you do that:

Grab the source:
cd $OPENJDK_ROOT; svn checkout http://jsr335-lambda.googlecode.com/svn/trunk/ jsr335-lambda
export JAVA_HOME=$OPENJDK_ROOT/bsd-port/build/bsd-amd64/j2sdk-image
Build the agent:
cd jsr335-lambda/agent; ant jar
export AGENTJAR=$OPENJDK_ROOT/jsr335-lambda/agent/lib/jsr335-agent.jar

Now we can actually build Lambda code and execute it, with a lot of command line options. For example, here is a simple program that we might want to execute:

public class Test {
  public static void main(String[] args) throws InterruptedException {
    Thread t = new Thread(#{ System.out.println("ran"); });
    t.start();
    t.join();
  }
}

Let’s first compile it:

export PATH=$JAVA_HOME/bin:$PATH
export LANGTOOLS_JAR=$OPENJDK_ROOT/langtools/dist/lib/classes.jar
javac -J-Xbootclasspath/p:$LANGTOOLS_JAR -source 8 -target 8 Test.java

Now we have Test.class file that will only work in a VM that supports some of the bytecode that we are using. In this case we don’t need the transformations so we can execute this code like this:
macpro:jsr335 sam$ java -cp . Test ran
Here is some code that will not run out of the box, that shows you how you can now write extension methods:

public class Test2 {
  interface Trait {
    int inc(int i) default Test2.inc;
  }
  static int inc(Trait t, int i) { return i+1;}
  public static class Test3 implements Trait {}
  public static void main(String[] args) {
    Trait t = new Test3();
    System.out.println(t.inc(1));
  }
}

If you try and run this without the agent you will get this:
macpro:jsr335 sam$ java -cp . Test2 Exception in thread "main" java.lang.AbstractMethodError: Test2$Test3.inc(I)I at Test2.main(Test2.java:9)
Now we execute with the agent enabled:
macpro:jsr335 sam$ java -javaagent:$AGENTJAR -cp . Test2 2
The last feature that you might want to use, are method handles. Essentially you can convert a method handle into a closure by using it in a SAM context. Here is an example of how they work:

public class Test3 {
    public void test() {
	System.out.println("Test");
    }
    public static void main(String[] args) {
	Test3 t = new Test3();
	Runnable r = t#test;
	r.run();
    }
}

In order to run this we need to enable InvokeDynamic on our VM and include the classes from langtools for some runtime support:
macpro:jsr335 sam$ java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic -Xbootclasspath/p:$LANGTOOLS_JAR -javaagent:$AGENTJAR -cp . Test3 Test
That command line has everything you should ever need to execute code generated using the Lamba enhancements so using anything else is probably unnecessary. Now let’s put all of this together into the final example from the Lambda proposal itself:

import java.util.*;

interface Sortable<T, U extends Comparable<? super U>>  extends List<T>  {
   void sortBy(Extractor<? super T, ? extends U>  e) default Impl.sortBy;
   static class Impl {
       public static<T, U extends Comparable<? super U>>
                 void sortBy(Sortable<T, U>  sortable, final Extractor<? super T, ? extends U>  e) {
           Collections.sort(sortable, #{T a, T b ->  e.extract(a).compareTo(e.extract(b))});
       }
   }
}
interface Extractor<T,U> {
    public U extract(T t);
}

class Person {
    public Person(String firstName, String lastName) {
        this.firstName = firstName;
        this.lastName = lastName;
    }
    private String lastName;
    public String getLastName() { return lastName; }
    private String firstName;
    public String getFirstName() { return firstName;}
    public String toString() {
        return firstName + " " + lastName;
    }
}

public class Test4 {

    public static class SortableList<T, U extends Comparable<? super U>> extends ArrayList<T> implements Sortable<T, U> {}

    public static void main(String[] args) {
        Sortable<Person, String> list = new SortableList<>();
        list.add(new Person("Sam", "Pullara"));
        list.add(new Person("Brian", "Goetz"));
        list.add(new Person("Bob", "Lee"));
        list.sortBy(Person#getLastName);
        System.out.println(list);
    }
}

Which results in:
macpro:jsr335 sam$ java Test4 [Brian Goetz, Bob Lee, Sam Pullara]
The punchline is that if you read through all of this you get a bonus: here is the distribution of above build products that you can download and run on your Mac (assuming you are running a 64-bit build of snow leopard):

openjdk7-lambda-macosx-x86_64-01222010.zip

Testing out Percona and HandlerSocket with AvroBase

Sam Pullara — Mon, 20 Dec 2010 21:34:00 +0000

The great thing about the AvroBase interface is that it can be used with many different datastores because of its relatively low requirements on those stores. HandlerSocket is a plugin for MySQL that gives you direct access to the low level APIs that let you do the operations that AvroBase requires without going through the high-level SQL APIs and therefore skip a lot of boilerplate parsing that happens for every single lookup or scan. You can read about the HandlerSocket benchmarks here. Percona Server is just a better implementation of MySQL with its own modified InnoDB engine called XtraDB. In combination you should be able to get even higher numbers of transactions per second for individual servers in a cluster. Building it on the Mac (or any unix system) is pretty straight-forward.

cd /Users/sam/Software/Percona-Server-5.1.53/ ./configure --prefix=/Users/sam/usr make && make install
Similarly, download the HandlerSocket source via Git and then build it:
git clone https://github.com/ahiguti/HandlerSocket-Plugin-for-MySQL.git cd HandlerSocket-Plugin-for-MySQL/ sh autogen.sh ./configure --prefix=/users/sam/usr --with-mysql-source=/Users/sam/Software/Percona-Server-5.1.53 --with-mysql-bindir=/users/sam/usr/bin --with-mysql-plugindir=/users/sam/usr/lib/mysql/plugin make && make install
You then need to make some changes to the my.cnf file:
[mysqld] user = sam port = 3406 loose_handlersocket_port = 9998 loose_handlersocket_port_wr = 9999 loose_handlersocket_threads = 16 loose_handlersocket_threads_wr = 1
Start up mysql for the first time:
export PATH=~/usr/bin:$PATH mysql_install_db cd /Users/sam/usr ; /Users/sam/usr/bin/mysqld_safe &
Then connect to mysql and install the plugin:
mysql -u root install plugin handlersocket soname 'handlersocket.so';
For AvroBase I am using the native i/o Java client:
svn checkout http://hs4j.googlecode.com/svn/trunk/ hs4j mvn install -Dmaven.test.skip=true
Haven’t fully put it through its paces but I expect to do some benchmarking here shortly. Because of the amount of work to deserialize objects vs the database access my guess is that it won’t make a huge different in terms of qps for a single front-end. However, it will likely let us go to a much higher number of frontends before accessing the database becomes the bottleneck.

What does the Facebook security thing mean to users?

Sam Pullara — Tue, 19 Oct 2010 17:53:37 +0000

You might have read a lot about the Facebook issue over the last couple of days. The technical description of this is pretty complicated but the implications are pretty clear. When you use applications on Facebook and they use a technology called ‘iframes’ to host their application (like Farmville) and then partner with other companies for advertising it is possible that your UID gets passed all the way from Facebook to the third-party advertising sites. This happens because of the details of how browsers work with referrers, third party javascript includes and iframes. At that point they could take that UID and set a cookie on your browser so that whenever they see you, wherever you are on the web, they know that it is you and can lookup your public Facebook information and use that information to target ads, customize content, etc.

People can argue that this information is public, however, the fact that the information is tied to your browser and can be accessed anywhere that the advertiser has an ad makes it especially powerful. For example, I could use the information to create advertisements on another website (not Facebook) that includes photos of you and your friends. I could address you by name in the ad, use the names of your parents or children, whatever you might have made available on Facebook to the world at large. My Facebook account is pretty tied down, however there is a certain base level of publicness about Facebook that makes the possibility of creating very creepy, misleading ads across the internet.

Here is what I can discover about myself using only my UID and publicly accessible URLs via the Facebook Open Graph API:

Name: Sam Pullara
First Name: Sam
Last Name: Pullara
Link: http://www.facebook.com/spullara
Gender: Male
Locale: en_US
Picture:

If you want to do the same and see what is available for you, start here https://graph.facebook.com/[your uid]?metadata=1 and try each of the links in the connections subsection to see what is easily available.

Using web scraping, I can discover even more information about me:

It is a good thing for Facebook to not allow this kind of UID transfer to third parties with whom you have no connection, especially when cookies are involved. Other large scale internet services go to great lengths to avoid this kind of data transfer. Facebook should fix the URLs so it isn’t inadvertently transferred from their application partners to the application partner’s advertisers.

HAvroBase: a searchable, evolvable entity store on top of HBase and Solr

Sam Pullara — Wed, 30 Jun 2010 19:55:44 +0000

Building out a social consumer internet product that could change quickly and evolve over time puts special requirements on the underlying data store. You need to be prepared for scale but not investing too much too early, your business may need to pivot in different directions so data models can’t be set in stone and you need to be able to search that data to enable many of the features users expect from an online social product. I have the additional requirement that I wanted all the entities in the system, regardless of where they are stored, to be accessed the same way and be described by the same data description language for consistency and maintainability. I’ve created what I think to be a novel solution to these requirements in the form of HAvroBase.

The first choice you have to make against these requirements is which data definition language are you going to use? Instead of depending on the native format of the storage system I’ve decided to use Avro. Similar to Protocol Buffers and Thrift, Avro lets you define your entities using schemas and store them efficiently in a binary format. Additionally, as long as you have the original schema available you can load old data into a new schema. This will let you evolve your stored rows lazily and not have to completely update your storage system when a new field is added or a field is removed.

Whereas the data definition choice is basically commodity at this point and your choice can be somewhat arbitrary, the choice of storage technology will likely be something that has more trade-offs to consider. After looking at the features and communities of a variety of projects including HBase, Cassandra, Redis, Riak, MySQL, Membase, Terracotta, etc. I finally chose HBase for a few reasons that may not be that important to you. First I settled on a BigTable type choice based on the data model. That left HBase and Cassandra as contenders. There also a few things I think are advantages:

multiple tables
better ordered key support
Zookeeper and HDFS were already in my solution
Hive and Hadoop support against the native data format
consistency
compare and set
atomic increment
versioning

Cassandra definitely has its own advantages that didn’t out-weigh the other considerations including

performance
simple deployment
always writable

You might make different tradeoffs or even use both solutions for different problems. I’m also biased somewhat as I am sharing an office with Cloudera and get top notch support at a moments notice.

Whether I chose HBase or Cassandra the implementation would have been much the same and with the HAvroBase framework you can have multiple independent storage systems. The framework does somewhat shield you from this decision, at least in your code.

When it comes to text search you really don’t get better than Lucene in open source and the features that Solr builds on top of Lucene make it even better. I don’t think there is reasonable argument for using something besides Solr at this point. Especially with their support for sharding and replication that comes with Solr Cloud. It also has the nice benefit that it supports multiple tables, like HBase, so I can efficiently separate entities and allow them to scale independently without managing an additional system.

The last choice that I made was runtime configuration. There are a few solutions including Spring, Guice and rolling my own and I finally landed on the side of Guice. I just like typed, programmatic configuration better than XML files and since that is the focus of Guice, I think it is the best solution for that.

The first thing that you need to do to use the system is to define an entity in Avro:

You can use the Avro compiler (or the Maven plugin) to convert that definition to classes (HAvroBase requires this). Then you do the typical things in code to setup your HBase connection but within a Guice module and set some of the configuration parameters for HAB:

That configuration can then be used to instantiate an instance of the HAB class that implements the connection to HBase like this:

You’ll notice that configuration is somewhat split between things that the base system needs to know and things that the HAB needs to know. For example, what table the schema is store in isn’t relevant to the base system as an implementer is free to store schemas however they like. But the assumption is that every system will have the concept of a table and a column family in order to separate entities in the underlying storage system. The Solr support is pretty elegant I think. Rather than specifying in more than one place which fields should be indexed, instead at runtime it queries the Solr core’s schema.xml that defines them and then when you put an entity in the system it automatically indexes those fields. Here is an example of using some of the APIs:

This example creates a new User, puts it into the HAvroBase, looks it up by row and then searches for it by field that is indexed and finally deletes it. In any fully implemented AvroBase system this should work identically. You’ll notice in the memcached implementation that only put/get/delete are implemented and it doesn’t support scan or search at all. I’ve considered breaking out that functionality into separate interfaces but haven’t done that yet.

If you are unhappy with the way the default Avro compiler generates code you can use my templated Avro compiler that lets you change the generated code. I have done that for my project to make them a little more developer friendly.

Things left to do:

implement guarantees around indexing
optimize schema keys (use an abbrev table instead of a hex sha-256 key)
indexing nested fields and arrays in Solr

I’m sure there are more things left undone but those are some of the more obvious issues.

Static-typing is a powerful metadata database, exploit it!

admin — Thu, 17 Jun 2010 00:20:47 +0000

Today someone decided to pretend they knew something about how a modern statically typed language developer works. Perhaps they are big emacs fans or something because they felt that static-types leading to autocomplete in an IDE was somehow a feature in the language FOR THE IDE. The IDE gets no benefit, in fact, editors like emacs and TextMate are much simpler. But I can tell you that the developer gets a tremendous benefit by having this metadata database not only present in the code and easily queryable but heavily exploited by her tools. What is amazing about this metadata database is that it is actually populated by that same IDE. There are very few times that I even type a Type — 90% of those times are when declaring a new Type — the rest of the time the IDE is very patiently maintaining and leveraging that database for me. Even better, when someone else is looking at the source code, or even using the library, they instantly get 80% of the documentation. All that is left are the semantics of the calls. Perhaps we need to have some sort of meetup where we look over each others shoulders and actually understand how someone who is on the other side really works rather than dynamic-typing language people assuming that the programmer is maintaining the huge metadata database present in the static-typing language developers code. And on the otherside that static-typers are assuming that the dynamic-typing language developer is memorizing all kinds of arcane method call names, argument shapes and the shapes of all the libraries they are using. Do you really memorize all that trivia?

Building the ideal web application template engine

admin — Wed, 16 Jun 2010 18:02:03 +0000

A month and a half ago I had put out a call for what I was calling the ‘ideal web application template engine‘ along with a list of requirements that I thought would be present in such a system. Since then I looked a bunch of them and decided that I like the simple markup that mustache defined but none of the implementations were up to doing what I wanted. This led me to embark on building a new engine with for that markup for my chosen platform, Java, which I called creatively, mustache.java. Though it claims to be ‘logic-less’ I would say that it has some amount of logic. It will loop over a set of objects and it will check booleans, but I think it is about as close to logic-less as you would want to be in a template language. So, let’s look at each of the requirements and how I ended up implementing them:

Works well with HTML5/CSS3 progressive enhancement

This one is straight-forward. Because I put no limitations on the generated text you can actually generate almost anything with mustach.java, in fact, I am using it in another project to generate Java source for Avro objects. It is really overkill for that project but it is so easy to use that I used it anyway.

Allows mock data within the template that is replaced at runtime
Client-side version that leverages the mock data for shift-reload debugging

These I implemented somewhat differently from the requirement because the mustache template language is not tag based. I have actually thought about whether it might make sense to make it tag-based and get the full requirement, but I digress. What I did get to is a system that is easy for a frontend developer or designer to use without having the whole system running on their machine. Included with mustache.java is a mini-server (called handlebar, thanks for the name Luke) that combines mock json data with mustache.java templates. This lets them view a fully rendered version of the page they are working on with the required templating information that will be used in production but without running a big production system. In fact, because mustache.java dynamically generates and compiles Java code, they can fire up handlebar in directory along with their mock data and just keep reloading the page to see the effect of changes they make.

Composable components, not monolithic pages
Very little or no business logic in the templates

The choice of mustache as a base template language ensured that this would be the case. Including a partial within a template is easy and makes pages very easy to compose. I still miss the Django template style extends/block system and may extend mustache to support it. Having to remember to include things like the header and footer on every page doesn’t seem necessary. As I said above, I think that mustache has the minimum reasonable logic.

Concurrent evaluation possible

This one was very important to me from a performance perspective. My implementation makes every effort to be as low latency as possible for the end user. At every possible point in template evaluation things are actually executed in separate threads or purely asynchronously in the case of external I/O like HTTP requests to other services. The one last thing I am going to do here is to allow not only a straight-line evaluation of the template but the possibility of specifying a timeout so that you can return different markup if evaluation doesn’t return in time. It may be unnecessary to have this in the base template engine though, still looking at the design of the feature.

So how easy is using the template system? I think I can answer that question by looking at the relevant portions of the implementation of handlebar:

final MustacheCompiler mc = new MustacheCompiler(new File("."));
....
Mustache mustache = mc.compile(new BufferedReader(new FileReader(filename)));
FutureWriter fw = new FutureWriter(res.getWriter());
File file = new File(mocks, base + ".json");
if (file.exists()) {
  BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file));
  JsonParser parser = jf.createJsonParser(bis);
  JsonNode json = parser.readValueAsTree();
  mustache.execute(fw, new Scope(json));
} else {
  mustache.execute(fw, new Scope());
}
fw.flush();

The most interesting part of this is the FutureWriter. The future writer lets mustache write evaluated templates completed out of order in template order. The scope is essentially a sophisticated context object that knows how to pull data from Java objects of various types, including raw object fields and methods, Maps and Json objects. Conveniently, you can also store values in it directly. There are some things I would like to extend at some point, however, for all practical purposes the template system is finished.

Android Dalvik VM performance is a threat to the iPhone

Sam Pullara — Wed, 26 May 2010 16:29:57 +0000

One of the peculiarities of Apple is that they have set themselves down a path where every Apple developer needs to learn Objective-C (and C/C++) to build applications for their platform. The biggest characteristic of Objective-C vs Java is dynamic dispatch. At runtime Objective-C can send arbitrary messages to objects and they may or may not respond to them. This has the nice property that you can write code that is very dynamic and loosely bound but it also has the property that method calls in Objective-C are very slow and the more code that you write in Objective-C instead of in C/C++ the slower your codebase becomes. Up until Android 2.2 (Froyo) the JVM (really a Dalvik JVM for licensing reasons) on the Android platform was playing with one hand tied behind its back. Different from desktop/server Java, the JVM was still an interpreter, like the original JVM back in the Java 1.0 days. It was very efficient interpreter but an interpreter none-the-less and was not creating native code from the Dalvik bytecodes that it uses. As of Android 2.2 they have added a JIT, a just-in-time compiler, to the stack that translates the Dalvik bytecode into much more efficient machine code much like a C/C++ compiler. You can see the results of this in the benchmarks of Froyo which show a 2-5x improvement. As they add more and more JIT and GC features that have appeared in HotSpot, JRockit, etc, you will likely see even more improvements over time — without having to change or recompile the 3rd party developed software.

This wouldn’t be that big a deal if Android software wasn’t already approaching the speed of the Apple iPhone even when running its applications through the interpreter (see the HTC EVO 4G running 2.1). This is likely going to mean that 3rd party developer applications created for Android, running on the same hardware, is going to be faster than the same code written against the Objective-C libraries that Apple provides for the iPhone. You’ll be able to get more done, have smoother user interfaces and all around build more powerful applications easier. A better experience for the user and the developer on Android is a bad thing for Apple.

The final issue that Apple has is that far more people know and need to know Java than Objective-C. Except for their platform, there is no need to ever learn it. The tool chain is much more robust, information is much easier to come by, garbage collection is available, and there are thousands more libraries written in Java than Objective-C and far more portable libraries than are written for C/C++. You might argue that you can always drop down to C/C++ code to make up for the lost performance of using Objective-C (at a significant cost development-wise). You can do that, except places where you need to make a call into Apple’s Objective-C runtime libraries or where you want to write callbacks from those libraries. Those are mostly Objective-C libraries and use the dynamic dispatch mechanism which I showed in 2004 was very much slower than Java. I’d really love to see the whole computer language shootout written with Objective-C calling conventions just to show how much slower it is at that level.

Obviously performance is not the only consideration, but it is a big one. Android has other issues like a fragmented operating system base, hardware feature base and a more complicated user experience. All those things will conspire to hold it back but I think we can see the writing on the wall that Android is going to dominate iPhone market-share wise which will eventually make it a more attractive platform business-wise.

Update: Just so people don’t get the wrong idea, I am a rabid iPhone user and develop for the iPhone first. This blog entry is to call out a threat that Apple should take seriously.

Update 2: Alright, you win. Don’t worry about competing with Google & Android on performance. It probably doesn’t matter that much for the user experience. Especially don’t fix obj_msgSend() with a JIT/LLVM, that would be crazy.