Soft Shake is an IT conference being held in Geneva on October 3rd and 4th and I’ll be speaking about data grids and data caching with Infinispan on the 3rd. In fact, I’ll be speaking twice! At 1pm you’ll see me doing an introduction to data grids and data caching, and at 5pm I’ll be delving into the data grid vs database debate.

So, if any of this topics interest you, come and join us at Soft Shake! It’s gonna be fun :)

And that is not all! I’ll be one of the Infinispan team members speaking in JUDCon London at the end of October. The agenda is now live and you’ll see me talking about near caching on the 31st of October. Don’t miss it!

Cheers,
Galder

1. First of all, the script kicks off the .sh script and keeps track of the script process:
   ^?View Code PYTHON

   def start_local(self): self.target_dir = "/tmp" self._unzip(self.target_dir) self.p = self._start("infinispan/test_local_config.xml", self.target_dir)

   ^?View Code PYTHON

   def _start(self, config, target_dir): script = "%s/infinispan-%s/bin/startServer.sh" % (target_dir, VERSION) os.chmod(script, 0755) p = Popen([script, "-r", "hotrod", "-c", config]) return p

2. The problem with this type of scenarios where you have a Java process started from a script is that Python, and probably any other scripting language out there, will give you the PID of the script process but not the Java process. So, when it comes to stopping the Java process, we need to find the PID of the underlying Java process, for example:
   ^?View Code PYTHON

   def _find_pid(self, shell_pid): ps = Popen(["ps", "-ef"], stdout=PIPE) out, err = ps.communicate() lines = out.split("\n") java_proc = filter(lambda l: l.find(str(shell_pid)) > -1 and l.find("java") > -1, lines) return java_proc[0].split(' ')[3]

   This function takes the PID of shell script and tries to find the Java process whose parent process is the script process.

3. Once the Java PID is know, things are pretty straightforward, just kill it:
   ^?View Code PYTHON

   def stop(self): java_pid = self._find_pid(self.p.pid) kill = Popen(["kill", "-2", java_pid.strip()], stdout=PIPE, stderr=PIPE) out, err = kill.communicate()

And that’s it :) – The code that I originally wrote did the same but with with a few more process open to do grepping and cutting the output. The final version that you can see in its entirety here and whose code snippets are shown uses Python functions as much as possible to reduce the number of system calls.

IntelliJ differentiates between test and main folders

This is in my opinion one of the biggest advantages of IntelliJ, because it means I can search for references to a class/method within the testing scope or within the production code. Why is this useful? Let me give you two examples:

• Imagine you want to find out if a method is actually used in production code. Sure, it might be used 1000 times in testing, but is it actually used in production code? Go and figure out from those 1000 references whether any comes from production code.
• Imagine you want to figure out how to test a particular class. With IntelliJ, you can find references to that class only within the test code, giving you a clear view of how this class might get tested. With Eclipse, if the class is used plenty of times in the production code, separating between testing and production code can be a PITA.

IntelliJ supports running individual test methods better

Eclipse used to drive me nuts whenever I want to execute a particular test method. First of all, it wasn’t intuitive, you had to go to the class overview panel to be able to do so. On the contrary, with IntelliJ you can do so directly from the class code which is much easier. But the most annoying thing about Eclipse was when you wanted to run individual test methods that are defined in super classes. Eclipse could not do this. If you went to parent class and selected the method, it would try to instantiate the parent class but not the child. IntelliJ does this very well, allowing to select the class and then the test method in the run configuration. Eclipse, at least a year ago, was not able to do so.

IntelliJ is the king of refactoring

Refactoring in IntelliJ is taken to limits where Eclipse has never reached. For example with Eclipse, whenever I renamed a test class, I always had to go and change the run configuration. Not any more with IntelliJ.

IntelliJ can shelve changes

A year ago I was not fully using Git yet, and so finding the ability of shelve changes on Subversion projects was really useful. Eclipse could not do this. With Git now, git-stash already does this job, so this feature is not as important this days if you’re coding in Git.

With IntelliJ you can define default VM parameters for all tests

In Eclipse, this is not possible. The only thing you can really do is define default parameters at JRE level, which can be too far reaching if you’re running different type of applications from the IDE.

IntelliJ provides plugins via a central repository

IntelliJ has always had a single place from which you install plugins, which makes it nice, easy and quick to install plugins. This might be a bit outdated now but with Eclipse, in the past at least, you had to go and set up individual plugin update sites in order to install plugins, and this used to be quite a PITA.

The first of those mixes is called ‘What Makes Elmo Tic?’ and It’s full of tracks that I simply can’t stop listening, in particular, Tensnake’s Battlehymn For Children remix, Vitalic’s Poison Lips and of course, Ossie’s (huge!!!) Holly Land. For those of you who are fans of the Rockstar’s classic PS3 ‘Red Dead Redemption’ game, you might remember Jose Gonzalez’s Far Away track. In this mix I’m featuring another legendary track from him called Crosses but with a more dancy touch ;)

I hope you enjoy the mix as much as I did putting it together :)

Tracklisting:

01. Annie – Intro
02. Randy Crawford – Street Life
03. Metro Area – Let’s Get…
04. Neon Neon – Raquel
05. The Outrunners – Runaway
06. Lindstrøm & Christabelle – Baby Can’t Stop (Aeroplane Remix)
07. The Faint – Battlehymn For Children (Tensnake Remix)
08. Paul Kalkbrenner – Atzepeng (Special Berlin Calling Edit)
09. Der Dritte Raum – Swing Bop (Tanz Variante)
10. Danny Howells – Right Off (Faze Action Remix)
11. Feadz – The Bright Side
12. Justin Martin – The Sad Piano (Jimpster Remix)
13. Jose Gonzalez – Crosses (Suburban Remix)
14. Masomenos feat. Curro Savoy – Ningeressa
15. Paul Kalkbrenner – Castenets (Special Berlin Calling Edit)
16. Swen Weber – First Stroke
17. 2Raumwohnung – Wir Werden Sehen (Solomun Vox Remix)
18. James Zabiela – No Other Way But Down
19. Vitalic – Poison Lips
20. Ossie – Holly Land

Links:

• Subscribe to Podcast
• Subscribe via iTunes
• Direct Download

I’ve taken the opportunity as well to upload my previous mix, ‘The Fat Beat Diet Mix’, to my podcast so that you can listen it without any spoken interruptions. As always, you can get this mix by subscribing to the podcast, or you can find a direct download here.

Infinispan is different to many applications out there using Protobufs because it has no control over what the users put in Infinispan and so such generic marshalling layer cannot make any assumptions on the type when trying to convert it from byte[] to a Java/Python/C++ object. So, the traditional methods used by Protobufs for example in Java don’t really work for Infinispan. In other words, if you have a Person java class generated with protoc precompiler, you’d transform the the byte[] into an instance of Person doing something along the lines of:

 byte[] buffer = ...;
 Person me = Person.parseFrom(buffer);

As I said earlier, such generic marshalling layer could not do that because the type is unknown. If you look at Java marshalling strategies such as Java Serialization or JBoss Marshalling, they don’t have these problems. Out of the payload byte[], they can figure out the type, instantiate it and populate it accordingly. So, I wondered whether something similar could be built on top of Protobufs. If you take Java as example for the rest of the article, a naive approach would be to simply prepend the payload generated by Protobufs with a UTF-8 String containing the Java class name. The receveiving end could read this, locate the class, and then using reflection call the static parseFrom method. However, what sort of Java class name would a Python generic marshaller provide?

A potential solution for this is hinted in the Techniques section in the Protobufs documentation. By generating a FileDescriptorSet out of your .proto files, at runtime you can build a mapping between the proto language descriptor names and corresponding Java class names. The proto language descriptor name is something that’s generic irrespective of the target language, so if the marshaller could prepend this to the payload sent, the reading part could read it, figure out the corresponding Java class name and use the method explained earlier. Let’s look at this in greater detail taking the Protobufs Basic Java Tutorial as starting point:

1. First, let’s generate the FileDescriptorSet file:
   ^?View Code BASH

   ./protoc -I=src/main/proto --descriptor_set_out=src/main/desc/addressbook.desc src/main/proto/addressbook.proto

2. Next, on startup inspect the .desc file generated and map the Protobuf full descriptor name with the corresponding fully qualified Java class name:
   ^?View Code JAVA

   import com.google.protobuf.DescriptorProtos.FileDescriptorSet; import com.google.protobuf.DescriptorProtos.FileDescriptorProto; import com.google.protobuf.Descriptors.FileDescriptor; import com.google.protobuf.Descriptors.Descriptor; import java.util.HashMap; import java.util.Map; import java.io.FileInputStream;   Map<String, String> mapping = new HashMap<String, String>();   FileDescriptorSet descriptorSet = FileDescriptorSet.parseFrom( new FileInputStream("src/main/desc/addressbook.desc"));   for (FileDescriptorProto fdp: descriptorSet.getFileList()) { FileDescriptor fd = FileDescriptor.buildFrom(fdp, new FileDescriptor[]{});   for (Descriptor descriptor : fd.getMessageTypes()) { String className = fdp.getOptions().getJavaPackage() + "." + fdp.getOptions().getJavaOuterClassname() + "$" + descriptor.getName(); mapping.put(descriptor.getFullName(), className); } }

3. Once the mapping is in place, it’s time to take an Protobuf object and try to generate a byte[]. To make sure the reading part can figure out what the Java class to use, the writing part needs to prepend the Protobuf generated byte[] with the Protobuf descriptor name:
   ^?View Code JAVA

   import com.acme.protobuf.AddressBookProtos; import java.io.ByteArrayOutputStream;   AddressBookProtos.Person.Builder personBuilder = AddressBookProtos.Person.newBuilder(); personBuilder.setId(1); personBuilder.setName("Galder Zamarreño"); personBuilder.setEmail("galder@galder.com"); AddressBookProtos.Person person = personBuilder.build(); byte[] buffer = objectToByteBuffer(person);   public byte[] objectToByteBuffer(Object o) throws Exception { ByteArrayOutputStream baos = new ByteArrayOutputStream(); Message message = (Message) o; byte[] name = message.getDescriptorForType().getFullName() .getBytes("UTF-8"); baos.write(name.length); // TODO: Length as int and not byte // Write the full descriptor name, i.e. protobuf.Person baos.write(name); byte[] messageBytes = message.toByteArray(); baos.write(messageBytes.length); // TODO: Length as int and not byte baos.write(messageBytes); return baos.toByteArray(); }

4. With the buffer written, it’s time to read it now:
   ^?View Code JAVA

   import com.acme.protobuf.AddressBookProtos; import java.io.ByteArrayInputStream; import java.lang.reflect.Method;   AddressBookProtos.Person person2 = (AddressBookProtos.Person) objectFromByteBuffer(buffer);   public Object objectFromByteBuffer(byte[] buffer) throws Exception { ByteArrayInputStream bais = new ByteArrayInputStream(buffer); byte[] name = new byte[bais.read()]; bais.read(name); // TODO: Read fully?? // Get the class name associated with the descriptor name String className = mapping.get(new String(name, "UTF-8")); Class clazz = Thread.currentThread().getContextClassLoader() .loadClass(className); Method parseFromMethod = clazz.getMethod("parseFrom", byte[].class); byte[] message = new byte[bais.read()]; bais.read(message); // TODO: Read fully?? return parseFromMethod.invoke(null, message); }

5. Finally, you can try to assert whether the object written and the object read are the same:
   ^?View Code JAVA

   assert person.equals(person2);

That’s it! It’s worth noting that this approach has some disadvantages. It uses reflection to call the parsing method which is slower than calling the static parseFrom method directly on Person class. On top of that, users need to add an extra step of generating the FileDescriptorSet and it needs to be passed to the generic marshalling layer. Finally, adding the descriptor name to payload increases its size.

So, this approach is not flawless by any means. In fact, we will instead use a marshaller based in Apache Avro which suits our use case better (see “Portable Serialization For Hot Rod With Apache Avro” wiki for more info on its use and you can find the actual Apache Avro marshaller code in our SVN repository). However, I thought it’d be of interest to anyone else out there who might be trying to do something similar or anyone who has done it in a different way.

Feedback appreciated :)

As far as I’m concerned, I’ll be at the JAOO conference in Aarhus, Denmark, where I’ll be speaking about the importance of the brand new Infinispan Server modules, emphasizing the motivation for developing them and showcasing some really exciting use cases.

Just a few days later I’ll be in Berlin for Europe’s round for hosting JBoss’ JUDCon, a developer conference by developers where I’ll be introducing a brand-new, innovative data eviction algorithm included in Infinispan 4.1, which increases eviction precision and reduces overhead. I will also be talking about the Infinispan Server modules in this conference. On top of that, Mircea Markus will be explaining how transactions are handled within Infinispan and how to do continuous querying of Infinispan data grids. In total, Infinispan has 4 talks @ JUDCon which is very exciting for us!!

To round things up, on the 14th of October I’ll be in Lausanne where I’ll be introducing the audience of the local JUG to Infinispan, talking about our motivations to create Infinispan, how different it is to JBoss Cache…etc.

So, if you happen to be around, make sure you come! It’ll be fun :)

However, it seems like there’s no such issue with Ableton Live 8 as indicated in this forum entry. In fact, I was suffering from the issues mentioned above and once I upgraded to Ableton Live 8, my problems dissapeared. If for whatever reason you’re unable to upgrade to Live 8, all is not lost! You can install a small tool called iToones created by misal that shows your iTunes library in a separate tool from which you can indeed drag and drop to Live 7. It’s a good workaround but in my opinion, nowhere near as good as dragging directly from iTunes where you might have extra information such as genre, rating…etc.

Now that I’ve solved my Ableton Live and iTunes issues, I can get on with my next mix :)

After very busy last few weeks with JUDCon (checkout my Hot Rod presentation here) and JBoss World/Red Hat Summit (check out my presentation with Manik Surtani here and the REST for Clouds one here), we’re proud to release Infinispan 4.1.0.CR1, the first candidate release of the Infinispan 4.1 series. The release is downloadable in the usual place. A lot of work has gone into this release primarily with the aim of stabilising new functionality wrote in previous beta/alpha versions. Here are some of the highlights included in this release:

• An fantastic demo showing how to run Infinispan in EC2. Check Noel O’Connor’s blog last month for more detailed information.
• Enable Hot Rod servers to run behind a proxy in environments such as EC2, and make TCP buffers and TCP no delay flag configurable for both the server and client.
• Important performance improvements for Infinispan based Lucene directory and Hot Rod client and sever.
• To avoid confusion, the single jar distribution has been removed. The two remaining distributions are: The bin distribution containing the Infinispan modules and documentation, and the all distribution which adds demos on top of that.

A more detailed changelog can be found here.

Finally, if you’re a user of Infinispan 4.0 or 4.1, please make sure you download and try this release out so that any outstanding issues are fixed in time for the final release. Also, if you’re interested in finding out more about Infinispan’s architecture, don’t miss Manik’s latest article explaining Infinispan’s ‘nuts and bolts’.

The Hot Rod Protocol in Infinispan – JUDCon
On the 21st of June, I’ll be introducing the audience to Hot Rod, Infinispan‘s client-server protocol, explaining the motivation for creating it and comparing it with other existing protocols. I’ll also be talking about the potential use cases and will showcase a demo of a cluster of Hot Rod servers running on EC2.

Using Infinispan for High Availability, Load Balancing, & Extreme Performance – JBoss World
In this talk which will be held on the 24th of June, myself and Manik Surtani will be talking about the most interesting uses of Infinispan that we’ve seen in the community. With the experience of the challenges faced and lessons learned, we’ll be providing the audience with some valuable tips for any Infinispan based projects.

Why RESTful Design for the Cloud is Best – Red Hat Summit (Cloud)
Finally, on the last day of the summit, I’ll be talking about the relationship between REST and Clouds, focusing on the principles that make REST a successful technology for cloud-like environments.

If you happen to be around in Boston during these dates and you’re interested in these topics, do not hesitate in joining us :)

The pictures:

F1 Monaco Grand Prix207 picturesMay 14, 2010

The videos: