Neil Bartlett

March of the IDEs — IDE Agnostic OSGi Development

2017-09-16T00:00:00+00:00

For a long time Eclipse has been, in practice, the only widely used and supported IDE for developing OSGi applications. Both of the premier IDE plugins — namely Bndtools and the older PDE — supported only Eclipse. While some tooling existed in IntelliJ and NetBeans it was nowhere near as comprehensive or usable.

This wasn’t much of a problem for me personally. I have been a fan of Eclipse since I first saw it in the early 2000s, and indeed the first IDE I ever used was VisualAge for Java, which later morphed into Eclipse. I’ve tried other IDEs but I always end up reverting to the familiar.

However IDE choice is very personal, and the devotion of some developers to their preferred IDE can make religious fanatics seem positively lukewarm. So if you met an IntelliJ fan and told her that she would need to switch to Eclipse in order develop for OSGi… well, it’s unsurprising that a common response was that OSGi could get stuffed.

So while I continue to use and support Bndtools in Eclipse, for the sake of OSGi adoption I have long wished for there to be a better development experience in the other popular IDEs. I’m happy to report that substantial progress has been made towards this goal. The latest version of bnd includes a suite of plugins for both Maven and Gradle that make development of OSGi applications practical irrespective of the IDE. Combined with some minimal IDE-specific tooling, such as that included in the latest IntelliJ builds, it is now a productive and even pleasurable activity.

On 23 October I will present a talk at the OSGi Users Forum Germany Tooling Workshop titled March of the IDEs — IDE Agnostic OSGi Development. In my talk I will demonstrate the development of a single OSGi application using Eclipse, IntelliJ, Visual Studio Code and a variety of command-line tools. This will highlight the opportunities for developers of all creeds to collaborate on high-quality modular code.

The workshop takes place all day on 23 October and is located at the Forum am Schlosspark, Ludwigsburg, Germany. This is the same venue as EclipseCon Europe and OSGi Community Event 2017 but it is organised as a separate event. Participation is free and you do not need a ticket for EclipseCon but you do need to register with the OSGi User Forum Germany by emailing germany-info@osgiusers.org. I hope to see you there!

Bndtools and Maven: A Brave New World

2017-09-13T00:00:00+00:00

At this year’s OSGi Community Event – co-hosted with EclipseCon Europe 2017 – I will present a new tutorial titled Bndtools and Maven: A Brave New World. My friend and Paremus colleague Tim Ward will co-present and will provide support during the exercises.

Bndtools has long provided the premier development experience for writing OSGi bundles and applications. However there was a significant gap: until recently it has been very difficult to use Maven to build Bndtools projects. This was partially because of legacy issues (after starting with ANT as our standard build tool, we eventually migrated to Gradle). Partially it was because Maven is far more prescriptive in terms of the structure and lifecycle of a project, and it took a long time to find a way to fit our models together. These problems meant that Bndtools was simply unusable for a large number of developers, in particular those working for organisations that have standardised on Maven – and also of course those developers who simply prefer Maven!

These problems are now essentially solved and, from Bndtools version 3.4, Maven is a fully supported build environment. Gradle also remains fully supported.

In this tutorial, we will walk through all of the day-to-day development tasks in the new Bndtools/Maven toolchain. These include:

Starting a new project from scratch;
Adding bundle modules to the project;
Managing library dependencies;
Integration testing (i.e. in-container testing);
Running bundles and tests from the IDE, and the rapid code/run cycle enabled by Bndtools;
Indexing, resolving and assembling applications for deployment;
Managing bundle and package versions;
Configuring a Continuous Integration build.

The following Maven plugins will be covered:

bnd-maven-plugin: the core plugin that generates OSGi bundle JARs;
bnd-testing-maven-plugin: executes OSGi in-container tests;
bnd-indexer-maven-plugin: indexes OSGi bundles and dependencies for assembly and/or runtime resolving;
bnd-export-maven-plugin: assembles a standalone OSGi application from its minimal bundles.

We do hope that you will attend this tutorial armed with lots of questions! Prerequisites are Bndtools 3.5 (which will have been released before the conference), JDK 8 and Maven 3. Also make sure you sign up to attend the tutorial on the EclipseCon website, and of course you also need a ticket to the main conference.

Using Let's Encrypt Certificates with OSGi HTTP Service

2016-06-07T12:00:00+00:00

Rather than one of my usual polemics, this is a quickly-written practical post. I hope the information here is useful to somebody.

Currently I am setting up effectiveosgi.com, the website for my upcoming book “Effective OSGi”. The site is a web app that will allow users to download preview PDFs, order print copies, and so on. Naturally I am developing it in Java and OSGi (in fact, many of the code samples in the book are based on the application that will be used to sell and deliver it).

Of course I want to follow best practices and use TLS (HTTPS), and this is much easier nowadays thanks to Let’s Encrypt. However, it wasn’t exactly obvious how to import the certicate provided by Let’s Encrypt into a Java/OSGi application, so I’m documenting the steps that worked for me.

Step One: Get the Certificate

This part I’m not going document in detail because it’s more than adequately documented on the Let’s Encrypt - Getting Started page. I used the “standalone” method. The subsequent steps assume that you have successfully obtained your certificate files, which will be saved under the path /etc/letsencrypt/live/[yourdomain]. The files you will find there are cert.pem, chain.pem, fullchain.pem and privkey.pem.

Step Two: Build Java Keystore

Note that this step is relevant for all Java developers, not just OSGi users.

Java applications cannot directly read PEM files; we need to generate a Java keystore file using the keytool program included with the JDK. Unfortunately keytool also can’t read PEM files directly, so we first have to use openssl to convert them to PKCS12 format:

openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem \
        -out cert_and_key.pkcs12 -name MyDomain \
        -CAfile chain.pem -caname root

You will be prompted to enter an export password. This will only be used in the next step, so it’s fine to use just password. Now we have a file called cert_and_key.pkcs12 which combines both the full certificate chain with the private signing key. Import this into a new keystore file as follows:

keytool -importkeystore \
        -srckeystore cert_and_key.pkcs12 -srcstoretype PKCS12 -srcstorepass password \
        -destkeystore letsencrypt.jks -deststorepass XXXXXX -destkeypass XXXXXXX

You now have a keystore file named letsencrypt.jks. Keep the store and key passwords safe and secret!

Step Three: Configure HTTP Service

This step is relevant to OSGi developers. I’ll assume you are using the Apache Felix HTTP Service.

The HTTP Service is configured with the standard OSGi Configuration Admin. You need to create a configuration record with a PID of org.apache.felix.http with the following settings:

org.apache.felix.https.enable true
org.osgi.service.http.port.secure: 8443
org.apache.felix.https.keystore: path/to/letsencrypt.jks
org.apache.felix.https.keystore.password: XXXXXX
org.apache.felix.https.keystore.key.password: XXXXXXX

This should be all you need to do get a green lock icon in your browser. Congratulations, you are now running a properly configured HTTPS server!

However… it’s not very secure yet. Once your site is running, I strongly recommend testing your TLS implementation using SSL Labs from Qualys. If you do this now, you will get at best a B grade.

Step Four: Getting the A Grade

One of the security warnings reported by SSL Labs is due to weak Diffie-Hellman groups. I don’t pretend to understand everything on the linked page, but fortunately the fix is easy: you just need to make sure your Java VM uses 2048-bit Diffie-Hellman keys by setting a system property. This can be specified on the command line as follows:

java -Djdk.tls.ephemeralDHKeySize=2048 ...

To achieve forward secrecy you also need to restrict the cipher suites to use only elliptic-curve algorithms. It’s also a good idea to limit the server to 256-bit rather than 128-bit algos. This is done in the configuration admin record for the HTTP Service; just add the following:

org.apache.felix.https.jetty.ciphersuites.included: \
    TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, \
    TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, \
    TLS_RSA_WITH_AES_256_CBC_SHA256, \
    TLS_RSA_WITH_AES_256_CBC_SHA

With these settings you should now be a grade A student:

Finally: you might be wondering how to get an elusive A+ grade on SSL Labs. Alas it doesn’t currently seem to be possible in a Java-based server (though I may be wrong – please let me know in the comments). The feature needed to unlock A+ is called TLS_FALLBACK_SCSV, and it requires an API change. Hopefully it will be implemented in Java 9.

Vertical Components

2016-05-06T10:00:00+00:00

Combining Web and OSGi Components

I consider myself a latecomer to Microservices; I only really started to understand and use them in 2005. Nobody called them “Microservices” at the time – as far as I can tell, the term was coined in 2006 by BEA for their microService Architecture (mSA). Regardless, with this history it was a little surprising to hear the term reinvented in 2014 and suddenly gain a great deal of traction. While implementation details vary, the fundamental concept has not changed: small, independently deployable, discoverable services.

I’m always keen to hear from people with a different take, and so in 2014 I found myself listening to a Software Engineering Radio podcast interview with Stefan Tilkov, an expert on REST and software architecture, on the subject of microservices. One of his ideas as elaborated on the podcast was to use microservices to slice applications vertically, rather than the horizontal layering that is much more commonly seen in enterprise architectures. He promotes the idea of a Self-Contained System that includes all of the logic, data and UI for a single business domain. Inclusion of the UI is what makes the approach unconventional: componentisation is relatively commonplace in the back-end, but does not often extend all the way up to the user. This is a fine idea, as it allows technical teams to align with business units. When building horizontal layers (data access layers, application servers, ESBs, etc) developers easily get sidetracked into endlessly polishing frameworks rather than delivering real business value.

The problem was, at least as I saw things at the time, that the UI tier was hard to componentise in this way. The web is by far the dominant UI platform, but web modularity has historically been difficult. The traditional unit of deployment and isolation in the web is the HTML page, but in a modern web application the page is the whole application. Frameworks like Angular and React help in wiring together single-page applications, but didn’t really allow applications to be composed out of independently deployable components. Also, crucially, code written for Angular is useless in a React app and vice versa.

Of course, I was missing the Web Components family of specifications. In my defence, I don’t think many people knew about Web Components, as all the noise seemed to be around the competing frameworks. Also they were not really practical until recently due to the lack of support in browsers. As I write this in mid-2016, however, things have moved forward and Web Components seem to finally be ready for production. This means we can now write independently deployable and reusable components that can be assembled into an application. There are various libraries for building components, but the specifications allow (theoretically!) for components to interoperate irrespective of their implementation details. The application developer can still use a framework to assemble these components into a complete application.

The parallels with OSGi components are very clear. So I started thinking about whether we can combine the two worlds to define vertical components. A web component is often required to invoke logic or retrieve data from a back-end server, which creates a coupling. So what if we could create an artifact that deploys both the front-end and the back-end components together? And then what if we could use the OSGi Resolver to assemble an application out of its required UI and back-end components? So that’s what I have experimented with, and the results seem quite encouraging.

I have shared a proof-of-concept on GitHub, which implements a very simple Shopping Cart component that displays the content of the customer’s cart, retrieved from a back-end REST service.

Web Component Code

I have tried to use as few libraries and assistive technologies as possible, in order to show that nothing magical is happening. So, while libraries like Polymer or Bosonic are handy for creating web components, it’s also pretty easy to define a simple component in plain vanilla ES2015:

(function() {
  let template = `
    <div class="cart-list">
      <h2>Cart Contents:</h2>
      <table class="items-table"></table>
    </div>
  `;
  
  class CartListWidget extends HTMLElement {
    createdCallback() {
      this.createShadowRoot().innerHTML = template;
      this.$table = this.shadowRoot.querySelector('.items-table');
      this.requestServer();
    };
    requestServer() {
      var that = this;
      var xhr = new XMLHttpRequest();

      xhr.open('GET', '/cart-list/servlet/');
      xhr.onload = function() {
        var cart = JSON.parse(xhr.responseText);
        var rows = '';
        for (var i = 0; i < cart.length; i++) {
          var row = '<tr><td>' + cart[i].sku + '</td><td>'
                  + cart[i].price + '</td></tr>';
          rows += row;
        }
        that.$table.innerHTML = rows;
      };
      xhr.send();
    };
  }
  document.registerElement('cart-list', CartListWidget);
})();

OSGi Component Code

The JavaScript file is served from an OSGi bundle that also provides the REST service. I could have used JAX-RS for this, but a plain old Servlet works just as well for such a small demo. I even build the JSON response with printf!

@Component(
    property = {
      HTTP_WHITEBOARD_SERVLET_PATTERN + "=/cart-list/servlet/*",
      HTTP_WHITEBOARD_RESOURCE_PATTERN + "=/cart-list/*",
      HTTP_WHITEBOARD_RESOURCE_PREFIX + "=/cart-list"
    })
@WebComponent(name = "cart-list")
@RequireHttpImplementation
public class CartServlet extends HttpServlet implements Servlet {

  private static final long serialVersionUID = 1L;
  
  @Reference
  private CartManager manager;

  public void doGet(HttpServletRequest request, HttpServletResponse response)
                                        throws ServletException, IOException {
    try (PrintStream out = new PrintStream(response.getOutputStream())) {
      Cart cart = manager.getCart("anonymous");

      StringBuilder builder = new StringBuilder();
      builder.append("[");
      int row = 0;
      for (CartEntry entry : cart.listEntries()) {
        if (row++ > 0) builder.append(',');
        builder.append(String.format("{\"sku\" : \"%s\", \"price\" : %d}",
                       entry.getSku(), entry.getSoldPrice()));
      }
      builder.append("]");
      response.setHeader("Content-Type", "application/json");
      out.print(builder.toString());
      out.flush();
    } catch (Exception e) {
      throw new ServletException(e);
    }
  }
}

Application Code

Finally, the application bundle contains index.html which simply uses the cart-list component we defined:

<!DOCTYPE html>
<html>
<head lang="en">
	<meta charset="UTF-8">
	<title>Bookstore App</title>
	<script src="webcomponents.min.js"></script>
	<script src="/cart-list/component.js"></script>
</head>
<body>
	<h1>Book Store</h1>
	<cart-list></cart-list>
</body>
</html>

Resolution

In the code there are a couple of new annotations, which I have adapted from OSGi en Route. In the web component bundle, the @WebComponent annotation generates a Provide-Capability header in the OSGi bundle manifest that declares our bundle provides the cart-list web component. The @RequireHttpImplementation generates a requirement for an HTTP Service implementation.

In the application bundle we use @RequireWebComponent to require the component that is provided by the web-component bundle. The application bundle also repeats the @RequireHttpImplementation annotation for completeness.

These annotations are used by an OSGi Resolver, such as the build-time resolver in Bndtools or the runtime resolver in Paremus Service Fabric. Because of this, we don’t need to manually list all of the components and libraries used by the application… we only need to list the top-level bundle (in this case org.example.webapp.bookstore which contains the index.html) and the tooling will automatically pull in the cart-list component bundle along with an HTTP Service implementation such as Jetty.

Conclusion

This is just a proof-of-concept and there are many details that could be improved. For example, the mapping between the component name and the URL of the JS and Servlet could be handled by a small server-side framework. Using third-party components from Bower or NPM could be made simpler, perhaps via an extender bundle.

What do you think… is this a useful approach? What flaws have I missed? Let me know in the comments, or using the contact details to the right.

If you are interested in learning more about web development with OSGi, and about modern component-oriented development with Java, then please consider supporting my new book, Effective OSGi, or buying a copy when it is published!

Jigsaw is a Shibboleth

2015-12-03T12:00:00+00:00

It comes as no surprise that Oracle has proposed to delay the release of JDK 9 for 6 months (but note that JSR 376 specifying the Java Platform Module System is being delayed by a full year, so Java SE 9 itself will likely be delayed by at least that long).

The reason for the delay will also be no surprise: Jigsaw is still not ready.

The crusade to develop a new module system for Java has been long and mostly fruitless. JSR 277 “The Java Module System” began over ten years ago, in June 2005… more than a year before Java 6 was released, and five years before Oracle acquired Sun Microsystems. It was meant to be delivered in Java 7 but it slipped to Java 8. When it looked like it was going to seriously delay the release of Java 8, it slipped again to Java 9. Now the same thing is happening yet again. The tongue-in-cheek prediction made by Robert Dunne is gradually coming true:

All I know about Java version 20 is that Project Jigsaw will be looking good and on target for version 21 http://t.co/CqHCI1nTaV
— Robert Dunne (@rsdunne) July 29, 2014

It really doesn’t have to be this way: Jigsaw has already achieved something extremely valuable, and Java 9 could be released on schedule with Jigsaw as it stands, in addition to the other nice features slated such as compact Strings and standard replacements for sun.misc.Unsafe.

What Jigsaw has achieved is modularisation of the JDK, including the standard libraries. With the jdeps tool that was released in Java 8, you can quite easily work out the minimal set of platform modules needed by your application, and then ship a cut-down copy of Java along with it. This will be invaluable for desktop and mobile apps, and for Java’s viability in the burgeoning IoT market.

What Jigsaw has not yet achieved – and frankly seems to be as far off as ever – is delivering a practical module system for applications. There are huge question marks over whether the uncompromising inflexibility of Jigsaw can ever work for Java EE, or Spring, or CDI, or Hibernate, or dynamic languages like Groovy and JavaScript. This much is apparent to anybody browsing the mailing lists. Unfortunately Oracle is holding up the release of the modularised JDK while they try to figure out all of the application-level problems.

Worse, Jigsaw is already quite complex and will only grow in complexity as it is forced to support code patterns that are widely used in the Java ecosystem. Jigsaw is repeatedly touted as a “simple” module system… simpler than OSGi, which is perceived as complex. But OSGi did not set out to be complex: it is only as complex as it needs to be in order to work in the messy real world. When Jigsaw reaches a comparable level of complexity, what will be the excuse for choosing it over OSGi?

OSGi is clearly a better module system for applications. It is dynamic, it has a powerful service registry, and has a dependency model that makes sense and is amenable to automated reasoning. As I proved in my last blog post, it works on Java 9 and can take advantage of the modular JDK. And of course, it has been maturing and solving real world problems for over 15 years. However I have long acknowledged that the JDK itself has some unique problems that were difficult to solve with OSGi, which is why Jigsaw was needed. Back when Jigsaw slipped from Java 8, even Oracle’s Senior VP of Development agreed that this combination was a good solution.

I think that the Java community would be better served by releasing Java 9 on time, including the modularised JDK provided by Jigsaw. Java today competes against many languages and platforms that are easier to use (or at least easier to start using), cooler, smaller, and fast enough. It can stay relevant by slimming down, speeding up and implementing more useful features – not reimplementing poor replicas of features already developed years ago by the community.

Jigsaw is great. It’s time to declare victory, release it, and move on.

OSGi and Java 9 Modules Working Together

2015-11-13T12:00:00+00:00

The best thing about the Java Platform Module System – commonly known as Project Jigsaw, or JSR 376 – is that it has been used to break apart the monolithic Java runtime into smaller modules. This makes Java a more attractive implementation technology in many scenarios where it has historically been weak. For example, modern desktop or mobile applications can’t rely on having a pre-installed copy of Java on the user’s device, so you need to ship Java as part of your application. IoT is also driving developers to seek smaller installed size and memory footprint.

Unfortunately, Jigsaw has some shortcomings as a module system for applications. I don’t want this blog post to be a critique of Jigsaw, but suffice to say there are many developers who will prefer to use OSGi for application modularity.

Still, it would be very useful if we could run a modular OSGi application on a modular Java runtime. This would allow us to benefit from a minimal Java install, and at the same time use the full power of OSGi. Also it would help to hide much of Jigsaw’s complexity from application developers.

I have just spent a day hacking together a demo that proves you can do this.

I must emphasise that this is a throw-away prototype. You should absolutely not start building on it. The OSGi Core Platform Expert Group (CPEG) will be working to specify official support for Java 9 features… I just wanted to show that a solution can exist, and perhaps feed into their discussions.

NB: The following description may get confusing because of the overlapping concepts used by Jigsaw and OSGi, so I have to be very careful with terms. Whenever I say “module” I always mean a Jigsaw/Java 9 module, and when I say “bundle” I’m always talking about OSGi.

Use Case: OSGi Bundles in a Known Platform Configuration

Suppose we have a Java runtime containing a known set of modules. We would like to start an OSGi Framework within the context of a module, such that the bundles in that framework can import packages from the platform modules.

As background, OSGi has long used the concept of an Execution Environment (EE), which models the Java platform on which the framework is running. These have names such as JavaSE-1.7. When an OSGi framework starts up, it detects what kind of platform it is running on and declares the EE as a capability that bundles can depend on.

If you are unfamiliar with the term “capability” in this context, it is basically how all dependencies work in OSGi. Any bundle can declare a capability, which has a namespace and a set of attributes. Then any other bundle can require a capability, optionally with a filter expression. The OSGi Framework ensures that for every requirement, there exists a bundle providing a matching capability, otherwise the requirement will not resolve. OSGi additionally defines the meaning of some special capability namespaces. One of these is osgi.ee, which is declared by the OSGi system bundle to indicate the Java platform version.

Why can’t we work out a bundle’s platform dependency based on its imported packages? That’s not possible for two reasons. First, the packages from the Java platform are not versioned. Second, a large subset of packages – namely those starting with java. (e.g. java.net, java.lang etc) cannot be imported; they have to be loaded from the boot class loader. So bundles have to declare a requirement on the EE in order to ensure they can safely call methods like Files.readAllBytes (added in Java 7) or String.isEmpty (added in Java 6).

An EE is basically a monolith: Java 7 is Java 7 and there are no other flavours. Things got a bit more complicated in Java 8 with the introduction of Compact Profiles… but these are basically still monolithic platforms. With just three profiles defined in the Java 8 specification, it was no problem for OSGi to define an EE for each profile. However in Java 9 there is a very large set of potential module combinations – too many to model with EEs.

In my prototype I posit a capability namespace of jmodule to describe the Java modules available in the platform. At runtime, the launcher inspects the set of available modules and uses them to declare the jmodule capability. The relevant code looks like this:

// Calculate the system capabilities
List<String> systemCapabilites = new ArrayList<>();
for (Module module : Main.class.getModule().getLayer().modules()) {
    String cap = String.format("jmodule;jmodule=%s;version:Version=1.9", module.getName());
    systemCapabilites.add(cap);
}

// Configure the framework
Map<String,String> configMap = new HashMap<>();
configMap.put(Constants.FRAMEWORK_SYSTEMCAPABILITIES_EXTRA,
              formatAsList(systemCapabilites));
Framework framework = frameworkFactory.newFramework(configMap);

An OSGi bundle can now declare a requirement for a Jigsaw module as follows:

Require-Capability: jmodule; filter:="(jmodule=java.xml)"

… and the OSGi Framework will refuse to resolve the above module if the required module (in this case, java.xml) is not present in the platform. This is valuable because it protects against NoClassDefFoundErrors and ClassNotFoundExceptions from occurring at some unknown time during the life of the bundle. OSGi provides the assurance that the bundle will actually work on the platform that it finds itself on.

But how do we know that our bundle depends on the java.xml module? At the moment we have to use the jdeps tool from the JDK:

$ jdeps -module org.example.xml.jar
org.example.xml.jar -> java.base
org.example.xml.jar -> java.xml
org.example.xml.jar -> not found
   org.example.xml (org.example.xml.jar)
      -> java.io                                            java.base
      -> java.lang                                          java.base
      -> javax.xml.parsers                                  java.xml
      ...

… which tells us that our bundle depends on java.base and java.xml. The java.base module is implicit so we can ignore it, but we do need to add java.xml to the requirements.

This process is manual and cumbersome right now, but in the future bnd should be able to generate the requirement automatically.

We’re not done yet though, because the Java platform provides a whole bunch of packages that are not in the java.* namespace. For example: javax.xml, javax.sql.rowset, org.w3c.dom, org.omg.CORBA etc. We don’t need to do anything different in our OSGi bundles for these, because bnd will generate imports as normal. However we need the OSGi launcher to work out which packages are available from the platform modules. It can do this again by iterating over the modules using reflection:

// NB this sample ignores package versions for now
private static List<String> calculateExports(Module module) {
    String[] packageNames = module.getPackages();
    List<String> exports = new ArrayList<>(packageNames.length);
    for (String packageName : packageNames) {
        // ignore java.* packages
        if (packageName.startsWith("java."))
            continue;

        if (module.isExported(packageName))
            exports.add(packageName);
    }
    return exports;
}

This is actually a massive boon to the OSGi Framework implementers. In previous Java releases, the Framework had to maintain a list of all the known packages provided by the platform; there was no standard API for enumerating classes or packages on the classpath (even if there were, it would reveal non-standard packages such as sun.misc, com.sun.* and so on). Take a look at the properties file that Felix maintains for this purpose… it’s larger than you might expect.

Implementation Issues

In the demo I have copied the source code for Apache Felix into my module. I did this to give myself freedom to hack the implementation, but so far have hardly needed to — more on that shortly.

Most of the code is in nbartlett.osgi_jigsaw.Main, and it’s mostly just a vanilla OSGi launcher. The main differences are the jmodule capabilities and system bundle exports as already described. The module-info.java looks like this:

module nbartlett.jigsaw_osgi {	
	exports org.osgi.framework;
	exports org.osgi.framework.dto;
	exports org.osgi.framework.hooks.bundle;
	exports org.osgi.framework.hooks.resolver;
	exports org.osgi.framework.hooks.service;
	exports org.osgi.framework.hooks.weaving;
	exports org.osgi.framework.launch;
	exports org.osgi.framework.namespace;
	exports org.osgi.framework.startlevel;
	exports org.osgi.framework.startlevel.dto;
	exports org.osgi.framework.wiring;
	exports org.osgi.framework.wiring.dto;
	exports org.osgi.resource;
	exports org.osgi.resource.dto;
	exports org.osgi.service.packageadmin;
	exports org.osgi.service.startlevel;
	exports org.osgi.service.url;
	exports org.osgi.service.resolver;
	exports org.osgi.util.tracker;
	exports org.osgi.dto;
 }

All of the packages of the OSGi Framework have to be exported by the module wrapping it. This is necessary because the bundles actually load into a different module – the “unnamed” module – and so we need Jigsaw to permit access to the OSGi API packages. If we fail to do this, the following error occurs:

java.lang.IllegalAccessError: superinterface check failed: class org.example.Activator
(in module: Unnamed Module) cannot access class org.osgi.framework.BundleActivator (in
module: nbartlett.jigsaw_osgi), org.osgi.framework is not exported to Unnamed Module

After fixing this, I encountered the following error when the OSGi Framework tried to instantiate the BundleActivator from any bundle:

java.lang.IllegalAccessException: class org.apache.felix.framework.Felix (in module
nbartlett.jigsaw_osgi) cannot access class org.example.Activator because module
nbartlett.jigsaw_osgi does not read <unnamed module @5cee5251>

This one was quite a puzzle! It turns out that if you create an instance of ClassLoader and use it to load classes dynamically, those classes will always belong to the unnamed module. In order to load classes into a real module you have to use ModuleClassLoader. However this class is final, so OSGI cannot override loadClass to implement OSGi loading rules.

Incidentally, the upshot is that any code that looks like the following – and this is a very common code pattern – will fail from inside a module other than the unnamed module:

URLClassLoader loader = new URLClassLoader(urls, this.getClass().getClassLoader());
Class<?> clazz = loader.loadClass("blah.Blah");
clazz.newInstance(); // <-- this line fails from within a named module

It fails because blah.Blah belongs to the unnamed module, and our module doesn’t require unnamed. You can’t write requires unnamed in the module-info.java, but you can dynamically add the requirement as follows:

this.getClass().getModule().addReads(loader.getUnnamedModule());

For now, we have to accept that OSGi bundles will load into the unnamed module. It’s an open question whether we can map them into the OSGi module, or even a module for each bundle. By adding the above line to Felix (BundleWiringImpl line 731), the Framework is able to load the bundle activator, and OSGi starts up:

g! lb
START LEVEL 1
   ID|State      |Level|Name
    0|Active     |    0|System Bundle (0.0.0)
    1|Active     |    1|Apache Felix Gogo Command (0.14.0)
    2|Active     |    1|Apache Felix Gogo Runtime (0.12.1)
    3|Active     |    1|Apache Felix Gogo Shell (0.10.0)
    4|Active     |    1|org.example.xml (0.0.0.201511120014)

In my demo I installed and started the three bundles of Felix Gogo, which gives me an interactive shell, and a fourth bundle that contains some XML parsing code. This code forces a dependency on the java.xml module and the javax.xml.parsers package from the platform, both of which are satisfied by the system bundle.

Use Case: Configuring the Platform for a Set of Bundles

Above we had a known platform and wanted OSGi to validate the bundles we wanted to run on it. Flipping this around, what if we have a set of bundle – our application – and want to work out the minimal platform that will run it?

You can already do this – kind of – with existing tools. The jdeps tool can print the module dependencies of each of your OSGi bundles, so the modules you need in your platform are just the aggregate of those jdeps outputs. The OSGi framework module I defined above doesn’t have any dependencies aside from java.base.

If you build your bundles using the future version of bnd that generates jmodule requirements for you, then you don’t need to run jdeps. It will be very simple to generate the aggregate dependencies just by looking at the bundle manifests. We can create a tool that builds an application artifact, including the minimal Java runtime modules plus OSGi bundles, as part of bnd/Bndtools. We already have an exporter for “standalone” JAR files, this would just be an extension of the same principle. But that’s for the future.

Open Questions

Here are some of the issues I haven’t even touched on, that will need a good answer before OSGi and Java 9 Module interop is a reality:

Versions! Jigsaw chose to ignore versions, so they will have to be layered on top somehow. This sounds like a recipe for lots of competing and incompatible schemes… the Maven way, the Gradle way, the OSGi way, etc. Should OSGi infer versions for the exports from the platform? If so, where should those come from?
Bundle-Module Mapping. The bundle classes load into the unnamed module, meaning that we have exactly as much protection and access control between bundles as OSGi has always had – no more, no less – and that’s fine. However, by using ModuleClassLoader we may be able to map each bundle to a Jigsaw module and benefit from increased isolation. I didn’t have time to figure out whether this would be possible.
Dynamics. With this prototype we can install, update and uninstall OSGi bundles dynamically. But if a newly installed bundle has requirements that are not satisfied by the current set of platform modules, can we install the required module into the platform dynamically? It looks like probably not, that’s just not within the scope of Jigsaw.
Uses Constraints. This is a tricky problem to describe and solve… see my post on the subject of uses constraints. Jigsaw module descriptors do not provide any information on the interdependencies between packages (e.g. the fact that javax.sql uses javax.transaction.xa, so if you import both of these packages then you really have to import them from the same source). Without this info, it will not be safe for OSGi bundles to offer exports that subtitute for exports from the platform.
Services. Jigsaw has a service model, though quite different from OSGi services. It would probably be useful for services from the platform to be bridged into the OSGi service registry.
Bidirectional Dependencies. Can we ever build a bidirectional link between OSGi and Jigsaw, so that bundles and modules can sit side-by-side in cooperative harmony? I don’t particularly care about this use case – I believe OSGi is the better application module system so it makes sense for the dependencies to flow downwards but not upwards. However, somebody might want to do this I suppose. It will be tricky though because of the dynamics in OSGi and lack of versions in Jigsaw.
Evolving EEs. The CPEG will need to decide what to do with EEs. Maybe they will be supplemented with something like thejmodule capability, or perhaps something completely different. We probably still need an EE to define the version and content of java.base.

Getting It

The prototype code is on GitHub at njbartlett/osgi_jigsaw.

To build the code, execute ./build.sh from the nbartlett-jigsaw-osgi directory. To run, execute ./run.sh.

Thanks to my employer Paremus for funding this research. We want to help our customers to build and deploy manageable, modular software, whatever the implementation technology.

Thanks to Sander Mak’s blog post for helping me to setup a Jigsaw workspace (though I can’t let this pass with commenting on his last paragraph… “vastly less complex”? Seriously dude are you feeling okay?)

What's New in Declarative Services 1.3?

2015-08-17T12:00:00+00:00

I’d like to tell you about two really cool new features in OSGi Declarative Services. They’re so cool that I feel kind of guilty for blogging about them, because some of the kudos may accrue to me even though I have done absolutely none of the work to either specify or implement them. So bear in mind that I’m only your humble messenger.

Configuration Property Types

As a bit of background, in Declarative Services (DS) it has always been possible to write configurable components by passing a Map into the activate method:

@Component
public class ServerComponent {

  @Activate
  public void activate(Map<String, Object> configProps) {
    String bindHost = (String) configProps.get("host");
    int port = (Integer) configProps.get("port");
    
    ServerSocket sock = new ServerSocket();
    sock.bind(new InetSocketAddress(bindHost, port))
    // ...
  }
}

There are some great things about the way DS handles configuration. First, the component itself has no idea about the origin of the config data… it could come from a properties file, or an XML file, or a database record, or transmitted over the air from a cellular provider, etc. Because the component is not coupled to the physical mechanism, we can change that mechanism easily without updating the code of all our components.

Second, the configuration is passed all-at-once and can be dynamically changed. Some other DI frameworks inject configuration data one field at a time (setHost, setPort…). Where they support dynamic reconfiguration at all, they tend to need methods to let the component know that a series of config changes is starting and ending, so that it knows when it’s safe to reconfigure. Otherwise the component could end up using a mixture of old and new config.

There is a problem, however. Did you notice that my code sample above had no error handling at all? What if the host property was missing from the map? What if the port was given as a String rather than an integer? What if it contained invalid numeric characters? Some fields may be optional, where do we specify the defaults? It turns out that most components have to do quite a bit of work to convert, parse and validate their configuration. There are libraries that encapsulate this – such as the Configurable API from bnd – but they add a runtime dependency, which is inconvenient (though manageable in OSGi with static linking).

Declarative Services 1.3 – part of the OSGi Release 6 compendium spec, which was released to the public last week – improves this enormously. It’s now possible to define a custom type for the configuration, rather than a generic Map. For example:

@interface ServerConfig {
  String host() default "0.0.0.0";
  int port() default 8080;
  boolean enableSSL() default false;
}

@Component
public class ServerComponent {

  @Activate
  public void activate(ServerConfig cfg) {
    ServerSocket sock = new ServerSocket();
    sock.bind(new InetSocketAddress(cfg.host(), cfg.port()));
    // ...
  }

}

Now the DS runtime is doing all the work of pulling out fields, converting them to the correct type and so on. If there are any conversion errors, or missing fields that don’t have defaults, then DS will refuse to load our component and will write an appropriate message into the OSGi Log.

This feature is known as “Configuration Property Types”. You may wonder why an annotation is used rather than an interface, given that we don’t actually use it like an annotation. There are a few reasons: in annotations it’s trivial to provide defaults; the methods cannot have parameters; and the return types allowed in an annotation are restricted to types that mesh well with configuration data.

Of course this version of DS interoperates with the OSGi Metatype specifcation, so that the administrators of our application can know what fields and types are actually expected. The metadata is generated at build time if we just add a couple more annotations:

@ObjectClassDefinition(name = "Server Configuration")
@interface ServerConfig {
  String host() default "0.0.0.0";
  int port() default 8080;
  boolean enableSSL() default false;
}

@Component
@Designate(ocd = ServerConfig.class)
public class ServerComponent {
  // ...
}

… and, from this definition, tools like the Apache Felix Web Console can automatically generate an admin GUI for our component:

Prototype Scope Services

The second really cool feature is support for prototype scope in services. The prototype scope has been available in the OSGi Core R6 specification for about a year now, but there was a significant gap between the release of the Core and Compendium documents. So until now we had to drop down to the low-level OSGi API in order to use prototype scope. In DS 1.3 we can use annotations and a more convenient runtime API instead.

Prototype scope is where a service has the ability to create multiple instances on demand. Traditionally, most OSGi services were conceptually singletons: each registry entry corresponded to a single back-end service instance that was shared by all consumers. (Aside: there were additionally bundle-scoped services, where each consumer bundle got its own instance of the service object, but if you used a service multiple times from within the same bundle you would still share the same instance).

This model had to be enhanced, since sometimes each consumer really needs to have its own instance of the service. Also some consumers need to have programmatic control over the creation and destruction of instances, in order to fit in with an external lifecycle. An example of this would be in web requests – some web standards (for example JAX-RS) require services to be instantiated and then destroyed in every request. Hence the prototype scope was added. This is an opt-in feature: both the provider and the consumer of the service need to be aware that it is being used.

In DS 1.3, a provider can opt in to the prototype scope very easily with an attribute on the @Component annotation:

@Component(scope = ServiceScope.PROTOTYPE)
public class MyComponent {
  // ...
}

As a consumer there are a few more choices. When we consume a service with the @Reference annotation, we can use an attribute to request a new instance per component:

@Reference(scope = ReferenceScope.PROTOTYPE_REQUIRED)
public void setFoo(Foo foo) {
  // ...
}

Using this annotation, each component that uses the Foo service will get its own instance. However within each component, the same Foo instance will be used repeatedly. Alternatively, the consumer can create and destroy instances programmatically whenever it wants – this requires the consumer to use a bit of API from DS as follows:

private ComponentServiceObjects<Foo> fooFactory;

@Reference(scope = ReferenceScope.PROTOTYPE_REQUIRED)
public void setFooFactory(ComponentServiceObjects<Foo> fooFactory) {
  this.fooFactory = fooFactory;
}

public void doSomething() {
  Foo myFoo = fooFactory.getService();
  try {

    // do something interesting with myFoo ...

  } finally {
    fooFactory.ungetService(myFoo);
  }
}

The use of the ComponentServiceObjects interface means we are no longer building strict POJOs, but still this interface is easily mockable for testing purposes.

Getting It

The specification for DS 1.3 is available now from http://www.osgi.org/Download/Release6. Felix SCR 2.0 implements the specification and can be downloaded from http://felix.apache.org/downloads.cgi. For full metatype support you will need Felix Metatype 1.1 from the same page.

To build components using the annotations you will need a development build of bnd/Bndtools 3.0. To install this, go to http://bndtools.org/installation.html and follow the instructions for installing a developer snapshot build.

Announcing the bnd Maven Plugin

2015-03-27T12:00:00+00:00

I am pleased to announce the availability of a new Maven plugin for building OSGi bundles, based on bnd. The plugin is called bnd-maven-plugin and it will be developed as part of the wider bnd and Bndtools projects.

The plugin has been kindly donated by one of Paremus’ customers – a very well known company whose identity we expect to be able to confirm in due course – under the terms of the Apache License version 2.

You might very well wonder: why did we build a new Maven plugin for OSGi development? Most OSGi developers who use Maven today are using the maven-bundle-plugin from Apache Felix. That plugin is mature and stable but it does have a number of issues that our customer wanted resolved.

The first major issue with maven-bundle-plugin is that it replaces Maven’s default plugin for JAR generation (the maven-jar-plugin). This means that it must use a different packaging type – “bundle” instead of “jar” – and must use the extensions=true setting to override Maven’s default lifecycle. As a result, the output artifacts are harder to consume in ordinary non-OSGi projects. Therefore, many Maven developers either don’t bother to release OSGi bundles at all because of the hassle, or they release both a “plain JAR” and a separate OSGi bundle. This is deeply frustrating because OSGi bundles already are plain JARs!… they can be used unalterered in non-OSGi applications.

The new bnd-maven-plugin instead hooks into Maven’s process-classes phase. It prepares all of the generated resources, such as MANIFEST.MF, Declarative Services descriptors, Metatype XML files and so on, under the target/classes directory, where they will be automatically included by the standard maven-jar-plugin (with one caveat discussed below).

The second issue is that maven-bundle-plugin has some questionable features and design choices. For example it automatically exports all packages that do not contain the substrings “impl” or “internal”. The motivation was understandable at the time – it is closer to normal Java with all packages being implicitly exported – but this is really just wrong from a modularity point of view, and goes against OSGi best practices. The bnd-maven-plugin instead includes all packages from the project in the bundle but exports nothing by default. The developer must choose explicitly which packages are exported.

The third issue with maven-bundle-plugin is that it is difficult to use in an incremental build environment. In Bndtools we have had literally years of discussions about how to best accommodate Maven users while still supporting the rapid development cycle that Bndtools users love. The new plugin will make this easier: for example after the process-classes phase has completed, the content of the target/classes directory is already a valid OSGi bundle, albeit in folder form rather than a JAR file. Also the new plugin takes its bnd instructions from the bnd.bnd file in each project, rather than from a funky XML encoding of those instructions deep inside the POM.

Finally, we believe that by delivering a Maven plugin directly from the bnd project, we can more quickly take advantage of new features in bnd, and develop features in bnd that directly benefit Maven users. Note however that the Gradle and Ant integrations of bnd will still be supported: we want everybody to use bnd to build their OSGi bundles, irrespective of IDE and build system.

So much for motivation, let’s look at some of the technical details.

The plugin has been released as version 2.4.1 because there was a previous experimental plugin with the same name. That experiment was not a success, and we abandoned it some time ago – the new plugin is a complete rewrite. The version of the plugin will track the version of bnd itself, so 2.4.1 of the plugin builds against 2.4.1 bnd, which is released and available from Maven Central. From now on we will work on version 3.0, which will track version 3.0 of bnd.

The plugin generates the JAR manifest into target/classes/META-INF/MANIFEST.MF, but by default the maven-jar-plugin ignores this file and replaces it with an empty, non-OSGi manifest. In order to pick up the generated manifest it is necessary to set the following configuration, which can be done just once in your parent POM:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jar-plugin</artifactId>
    <configuration>
        <useDefaultManifestFile>true</useDefaultManifestFile>
    </configuration>
</plugin>

The plugin is available now from Maven Central or jCenter. It is not yet thoroughly documented, although bnd itself does have very good documentation. All options from bnd automatically work in bnd-maven-plugin, with the exception of “sub bundles” (-sub) since Maven projects are meant to produce exactly one output artifact per project.

To add the plugin to your project, insert the following into your POM:

<plugin>
    <groupId>biz.aQute.bnd</groupId>
    <artifactId>bnd-maven-plugin</artifactId>
    <version>2.4.1</version>

    <executions>
        <execution>
            <goals>
                <goal>bnd-process</goal>
            </goals>
        </execution>
    </executions>
</plugin>

We have example projects to get started. Questions, problems or offers of help can be can be directed to the bnd/Bndtools mailing list.

Static Linking in OSGi

2014-05-26T12:00:00+00:00

When building native executables in a language like C or C++, we have a choice about how to deal with the libraries our code depends on. We can choose to link them statically, meaning that the library contents are physically copied into the executable file at build time, or we can link dynamically which means the library contents must be found externally to the executable by a special runtime component called the linker.

Static linking has the great advantage that the executable can rely on always having the library it needs, and it’s always the correct version. Installation is a breeze because there’s just a single file to copy. The clear downside of static linking is that commonly used libraries will be duplicated in lots of executables, and therefore take more disk space and perhaps memory.

In standard Java, a JAR is a kind of executable that has dependencies on libraries. However static linking is unheard of in this world because it is terribly unsafe. All JARs sit together in a global classpath – if one JAR contains a copy of a library and another JAR contains a copy of the same library then the first one on the classpath will always “win”. If the two copies are of different versions then chaos ensues.

Therefore the state of the art in standard Java is a kind of dynamic linking, except without any metadata in the JARs to declare exactly what we should be linked with… and no linker! Instead, developers often have find dependencies through a process of trial-and-error: keep adding JARs to the classpath until all the NoClassDefFoundErrors go away.

OSGi applications predominantly use dynamic linking as well. It’s a lot more manageable because we have precise metadata defining the dependencies, and the OSGi Framework itself can be seen as a kind of linker. However, static linking is also perfectly possible and safe because of the isolation provided by OSGi, and it has many of the same advantages as statically linked native executables. Bundles with fewer dependencies are simply easier to manage. To put it another way: OSGi is the best system I have ever seen for managing dependencies, but the easiest artefacts to manage are still those with no dependencies at all.

Static linking in OSGi is sometimes done by means of the Bundle-ClassPath, which allows us to embed an entire library JAR inside a bundle. This works, but it can inflate the size of our bundle quickly because we pull in the entire JAR rather than just the parts we need. An alternative solution is to use Private-Package to include individual packages out of the dependency JARs. However once a package is added this way it becomes a permanent part of our bundle until we explicitly remove it from the Private-Package list. In the future our core code might change such that some or all of these packages will not be needed. Will we remember to review Private-Package regularly to ensure everything there is actually still used? Not likely.

A Better Way

And so we come to a little-known bnd instruction called Conditional-Package. This instruction is used to include a package or set of packages if (and only if) they are used from the core packages of our bundle -– where core packages are defined as those that are listed in Private-Package and/or Export-Package. We can even use wildcards, and bnd will calculate the minimum set of packages that need to be included. For example, many of my bnd files include a line like this:

Conditional-Package: org.example.util.*

where the org.example.util prefix is for a whole set of packages designed as generic utilities.

Service vs Library

Static linking with Conditional-Package can help to resolve a common head-scratcher when designing OSGi bundles.

Sometimes we want to write a piece of utility code that will be used across a number of other bundles, but that doesn’t seem to warrant a service interface. Services are great for application components that can be implemented in more than one way, but sometimes there is only one conceivable implementation. A classic example is encoding, e.g. Base64. When a client wants to convert to/from Base64 representation, it doesn’t want to have an alternative encoding swapped in at runtime! In this case the client is quite happy to be tightly coupled to a specific implementation of the function it is calling.

It feels wrong to export a package that contains implementation code in OSGi. Ideally, exports are for service interfaces, i.e. contracts. Implementation code is subject to rapid change and is hard to version correctly (tools like bnd’s baselining can’t detect semantic changes in executable code, only on method signatures). Nevertheless we still want to reuse the Base64 encoder across all the bundles that need it. So we use Conditional-Package to include that functionality directly. No versioning headaches because the bundle always gets the exact version was built with.

Incidentally this practice doesn’t violate the DRY (Don’t Repeat Yourself) principle because the source code is still defined in just one place; it’s the compiled class files that are copied.

Contraindications

A few words of warning. Like all power tools, Conditional-Package is dangerous if misused. When we pull a package into our bundle we inherit all of its dependencies, so it is best to do this with small, coherent packages that do not have a large tree of transitive dependencies. Try to design your packages so that they do only one thing each… don’t make a big generic util package, instead make subpackages such as util.xml, util.io, util.encoding and so on. Note that Conditional-Package will pull in any transitive package dependencies that match the pattern we have specified – all others are treated as external dependencies and end up in Import-Package.

Using a qualified or prefixed wildcard such as org.example.util.* is okay. Using a bare wildcard is not okay. I have seen this done accidentally: as a result, the bundle contained large chunks of the JRE as well as OSGi core packages.

Another danger is if we pull in packages that are used by an exported interface. Take the following example:

package b;
import a.A;
	
public interface B {
     void doSomeThing(A a);
}

This should be familiar to OSGi developers as a uses constraint, because B communicates type A from package a. It would be very bad to include a private copy of package a in our bundle, because it would not be compatible with other bundles’ idea of what the package looks like. Actually bnd will report a build warning when it sees an exported package having a uses constraint on a private package, so you easily can avoid this situation.

Conclusion

Static linking in OSGi can improve your bundles, making them safer and easier to deploy. As always there is a trade-off and the technique can lead to larger bundles overall, but this can be mitigated by keeping packages small and coherent.

Maven Support in Bndtools — Future Directions

2013-06-13T12:00:00+00:00

Enhancing support for Maven is a perennial topic amongst Bndtools developers and users, and during the Bndtools Hackathon last week we discussed the topic in-depth. As a result, we have determined that there will be two broad approaches towards Maven support in Bndtools. The choice between these approaches will be largely a matter of taste, and the user’s attitude and motivation towards the tools.

“Maven First”

One set of developers comes from the Maven world and is generally happy with the “Maven Way”; they do not want it to change significantly. We feel that these users are already well served by the Felix Bundle Plugin, m2eclipse and even other IDEs such as NetBeans.

Bndtools cannot and should not compete with those tools as a generic Maven-centric IDE; yet we can still add value. For example even when using m2eclipse, we can offer our wizards for creating Declarative Services or Blueprint components, and we can provide a great way to launch, test and debug OSGi applications from within the IDE.

Also our support for the OSGi R5 Repositories and Resolver specifications means that you can work with bundles built by Maven and deployed to a repository – e.g. Nexus, Artifactory, or even just the bundles installed in your “local” repository – resolve your top-level bundles and generate an application descriptor. This descriptor can then be fed back into the Maven build chain to create a fully assembled and deployed application. We feel we can do a much better job of this than other tools because we take full advantage of OSGi’s rich Capabilities and Requirements dependency model.

We call the above a “Maven first” approach. To make it work, certain parts of Bndtools will have to be decoupled from the internal build system so that, for example, we can resolve and run without having to be inside a bnd-style project. This decoupling is anyway good for Bndtools’ internal modularity, and we expect to complete it in time for the 2.2 release.

“Bnd First”

The second approach is for developers who prefer a more OSGi-centric development process, and who want to use the advanced build-time features of bnd/Bndtools while still using a fully Maven-based offline build.

For example one of the most exciting features we are working on in 2.2 is baselining, which (briefly) means the tool helps to ensure your packages and bundles are correctly versioned according to the OSGi Semantic Versioning guidelines, by breaking the build when versions are incorrect, and offering quick-fixes to get it back into shape. Ferry Huberts has written an overview on his blog.

Another feature that Bndtools has always offered is very fast incremental builds, which are integrated with the launching subsystem so that your code is compiled, built and already running in your application as soon as you save it. Incremental building is always a problem with Maven, and even m2eclipse doesn’t do it very well (though it seems to be improving gradually).

These features are hard-to-impossible to implement given the control (or rather, lack of control) afforded to us by the existing Felix Bundle Plugin. That plugin basically only creates the JAR file during the packaging phase, whereas in Bndtools bnd maintains the build model so that it can be used in Eclipse, Ant, Maven, Gradle and others.

Therefore we are working on a new plugin for Maven that allows bnd to take more control. We call this the “bnd first” approach. Actually the plugin was started by Toni Menzel, and Peter Kriens has begun enhancing it. So far it looks extremely promising, but unfortunately it is still quite experimental and will probably not be release-quality in time for 2.2. If you are interested in trying it before then, please engage on the Bndtools mailing list.

Other Tools

As ever, enhancing Maven support does not mean that we are reducing our support for existing Ant builds. We will continue to offer a modular Ant-based build system with our project templates, and in fact some of the work done at the hackathon by PK Søreide helped to improve the performance of our Ant tasks by a factor of three to four.

Also, Paul Bakker worked on a new Gradle build template, which turned out up to ten times faster than the old Ant build! We hope to have this ready in 2.2.

So our focus continues to be this: enabling you to be as productive as possible as an OSGi developer, irrespective of your choice of build tool.

Bndtools 2.1 Released

2013-02-11T12:00:00+00:00

GUEST POST: This post was written by my friend and open-source-colleague Ferry Huberts. Read the original here.

We just released Bndtools version 2.1. We’re very proud of it and are convinced that it will make your life as an OSGi developer much easier. Easier than any other tool around.

All together we believe that Bndtools is now the foremost tool for developing OSGi applications, and continues our mission to make OSGi development easier and more productive than traditional Java development.

We’ve been tuning our development flow so that we can do releases on a shorter schedule and it seems to be working: it’s been 11 weeks since the release of 2.0.

This new version has many improvements that - not only - make it faster again, but also expand the functionality to offer a more complete experience. We’ve focused on bug fixes with an increased attention to details and also added some new functionality.

A brief overview of some noteworthy changes:

You can now specify command line arguments on the Run tab, which enables running/debugging in Eclipse with exactly the same arguments as the in the bug report that one of your users just reported :-). More details here.
Bndtools will now (by default) generate .gitignore files when you create new Bndtools projects so that you don’t commit derived files. This can be switched of in the Bndtools preferences panel.
The JAR viewer now by default opens the manifest file and also remembers the selected file so that when the JAR is updated, it will reload the JAR and select that file again.
Bndtools now finally fully supports paths containing whitespace. This was a long standing bug that affected launching from Bndtools. See also the notes below.
A compiler error is now generated on incorrect usage of the @Reference annotation. An example: you have a setter (e.g. add) for a dynamic dependency, but no unsetter (e.g. remove).
We again improved the build speed.

Of course there are many more gems in the full list of changes.

Important Notes:

We’re deprecating running from bnd.bnd files, which is why we’re no longer showing the ‘Run Requirement/Resolve’ panel on the Run tab. Users are strongly advised to migrate their run settings into .bndrun files. Expect the Run tab to be removed completely for bnd.bnd files in a future release.
Existing workspaces should update their cnf/buildrepo/biz.aQute.launcher to version 1.0.6 to fix issues with paths containing whitespace. Download it here, or generate a new cnf project by removing the existing one.

Installation:

You can install Bndtools into Eclipse through the marketplace or use our Eclipse update site, see further instructions.

For those of you that want to live on the (bleeding) edge, our latest stable build lives here

As usual, please report any issues you find on our Github bug tracker: https://github.com/bndtools/bndtools/issues

Bndtools 2.0 Released

2013-02-11T12:00:00+00:00

After a long and grueling process, I’m pleased (and considerably relieved) to announce that Bndtools 2.0 has been released!

There are many, many improvements in this release, which I will only summarise below; see the release notes for more details. All together I believe that Bndtools is now the foremost tool for developing OSGi applications, and continues our mission to make OSGi development easier and more productive than traditional Java development.

Support for OSGi Release 5 Resolver and Repository specifications;
Export run descriptors as standalone executables;
Baselining (build errors for incorrectly versioned bundles);
Enhanced Semantic Versioning, using annotations for Consumer and Provider roles;
Exported Package Decorations;
Improved Incremental Builder;
Support for Apache ACE;
Lots and lots of bug fixes and performance improvements (e.g. Conditional-Package processing was sped up over 100 times!).

Version 2.0 has been such a huge release because of the sheer amount of changes introduced in both bnd and in Bndtools. In the future we expect to make incremental releases much more frequently, and this includes a commitment not to change the structure of Bndtools workspaces (or if they need to change, to provide tools that make the migration transparent).

Bndtools 2.0 is available now on the Eclipse Marketplace. For other installation options refer to the installation guide.

No Solution for Complexity?

2013-02-04T12:00:00+00:00

Yesterday I read a fascinating article on the BBC News website about the complexity of banking software. To summarise: last year saw a number of severe IT glitches at banks causing outages in payment processing and other problems, and the article predicts that such glitches will get more common in the future due to increasing sofware complexity.

Here’s the part that got my attention:

The core of the problem is that the business software used by the institutions has become horrifically complex, according to Lev Lesokhin, strategy chief at New York-based software analysis firm Cast.

He says developers are good at building new functions, but bad at ensuring nothing goes wrong when the new software is added to the existing mix.

“Modern computer systems are so complicated you would need to perform more tests than there are stars in the sky to be 100% sure there were no problems in the system,” he explains.

Sigh.

If only there were a way to create “firewalls” between different parts of a large system, so that we could be absolutely sure that the functionality within each firewall cannot break merely from adding new functionality outside it. Then we could know precisely the scope of any change, and test only the things that can potentially break rather than the entire universe.

Of course there is a way to do this. It’s called modularity. Sadly it’s not very popular because it requires planning and discipline – two words that developers seem to be averse to, even though they pay off handsomely by averting cock-ups in deployment. Also to be fair to the developers, it requires strategic investment in infrastructure, which has been sorely lacking. Businesses always have difficulty justifying measurable up-front costs leading to unknown future gains. So let me tell you a story about real, measurable gains from modularity.

A few years ago I worked for a manufacturer of medical devices. These were large, powerful machines that could quite easily kill a patient if they were to malfunction. Incidentally, Mike Milinkovich tells a terrifying story about how he was nearly killed by a software bug… all things considered I prefer banks; when they go wrong the only thing you lose is money.

Naturally, the business was heavily regulated. Whenever a module of the system (hardware or software) was changed, it would have to be thoroughly retested, and those tests certified by the regulator. The process was terribly slow and expensive, and it would have to be done not just for the changed module but for all the other modules that could potentially be affected by it. By using a strong modularity solution, the manufacturer could prove to the regulator that a change in module A could not possibly affect the working of module B, and therefore module B did not need to be retested. The engineering effort required to achieve this was not cheap, but it ultimately saved them millions of dollars and allowed them to bring new features to market quicker.

Modularity is a technique and a mindset as much as it is a specific technology. Nevertheless there is one technology that is mature, well proven and has stood the test of time: OSGi. Another quote, this time from Peter Kriens:

OSGi … is a messenger that tells you that your code is ugly. In more traditional environments you would have this fuzzy warm feeling that all was ok due to the absence of warnings and errors. The power of OSGi is that it actually allows you to go to a warm and fuzzy place that is not fiction (like all other Java build environments) but based on hard facts.

Look it up, and perhaps you could save your bank from making headlines for all the wrong reasons.

The Dreaded Thread Context Class Loader

2012-10-23T12:00:00+00:00

Tomorrow at OSGi Community Event 2012 I will be talking about “How to make your library OSGi friendly without depending on OSGi”. During this talk I will make reference to Java’s Thread Context Class Loader (TCCL), but since it is only a 25-minute talk I will not have time to go into great detail on this subject. Therefore I thought it would be helpful to write this post with some of the background information.

Much of the technical information and research in this post is derived from an old, unpublished OSGi RFP authored by Peter Kriens. I have used it with his permission.

History

The thread context class loader (TCCL) has a very interesting history in Java.

Java defines a hierarchy of class loaders, and in a typical Java runtime the “application” loader at the bottom is aware of the “classpath” (as supplied via the -classpath/-cp switch), and the boot loader is aware of the JRE classes in rt.jar etc. There is also an “extension” class loader in the middle, which loads from the JRE ext directory. Class loaders delegate upwards to their parent, which means the application loader can see any class offered by the boot loader but the boot loader cannot see the application classes.

Sun first ran into problems with this scheme when implementing Java Serialization, as used in Remote Method Invocation (RMI). Deserializing data from a stream as Java objects required knowledge of the application classes, but the deserialization code was loaded by the boot loader; therefore it would not normally have visibility of the classes. The problem was fixed by adding private native methods to the JRE that would allow inspection of the call stack in order to find the first non-null class loader.

However, many other extension libraries soon ran into the same problem, and they could not (and should not!) all solve it with private native methods in the Sun JVM. At about the same time, J2EE became very important. J2EE is an application model where the Java code runs in a severely restricted environment. Applications run in a “silo”. Each application runs in a separate class loader and threads may not cross application/silo boundaries. Applications can use the extensions provided by the VM and the libraries provided by the container, but they cannot use any code from each other. This model also suffers from the problem that the libraries provided by the container cannot access the application code.

Therefore in Java 1.2 the TCCL was introduced: a class loader that is associated with a thread, as a thread-local variable. This means that any libary can at any time access the “current” context loader by asking the calling thread. This context loader is expected to have visibility to the specific application class loader that is responsible for the call, and therefore visibility of the application classes.

Sun also started to use the TCCL heavily itself, though without proper specification or guidance. The 1.5 JDK has 79 references to Thread.getContextClassLoader(). Since Java 1.4, the Java VM was modified to use the context class loader in JNDI, JAXP, CORBA, JMX, Xalan, Xerces, AWT, Beans, SQL, Logging, Prefs, RMI, Security, Swing, and most of the XML subsystems. Additionally most middleware libraries like Hibernate, Saxon, Jakarta Commons Logging and many others started to use the TCCL as either the first or last resort. Java has provided virtually no specification or guidelines on the use of TCCL. This has caused almost every library to follow a different strategy.

The key questions to answer regarding the correct use of TCCL are:

when should it be set, and by whom?
of what set of classes should it have visibility?

From a J2EE point of view these questions are relatively easy to answer because the programming model is so constrained. The container owns all the entry points; i.e., it controls the HTTP thread running the Servlets and the RMI sockets for EJBs, creation of additional threads is forbidden, etc) and it can therefore ensure that the TCCL is always set on entry into the application code. Since applications are entirely self-contained, the set of visible classes is simply all classes owned by the application.

In a modular runtime such as OSGi these questions are far harder to answer. Bundles are free to create their own threads and entry points, and there is no general way to identify the set of bundles that contribute classes for an “application”; indeed, the very concept of an “application” is much harder to pin down. We could constrain the programming model and require that an application be deployed as a single bundle with no dependencies, but this would negate most of the benefits of using OSGi. Therefore OSGi does not attempt to define what the TCCL should be at any particular time, and typically it is left null.

Alternatives to TCCL

In our own OSGi-compliant code, or in any library with explicit OSGi support, we need not worry about the TCCL, because OSGi Services provide a far cleaner approach than any ClassLoader-based approach to loading extensions. However, legacy third-party libraries are still an issue. Many such libraries attempt to load application classes by name at runtime: for example, Hibernate reads class names from .hbm.xml files and then creates instances of those classes for each database record.

When you move such a library to any kind of modular environment – including OSGi, JBoss Modules or even Jigsaw – you find that it breaks, because the name of a class is not sufficient to uniquely identify it. The identity of a class consists of its fully qualified name AND the class loader that defined it, which in OSGi usually corresponds to the bundle that contains it. So in addition to the name, we need to know the class loader from which to load the class. Due to the wide variety of class loading environments created by various application servers, many libraries attempt to solve this problem with a set of heuristics. Consulting the TCCL is usually one of the heuristics, along with checking the library’s own class loader, the JRE extension class loader, etc.

If a library only consults the TCCL then we are somewhat stuck: we will have to explicitly set the TCCL from our own code before calling into the library. Fortunately this is rarely the case, for example most such libraries will also call Class.forName(), which means the library will consult its own class loader to load the class. This is still far from ideal, but we can work around it by deploying a single fragment rather than scattering calls to setContextClassLoader() throughout our code. Far better is when a library eschews class names entirely and allows Class objects to be passed; or at least offers an API method to set the class loader from which the class names should be loaded.

Unfortunately it’s very hard to predict in advance – at least without closely inspecting the code – what kind of heuristics the library employs, again because of the complete lack of specification and guidance in this area.

Summary

I hope you will attend my talk tomorrow, which is mainly addressed to authors of Java libraries that want their code to work well in OSGi, but not only in OSGi. As you can doubtless infer from this blog post, avoiding the TCCL and other weird ClassLoader-based hacks is a big part of being OSGi-friendly, but I will also talk about service dynamics and configuration issues. See you there!

Jigsaw Postponed

2012-07-19T12:00:00+00:00

The dust is now settling on the announcement that Jigsaw – the prototype project that was widely expected to become the Java SE 8 Module System – will be deferred to Java 9. This is just the latest delay to a promised feature that started way back in 2005 with JSR 277, was supposed to be delivered in Java 7, then in Java 8, now in Java 9… it’s not unreasonable to ask whether Java Modularity will ever happen.

Believe it or not, I am disappointed. It’s true that some in the OSGi community consider the news to be unequivocally a good thing: the removal of a competitor can certainly take the pressure off. However we need pressure to raise our game and continue improving the usability and productivity aspects of working with OSGi. Frankly if Jigsaw could truly do a better job than OSGi then it would deserve to take over.

The big problem with Jigsaw is not that it competes with OSGi: as currently defined, it does not. The problem is scope creep.

First an unavoidable fact: whatever module system is used to modularise the JRE itself must be able to run inside the JRE, i.e. it has to be part of the core. OSGi is a general-purpose module system that is conceptually very simple (export a package, import a package) but due to the nature of real systems it has to handle an awful lot of edge cases. These edge-cases and the OSGi support for them have been built up over around 14 years of putting OSGi into real production systems, but they contribute to the size of a typical OSGi framework (480k for Apache Felix 4.0.3) and to the length of the specification document. In an enterprise application, and even in many modern mobile applications, 480k is nothing to worry about, but it’s still too big for the inner sanctum of the JRE.

Add to this that the JRE has some very unusual problems that don’t affect normal libraries and applications, and OSGi’s general-purpose module support doesn’t look so useful for JRE modularity. In fact the JRE is almost nothing but a bunch of edge-cases. So there’s a good case for putting a special-purpose module system into the JRE that supports only the JRE requirements… and that’s what Jigsaw was meant to be. Incidentally if you took OSGi and stripped out all the general purpose features, leaving in only features needed for the JRE… you’d get something surprisingly close to Jigsaw!

Unfortunately as they say, when all you have is a hammer everything looks like a nail. After building themselves a shiny new hammer for the JRE, the Jigsaw developers started to see nothing but nails in the Java application and library space. Once you step into that space though you need the general purpose solution with all the edge-cases. I wasn’t really concerned that Jigsaw was heading to take over this space because it was so clearly deficient, though I was concerned about developers who would waste huge amounts of time and money trying to get it to work. (Incidentally I have been accused of opposing Jigsaw for selfish reasons, since I make most of my living from OSGi. On the contrary: in fact I’m confident that Jigsaw would double my consulting rates, since so many people will need rescuing from tangled messes. That’s just not how I want to make a living.)

So because of this scope creep it looks like we will not get the benefits of a modular JRE until at least 2015. This is a great shame for developers using OSGi just as it is for everyone else developing on the Java platform. For starters, who wouldn’t like to have the ability to strip down the JRE to just the core functionality that we need?? Also the way OSGi interacts with the JRE today leaves a lot to be desired. Ideally an OSGi bundle wants to depend entirely on versioned packages, but due to a JVM security restriction we have to treat the entire java.* namespace differently. The best we can do is use the Bundle-RequiredExecutionEnvironment header to assert the version of Java our bundle requires, i.e. Java 5 or Java 6 etc. This kind of works but is a very blunt tool. Another problem is that the other packages from the JRE – javax.swing, org.w3c.dom etc – are all completely unversioned. Then you have garbage like the inclusion of package javax.transaction since Java 1.3, but only a small subset of the classes that belong in that package… this sounds like a trivial thing but it breaks an amazing number of libraries.

A modular JRE could have fixed these problems, allowing OSGi bundles to require a JRE containing a specified set of versioned modules. We could then perform compile-time validation that only the required APIs were actually used within the bundle, and we could even use the OSGi R5 Resolver to work out which set of JRE modules to install when provisioning that bundle. I do still hope we can get to this point in the future.

In conclusion I have a message to the OSGi community: this is not the time to sit on our laurels. Yes, we are the only realistic solution today for Java modularity, and we have just been given some extra time to consolidate our lead. However OSGi is still not as easily usable or as productive as it should be – if we don’t address these issues now, then we deserve to be forgotten after Java 9.

Are We There Yet?

2012-03-21T12:00:00+00:00

Peter wrote a blog post yesterday about the OSGi Coordinator service, which is new in the R4.3 Compendium. I wanted to give a more concrete example of how it is used.

Sometimes we want to perform an expensive operation after a bunch of things have changed. For example, in the writable OBR repositories used by Bndtools we need to regenerate the index XML file after the user deploys a new resource into the repository. However, what if the user is deploying a hundred resources? It would be wasteful to regenerate the index after each of those deployments, because 99 times out of the hundred we will throw away the result.

The Coordinator service is therefore used as a way of chunking operations. When deploying a file to the repository, we check if a coordination is ongoing; if one is, then we join it and defer the regeneration of the index. Otherwise, we just regenerate the index immediately.

Here is the abbreviated code for put(), i.e. the method that does the deployment:

public File put(Jar bundle) throws Exception {
	File newFile = storageRepo.put(bundle);
	newFilesInCoordination.add(newFile);

	if (coordinator == null || !coordinator.addParticipant(this)) {
		finishPut();
	}
	return newFile;
}

When the coordinator service exists AND there is a current coordination, indicated by addParticipant returning true, we simply begin the put operation and remember the file that we have created in the context of this coordination. However when there is no coordinator or current coordination, we both begin and immediately finish the put operation. The finishPut method looks like this:

private void finishPut() throws Exception {
	regenerateIndex();
	newFilesInCoordination.clear();
}

In order to join the coordination we have to implement the Participant interface, which has two methods ended and failed. The ended method is very simple, we just call finishPut:

public void ended(Coordination coordination) throws Exception {
	finishPut();
}

In the failed method we clean up the changes that we started to make. The repository will be back in the state it was in before the put method was called:

public void failed(Coordination coordination) throws Exception {
	for (File file : newFilesInCoordination) {
		file.delete();
		// omitted some extra error handling and logging here
	}
	newFilesInCoordination.clear();
}

The thing I like about Coordinator is that it provides a very useful optimisation, but things can still work without it. In this case, when there is no Coordinator service available the repository regenerates the index on every put… which is exactly the behaviour that it had before I added Coordination support. Simply by adding a Coordinator bundle I can improve the performance of bulk repository updates.

There is a Coordinator implementation available in Equinox 3.8M6. As far as I know Felix does not have its own implementation yet, but the Equinox Coordinator works fine on Felix in my testing.

Incidentally, the use-cases for the Coordinator may remind you of the archetypal chunking problem in OSGi: when should the framework resolve bundles after a series of install, update and uninstall operations? Bundle resolution is not merely expensive but may actually produce spurious errors if done at inappropriate times, so the OSGi framework has always had the ability to chunk these operations: we call FrameworkWiring.refreshBundles method (or PackageAdmin.refreshPackages() prior to R4.3) to signal when we are done. The Coordinator service generalises this idea to make it available for our own applications.

Free OSGi BOF at DevCon 2012

2012-01-12T12:00:00+00:00

The programme for the Birds-of-a-Feather (BOF) session at OSGi DevCon 2012 – co-hosted with EclipseCon 2012 – has been announced. I will be speaking about the Java 8 module system, Jigsaw and how they relate to OSGi: what are the differences; the pros and cons of each in different environments; and the potential for them to work together.

There will also be sessions on: OSGi Subsystems; simplifying development with the upcoming OSGi Bundle Repository (OBR) standard; and finally a surprise session from a mystery speaker.

The BOF is free to attend, you don’t need to be registered for OSGi DevCon/EclipseCon. So if you’re in the Washington DC area and interested in learning more about OSGi or meeting several of the top OSGi experts… you know where to go.

OSGi Training in 2012; London, NY, DC and Sydney

2012-01-11T18:00:00+00:00

I will be running a series of OSGi training courses this year, with the first two being held in London and New York in February. The early bird pricing for London will expire in the next couple of days, so please do hurry to secure your place!

If you’re in North America and can’t make NY in February then there will be another chance to take the training in the Washington DC area (very close to Dulles airport). Then later in the year I will be down in Sydney, Australia.

You might be wondering about the “Masterclass in OSGi” series that I previously ran with Peter Kriens, who recently announced his resignation from the OSGi Alliance. Peter is very busy completing his duties for the Alliance and then will be fully engaged with his next venture, so he has told me that he is not able to participate in the masterclasses for the foreseeable future.

My new course is heavily based on the subjects and discussions that arose from the masterclasses that took place in 2010/11, though it is intended to be more accessible for OSGi beginners. I still believe it’s the absolute best way to quickly achieve a deep understanding and mastery of OSGi and modularity in Java, and I very much look forward to seeing you there.

Eclipse Dock Icons on Mac OS X

2011-10-09T18:00:00+00:00

If you’re a heavy Eclipse user on Mac OS, you’re probably just as sick as I am of seeing the following in your Dock:

… and the following in your Application Switcher:

How on Earth am I supposed to keep track of all those icons?? Well I just can’t, and I’ve been struggling with this nonsense for years. I’ve tried changing the Dock icon, and the name of the application, but these settings are still shared by all the workspaces that run with that copy of Eclipse.

Enough is enough, I need a per-workspace label for my dock icons, so I created a plug-in that I’m calling the Workspace Badge Plug-in for Mac OS. It looks like this:

… and this:

By default it uses the last segment of your workspace path, but there isn’t much space so that often gets shortened with ellipses. So you can set your own short string in the workspace using a preference page:

The amazing thing is, this is probably the tiniest plug-in I have ever written for Eclipse, and is likely to be one of the most useful! If you want to try it out, install from the following update site URL (Marketplace listing coming soon):

http://macbadge-updates.s3.amazonaws.com/

The plug-in has not been exhaustively tested. It works for me on Eclipse 3.7 Indigo. I have no idea what would happen – and take no responsibility for what might happen – if you try to install it on an OS that is not Mac OS.

Why OBR?

2011-09-12T11:45:00+00:00

Pascal Rapicault has written a blog post on his reasons for not choosing OBR as the provisioning technology for Eclipse.

I have absolutely no wish to start a fight, certainly not between myself and Pascal whom I regard as a friend. However it would be remiss not to correct the errors in his post, because that could leave the impression that OBR is an unworkable technology. The fact is that, today, both OBR and p2 are workable technologies for provisioning OSGi applications, but I would argue that OBR is cleaner and more usable, and much easier to integrate. That’s why I am incorporating OBR support into Bndtools at pretty much every level.

The OSGi Bundle Repository draft specification defines a repository index format and a resolver API. It allows us to do the following:

Add a repository index (or multiple indexes) to a resolver.
Specify the starting environment, for example which platform, which Java version, etc.
Add a set of requirements: what we want to get out of the resolver.
The resolver calculates the set of required resources that will satisfy those requirements. It also reports optional resources that may enhance the functionality of the selected set.

To address Pascal’s arguments in turn:

OBR focus was bundle centric. No “group” (aka features), no native files, …

Incorrect. An OBR repository can reference any kind of resource. The dependencies between resources are modelled as abstract “requirements” and “capabilities”. So a group/feature can be modelled as a resource that requires a bunch of other resources using their “name” capability.

Naturally, most extant OBR repositories overwhelmingly contain OSGi bundles, but the same is true of p2.

The repository format was fixed and was XML only, no possibility of extension.

Yes the repository format is XML, but it is quite extensible thanks to the generic requirement/capabilities model.

OBR’s format has the advantage of being a single file, repository.xml, compared to p2’s twin files content.xml and artifacts.xml, and it is more readable than either of them.

Each bundle listed in the repository was referring to the payload (the actual jar for the bundle) using a fix URL rather than abstracting the artifact location in a coordinate – like Maven does with its concept of GAV or like p2 does with “artifact keys” – which makes mirroring just plain painful (I know it is possible to provide URL handlers, but come on…).

Incorrect. OBR repository indexes use URIs, not URLs. It is up to the downloader to resolve those to a physical location, just as p2’s downloader must resolve its abstract “coordinates”. Mirroring is therefore perfectly possible. Indeed there is no functional difference between a URI and a Maven “coordinate”, and Maven coordinates can be trivially encoded as URIs.

The provided API was not allowing for transactional state change (some bundles could be installed, but not all the required ones if the download failed).

OBR does not specify how to download resources, it only calculates which resources are required to satisfy the question we asked of it. Transactional behaviour can be built into the code that we use to download those resources. So while this was missing from OBR when Pascal evaluated it, he could have chosen to implement a transactional downloader on top of OBR rather than redesigning the whole repository and resolver.

Since it is a standards document, OBR avoids mandating implementation details such as how to download resources, because whatever is convenient for Eclipse may be inconvenient or impossible in other environments.

The metadata did not accommodate the expression of what needed to happen to the bundle (e.g. should it be started, set the start level, etc.) and thus even less to other things.

State requirements – e.g. the requirement for the bundle to be in active state – can again be represented in the requirements/capabilities model. However it is admittedly non-obvious how to do this, and the latest (non-released) draft of the specification will make this area clearer.

The resolver always insisted on installing all the fragments available (which in complex repos like Indigo means that installing JDT will bring in parts of WindowBuilder since it has some fragment to JDT UI iirc)

The resolver tells us about available OSGi fragments as optional resources. The UI for a resolver (e.g. Eclipse Update Manager) should present those optional resources to the user and allow some or none of them to be selected.

Platform-specific fragments (e.g. SWT) can use the requirements/capabilities model to ensure they are not selected on an incorrect platform. For example, the Win32 SWT fragment would have a requirement of type platform and filter value (&(os=win32)(ws=win32)(arch=x86)). Before running the resolver we would add a “global” capability for the platform we actually want to resolve on.

Why not p2?

Almost nobody believes that p2 was ready to be released when it actually was released, back in Eclipse 3.4 (Ganymede). As a result, installation of plug-ins in Ganymede, and even in the following release (3.5 Galileo) was so error-prone that most Eclipse users reverted to unzipping files directly into their plugins directories.

P2 finally became sort-of usable in 3.6 (Helios), but it still does not support one of the most important aspects of OSGi resolution: uses constraints. That is, the p2 metadata do not support expression of uses-constraints and the resolver algorithm does not take account of them. As a result it is quite easy to get p2 to resolve a set of bundles but later find that those bundles cannot work in the actual OSGi framework.

Uses constraints are fully supported by OBR. It is still theoretically possible to generate a resolution set that does not resolve in OSGi, but this will always be possible unless we use the actual OSGi resolver itself, and under identical conditions. In practice, I have found OBR always produces high quality resolution sets so long as I feed it an accurate set of initial capabilities – in other words, GIGO.

The repository index generator for OBR is a simple command line tool provided in a single small JAR, which can also be used from ANT and Maven (it is already embedded in the Maven Bundle Plugin). In contrast, the p2 generator tool is an ANT task that only runs embedded inside Eclipse. This necessitates firing up a headless Eclipse application from an enclosing ANT build… and naturally you have to have a full Eclipse installation on your build machine.

Finally, last time I checked, the p2 resolver only ran on Equinox, not on any other OSGi framework.

On the other hand, there are certainly some valuable contributions in p2. The downloading, transactional updates, touchpoints etc would all still be useful in a provisioning system built on top of OBR’s repository and resolver.

The Future

Pascal is right to try to identify missing features in OBR, but I feel that he fell into the trap of mistaking one particular implementation – the Felix OBR bundle as it existed back in 2007/8 – with the abstract features of the specification itself. All of the requirements he states could have been implemented alongside an improved OBR bundle rather than throwing out the baby with the bathwater

However, what’s done is done. I’m glad that Pascal has participated in the effort to update the OBR specification, and I hear from my contacts inside the Alliance that the new specification is close to release. At that point I will incorporate the new OBR into Bndtools, and continue to recommend it to my clients who are seeking an OSGi provisioning story.

Solving OSGi "Uses" Constraint Violations

2011-09-02T21:45:00+00:00

One of the thornier deployment problems we sometimes come across in OSGi is the dreaded “uses constraint violation”. I recently helped a client to solve one of these, and it occurred to me to document the problem and the approach to solving it.

Be warned that this post is quite long and boring. You may not want to read it all right now, but perhaps Google will bring you back here when you first encounter a uses-constraint violation in the wild.

What is a Uses Constraint Anyway??

You may have already read Glyn Normington’s excellent article, which goes into quite a bit of depth on the subject, but regretably without the assistance of pretty pictures, so if you are a visual thinker like me then you might find the following explanation more accessible.

Starting with the basics, in OSGi we have dependencies based on Java packages. Some bundle exports a package, possibly with a version number, and another bundle imports it, possibly specifying an acceptable range of versions:

This notation is borrowed from the OSGi specification: a black rectangle is an export and a white rectangle is an import. The surrounding yellow blobs are bundles. The line between the export and the import means that OSGi has chosen that specific export as a match for the import – they have been “wired” together. So B imports package foo (version 1.0) from A.

Let’s build this up. Suppose bundle B, in addition to importing package foo 1.0, also exports package bar (the version of bar is unimportant). The C bundle imports package bar and gets wired to B:

So far so good. Now let’s complicate things a bit more: let’s say that bundle C, in addition to importing package bar, also imports package foo… BUT it imports version 2.0. Wait, we don’t have foo version 2.0! Never fear, there’s another bundle that does export foo 2.0. We’ll call him D:

Are you surprised that this works? Perhaps not, you may have heard that OSGi supports multiple versions of the same library at the same time, and here it is in action. However there are limits: while we can have multiple versions of the foo package, we must still be able to construct a consistent “class space” for each bundle that has exactly one version of every class. The “class space” for bundle C is shown by the shaded blue area:

Notice how the shaded area avoids the import of foo inside bundle B. This is only possible if the package bar exported by B ~~has no internal dependency on foo~~ does not “expose” foo via its signature (thanks to BJ for this correction).

Exposing foo means that a type in foo is visible through the signature of a type in bar, for example:

package bar;

import foo.Foo;

public class Bar extends Foo {   // exposure via subclassing
    public Foo getFoo();         // exposure via method return type
    public void setFoo(Foo foo); // exposure via method parameter
}

If package bar does expose package foo, as in this example, then we have a “uses constraint”. We illustrate this with a little rubber band, like so:

Now bundle C cannot be resolved. The rubber band means we cannot exclude the import of foo 1.0 from the blue shaded area, i.e. C’s class space must contain foo 1.0. But it is not allowed to contain both foo 1.0 and foo 2.0, so C’s second import cannot be satisfied.

I hope this is clear enough, but I’m sure you’re wondering: how the hell do I fix this? We should certainly avoid tempting runtime options that turn off the uses constraint validation. Neither should we start mucking about trying to “fix” the manifests of third-party bundles. Both of these are hacks that only mask the problem. After all, package bar in bundle B really does depend on a specific version of foo, so we shouldn’t blindly force it to use a different version.

In this example the proper solution is to find an alternative provider of the bar package, one that is compatible with foo 2.0. For example maybe there is a bar version 2.0 available somewhere, that happens to be compatible with foo 2.0. We may need to change the import in bundle C to make sure it gets bar 2.0. If bundle C is our own bundle then this is a reasonable change to make.

How About a Real Example?

Good idea! Let’s move away from foos and bars and look at a real uses-constraint problem. We’ll take the specific example of the client I mentioned: he was trying to use Apache ActiveMQ from an Eclipse RCP application. In the RCP application bundle there was a single import of the org.apache.activemq package. At runtime the RCP bundle failed to resolve with the following error:

!MESSAGE Bundle org.example.rcp.activemq_1.0.0.qualifier [37]
was not resolved.
!SUBENTRY 2 org.example.rcp.activemq 2 0 2011-02-08 14:45:37.513
!MESSAGE Package uses conflict: Import-Package: org.apache.
activemq; version="5.4.2"

Here we see the most frustrating aspect of uses-constraint violations: the almost total lack of information about the cause of the problem! We need to put our detective hat on.

Our bundle, org.example.rcp.activemq, is failing to resolve because it cannot import package org.apache.activemq version 5.4.2. That package exists, it is exported by bundle number 33, org.apache.activemq.activemq-core:

osgi> packages org.apache.activemq
org.apache.activemq; version="5.4.2"<org.apache.activemq.activemq
-core_5.4.2 [33]>

The uses constraint violation tells us that something used by the package org.apache.activemq clashes with something else that is imported into our bundle. We can look at the imports of the ActiveMQ bundle:

osgi> bundle 33
...
Imported packages
  javax.annotation; version="1.0.0"<com.springsource.javax.annotation...
  javax.jms; version="1.1.0"<org.apache.geronimo.specs.geronimo-jms_...
  javax.management; version="0.0.0"<org.eclipse.osgi_3.6.1.R36x_...
  javax.management.j2ee.statistics; version="1.1.0"
       <org.apache.geronimo.specs.geronimo-j2ee-management_1.1_...
  javax.management.openmbean; version="0.0.0"<org.eclipse.osgi_...
  javax.management.remote; version="0.0.0"<org.eclipse.osgi_...
  javax.naming; version="0.0.0"<org.eclipse.osgi_...
  javax.naming.directory; version="0.0.0"<org.eclipse.osgi_...
  javax.naming.event; version="0.0.0"<org.eclipse.osgi_...
  javax.naming.spi; version="0.0.0"<org.eclipse.osgi_...
  javax.net; version="0.0.0"<org.eclipse.osgi_...
  javax.net.ssl; version="0.0.0"<org.eclipse.osgi_...
  javax.security.auth; version="0.0.0"<org.eclipse.osgi_...
  javax.security.auth.callback; version="0.0.0"<org.eclipse.osgi_...
  javax.security.auth.login; version="0.0.0"<org.eclipse.osgi_...
  javax.security.auth.spi; version="0.0.0"<org.eclipse.osgi_...
  javax.sql; version="0.0.0"<org.eclipse.osgi_...
  javax.transaction.xa; version="0.0.0"<org.eclipse.osgi_...
  javax.xml.parsers; version="0.0.0"<org.eclipse.osgi_...
  org.apache.commons.logging; version="1.1.1"<jcl.over.slf4j...
  org.apache.kahadb.index; version="5.4.2"
       <org.apache.activemq.kahadb_5.4.2 [30]>
  org.apache.kahadb.journal; version="5.4.2"
       <org.apache.activemq.kahadb_5.4.2 [30]>
  org.apache.kahadb.page; version="5.4.2"
       <org.apache.activemq.kahadb_5.4.2 [30]>
  org.apache.kahadb.util; version="5.4.2"
       <org.apache.activemq.kahadb_5.4.2 [30]>
  org.osgi.framework; version="1.5.0"<org.eclipse.osgi_...
  org.w3c.dom; version="0.0.0"<org.eclipse.osgi_...
  org.xml.sax; version="0.0.0"<org.eclipse.osgi_...
  org.w3c.dom.traversal; version="0.0.0"<org.eclipse.osgi_...
  javax.xml.stream; version="0.0.0"<org.eclipse.osgi_...
...

Wow, quite a long list, and I’ve not even included the optional imports! In the worst case we need to review all of these, and then possibly the transitive dependencies of the bundles that export those packages, in order to find the conflict. Nightmare! Fortunately there is a shortcut we can take.

Recall the conflict in the simplistic example arose because there were two versions of the foo package available. This is a necessary precondition for a uses constraint violation, and it gives us a really big clue:

Look for packages that have more than one exporter.

I’d like to be able to say that I jumped straight to the answer at this point, but truthfully I hummed and hawwed a bit first (oh well, it was billable time). I speculated that there might be two copies of the Commons Logging APIs, so I asked the OSGi shell:

osgi> packages org.apache.commons.logging
org.apache.commons.logging; version="1.1.1"<jcl.over.slf4j_1.6.1 [31]>
  org.apache.activemq.kahadb_5.4.2 [30] imports
  org.apache.activemq.activemq-core_5.4.2 [33] imports

Nope, just one export, and it’s imported by both org.apache.activemq.kahadb and org.apache.activemq.activemq-core. No problem there.

Then I realised: we’re running on Java 6! That’s significant because Java 6 gratuitously added a bunch of APIs to the base JRE library… APIs that were previously available as optional libraries. This is a rich seam of duplication! So my next guess was javax.xml.stream:

javax.xml.stream; version="0.0.0"<org.eclipse.osgi_3.6.1.R36x_... [0]>
  org.eclipse.ui_3.6.1.M20100826-1330 [2] imports
  org.eclipse.core.expressions_3.4.200.v20100505 [7] imports
  org.eclipse.ui.workbench_3.6.1.M20100826-1330 [10] imports
  org.eclipse.core.runtime_3.6.0.v20100505 [11] imports
  org.eclipse.help_3.5.0.v20100524 [23] imports
  org.apache.activemq.activemq-core_5.4.2 [33] imports

Again no. Cutting to the chase, the culprit was javax.annotation:

javax.annotation; version="0.0.0"<org.eclipse.osgi_3.6.1.R36x_... [0]>
  org.eclipse.ui_3.6.1.M20100826-1330 [2] imports
  org.eclipse.core.expressions_3.4.200.v20100505 [7] imports
  org.eclipse.ui.workbench_3.6.1.M20100826-1330 [10] imports
  org.eclipse.core.runtime_3.6.0.v20100505 [11] imports
  org.eclipse.help_3.5.0.v20100524 [23] imports
javax.annotation; version="1.0.0"<com.springsource.javax.annotation >
  org.apache.activemq.activemq-core_5.4.2 [33] imports

Bingo! The package is exported by two bundles: the first is the system bundle itself, which is responsible for exporting all the JRE packages (aside from java.*), and this version is being imported by all the Eclipse plug-ins.

The other version is exported by com.springsource.javax.annotation, which is one of the wrapper bundles available from the repository hosted by SpringSource. Our RCP application naturally depends on the Eclipse stuff, therefore it cannot get the ActiveMQ library because of this conflict. Time for another pretty diagram:

How Can This Be Fixed?

Another fine question! In our abstract foo/bar case the solution involved finding a provider of “bar” that was compatible with foo version 2.0.0, and then changing our bundle C to make sure it used that new “bar”. In this case, both ActiveMQ and the Eclipse bundle are third-party binaries that we should not change.

Our goal is to make both ActiveMQ and Eclipse import the same version. ActiveMQ cannot import version 0.0.0 from the JRE, but Eclipse can import version 1.0.0.

But wait a moment, there is no such thing as version 0.0.0 of javax.annotation! It’s nonsense. The OSGi system bundle’s job is to export all of the JRE packages but it has no idea what version numbers to use for those exports. Indeed, nobody knows what the version numbers of all the JRE packages are. There is documentation for some of them, but none of that documentation is normative, and anyway it doesn’t cover everything in the JRE. We really need a JSR or OSGi specification that tells us definitively. Anyhow because of the lack of information, the OSGi framework clumsily exports everything as version 0.0.0.

Therefore the solution involves configuring the OSGi framework in one of the following ways:

Removing javax.annotation from the exports of the system bundle altogether; this will make both ActiveMQ and Eclipse resolve from the SpringSource bundle.
Exporting the javax.annotation package from the system bundle as version 1.0.0; then we don’t need the SpringSource bundle.
Using a special bundle that pulls the javax.annotation package from the JRE (using Require-Bundle!) and reexports it as version 1.0.0.

The first two approaches require us to create a “profile”, i.e. a list of JRE packages, and pass it to Equinox using the osgi.java.profile configuration setting.

The third approach is quite tricky if we have to create the bundle ourselves, but there is an existing bundle in most Eclipse IDE distributions that does exactly this. Unfortunately the bundle doesn’t appear in the RCP target platform downloads, which is why my client went to the SpringSource bundle repository to find a provider for the package and ended up in the mess he was in.

What’s my recommendation? Really I prefer the second option because the third just feels like a hack to me. There is a further issue though: if you use Eclipse PDE and click the “Validate Plug-ins” button in the Run Configuration dialog, then PDE will report a missing import, because it doesn’t know about the JRE profile setting. Clearly a missing feature or even a bug in PDE, but because of this you may prefer the third option.

Another Rant About Require-Bundle

Notice the arrows in the previous diagram, they indicate a Require-Bundle dependency rather than an import/export wiring. It turns out that Require-Bundle is responsible for this uses constraint problem, so I might as well have another rant about it.

The packages command told us that javax.annotation from the JRE was imported by Eclipse core runtime, core expressions, UI, workbench, and help. But I happen to know that none of those bundles actually use the javax.annotation package!! I know this because Eclipse still runs on Java 1.4, where those annotation classes won’t even load. But Eclipse uses Require-Bundle as follows:

Require-Bundle: org.eclipse.osgi

By requiring the system bundle, Eclipse implicitly imports the entire JRE. This is a terrible idea, and it is only done in Eclipse because of backward compatibility and because the PDE tooling is so inadequate for working with Import-Package dependencies.

(In fact, the uses constraint problem in this example can be solved by switching the RCP plug-in to use Import-Package to get hold of its dependencies on the Eclipse APIs. Sadly PDE makes Require-Bundle so convenient, and Import-Package so inconvenient, that very users are disciplined enough. I admit that even I use it when I’m feeling lazy.)

In my opinion Eclipse 4 should be used as an opportunity to break with the uglier parts of Eclipse’s legacy. It’s admittedly hard to do that when you have an ecosystem of literally thousands of 3rd party plug-ins that would stop running on the new version… but must this crap really be maintained forever? Will Eclipse 5 make the break, or Eclipse 6? A big part of what ails Java itself is the legacy compatibility burden; I hope Eclipse can be bolder.

Anyway, ranting over, here comes the recommendation:

Eradicate Require-Bundle dependencies from your bundles.

Installation Order

Consider the following modification to the original abstract scenario. Bundle B now accepts a range for the foo package: either version 1 or 2. If these bundles are all installed at the same time, then C is now resolvable because both C and B can wire to the version 2.0 foo offered by bundle D:

In this scenario bundle A will be ignored. However now suppose that only bundles A and B are available at startup, with C and D being installed later.

At startup B will resolve to the single version of foo available, which is version 1.0.0 from bundle A. When C and D are installed, C will be unresolvable, since it cannot import bar from B at the same time as it imports foo from D. OSGi tries to keep the existing resolved bundles stable, so it does not force B to rewire its import of foo to the new version.

This behaviour can be counter-intuitive. OSGi never dynamically rewires imports when new bundles are installed, it only tries to find wirings for the newly installed bundles. This is because rewiring is expensive: it can force the shutdown and restart of many bundles, and just calculating all the wirings again takes some time (it is NP-complete).

We can force OSGi to rewire everything by issuing the refresh command at the shell. If we do this, B’s wiring will switch to use foo 2.0 from D, and C will resolve. Therefore to check whether you have a problem with installation order:

Just type refresh and see whether the problem goes away.

Any More General Advice?

Solving a uses-constraint violation requires us to find an accurate diagnosis first, before we attempt to prescribe a cure.

Once the diagnosis is confirmed, seek a cure based on configuring the platform rather than messing with bundle manifests: uses-constraint violations are signals of an invalid deployment, i.e. a set of bundles that cannot work together. They are usually not signs of a development problem to be fixed within the bundles themselves – unless the bundles can be shown conclusively to have inaccurate or invalid manifests.

Regarding diagnosis, a necessary precondition for any uses-constraint violation is multiple exporters of the same package. So the first question to ask ourselves is:

Are there any obvious sources of multiple exports of the same package?

For example, have we included two versions of the same bundle? Are there bundles with overlapping exports?

Using the packages command in the Equinox shell can help to find multiple exporters, and tells us which bundles are using each export.

If running on Java 6, could one of the new JRE packages be the source of the problem?

Here’s a list of all the packages in Java 6 that were not in Java 5:

javax.activation                    javax.xml.crypto.dom
javax.annotation                    javax.xml.crypto.dsig
javax.annotation.processing         javax.xml.crypto.dsig.dom
javax.jws                           javax.xml.crypto.dsig.keyinfo
javax.jws.soap                      javax.xml.crypto.dsig.spec
javax.lang.model                    javax.xml.soap
javax.lang.model.element            javax.xml.stream
javax.lang.model.type               javax.xml.stream.events
javax.lang.model.util               javax.xml.stream.util
javax.script                        javax.xml.transform.stax
javax.tools                         javax.xml.ws
javax.xml.bind                      javax.xml.ws.handler
javax.xml.bind.annotation           javax.xml.ws.handler.soap
javax.xml.bind.annotation.adapters  javax.xml.ws.http
javax.xml.bind.attachment           javax.xml.ws.soap
javax.xml.bind.helpers              javax.xml.ws.spi
javax.xml.bind.util                 javax.xml.ws.wsaddressing
javax.xml.crypto                    org.w3c.dom.xpath

With regard to prescribing a cure, the general approach is to find an alternative bundle or set of bundles that can provide the dependencies we need.

Does an alternative non-conflicting source for the dependency exist?

If JRE packages are involved, can the additional exporters be removed? Try specifying package versions in the Java profile.

If the JRE is not involved and no alternative supply of the dependencies can be found, then maybe these bundles really cannot be used together: they are simply incompatible. This is bad news to be sure, but it’s better to find it out early from the OSGi resolver than from weeks of debugging a misbehaving runtime.

Does It Really Have to Be This Hard?

I hope not! Clearly this kind of analysis is still based on too much in-depth knowledge of OSGi’s internal workings, which puts it beyond the capabilities of most developers. This could be a real problem for adoption of OSGi in complex enterprise applications.

As ever, we need better tools. My dream is to create tooling in Bndtools that will help to quickly diagnose and offer fixes for problems like this. Here I’d like your suggestions… what kind of tooling would help?

I’m thinking about some kind of graphical tool that can build diagrams on the fly similar to those in this blog post, including the “class space” visualisation. My concern is that it might not scale to a realistic scenario: perhaps the diagrams would just be too big and complex to be any use. I’m just not sure at this point.

Anything Else to Say?

Yes: well done for making it all the way through, and happy hunting.

Master OSGi this Summer

2011-08-12T00:00:00+00:00

This is a final call for anybody wishing to seriously master OSGi.

:http://www.aqute.biz/MasterClass/20110830

The Masterclass on OSGi is approaching at the end of this month. It is a gathering of elite architects and developers from across Europe to learn about how to apply modularity in the real world.

The plan is to lock ourselves into a hotel on a beautiful but remote Swedish island in the archipelago around Stockholm, and not emerge until we have solved the problems of modularity once and for all! Everything is covered: from top-down architecture to low-level details; through IDEs, build systems, repositories, deployment, distribution and provisioning.

Okay so we’re not literally going to be locked in, you will be free to explore the area, and you could even bring your partner and children. Make no mistake though, this is far more intense than a normal training course and you will probably be up late each evening bouncing around ideas or coding up prototypes to show off next morning.

Currently we are ten people, including Peter and me. A few words about us: Peter has been working on OSGi since its beginning 12 years ago, and has vast experience in architecting modular systems. He also has encyclopædic knowledge of the specifications and the motivations behind all the design choices. I have been using OSGi “only” since 2003, but have developed several large and complex applications using it. Since I often do training or work with less experienced developers, I am acutely aware of where OSGi’s ease-of-use is lacking, and I have built tools (e.g. bndtools) to make OSGi more comprehensible and accessible.

If this sounds good then you’re going to have to act fast, since the hotel has only a few rooms left. Just click the big banner above, and I look forward to seeing you there.

How To Embed OSGi

2011-07-03T09:00:00+00:00

A popular question on mailing lists and Stack Overflow is how to embed OSGi in a larger application. This is really easy to do, especially since a standard API for it was introduced in OSGi Release 4.1. Because of the standard, switching between Equinox, Felix and Knopflerfish can be as simple as a configuration change.

(Tangent: another popular question is “which OSGi framework implementation should I choose?”. The only possible answer is: “it depends”. Each framework has different non-functional characteristics, e.g. with respect to memory consumption, scalability, and so on. By sticking to the standard and remaining agnostic, we can develop our entire application and then test which framework best fits our specific needs).

Here’s how to do it.

Create and Start the Framework

The framework launching API uses the Java SPI mechanism to load a “framework factory”. Assuming we have some R4.1-compliant OSGi framework on the classpath, the following will work:

FrameworkFactory frameworkFactory = ServiceLoader.load(
                     FrameworkFactory.class).iterator().next();
Map<String, String> config = new HashMap<String, String>();
// TODO: add some config properties
Framework framework = frameworkFactory.newFramework(config);
framework.start();

We now have an OSGi framework running, but it’s not very useful until we add some bundles. Let’s do that next.

Add Bundles

The Framework object that was returned from newFramework is actually a sub-interface of Bundle, from which we can get a BundleContext. If you have ever written a bundle activator then you are familiar with BundleContext already: it’s the central point of access to the entire OSGi API.

Let’s install the Felix Shell bundles (NB these bundles work just fine on Equinox and Knopflerfish also):

BundleContext context = framework.getBundleContext();
List<Bundle> installedBundles = new LinkedList<Bundle>();

installedBundles.add(context.installBundle(
                     "file:org.apache.felix.shell-1.4.2.jar"));
installedBundles.add(context.installBundle(
                     "file:org.apache.felix.shell.tui-1.4.1.jar"));

for (Bundle bundle : installedBundles) {
    bundle.start();
}

Note that as a general principle, when we want to install a set of bundles we should install them all first and then start them. If you tried to install and start them individually then you would have to be careful about ordering… much better to let the framework sort it all out!

Also bear in mind that there is no error handling in the above code. To improve this we should catch any exceptions on install or start, so that we can still install/start the other bundles.

UPDATE: another thing to be careful of is fragment bundles. Fragments cannot be started, so the bundle.start() line will throw an exception if the installed bundle was a fragment. You may want to add the following check to avoid those exceptions:

if (bundle.getHeaders().get(Constants.FRAGMENT_HOST) == null)
	bundle.start();

Handling Shutdown

Once the framework is running with the bundles you want, you should just let it run until it finishes. Usually the shutdown signal should come from within OSGi, for example the user might type shutdown at the Felix shell. This would stop the framework, and you would normally want to do something after this has happened.

For example, if you are writing a straightforward launcher then you probably want to call System.exit() after OSGi stops. The cleanest way to do this is as follows:

try {
    framework.waitForStop(0);
} finally {
    System.exit(0);
}

The waitForStop method simply blocks the main thread until OSGi stops, so it is the last thing we do after performing all of our initialisation steps. We can’t just allow the main thread to finish and expect the JVM to shut down, because any of the bundles in the OSGi framework might have started a non-daemon thread, so we need System.exit() in order to actually shut down.

Exposing Application Packages

When embedding OSGi we might want the bundles inside the OSGi framework to have visibility of types from the parent application. The way to do this is to expose those types as exports of the system bundle; then the bundles that wish to see those types can import them with Import-Package in the usual way.

For example suppose our parent application contains a domain model in the package org.example.mydomain. This package needs to be added to the system bundle exports, so go back to the first code sample and insert the following where we had the TODO marker:

config.put(Constants.FRAMEWORK_SYSTEMPACKAGES_EXTRA,
           "org.example.mydomain");

The bundle that uses this package should have the following in its manifest:

Import-Package: org.example.mydomain

The framework will wire up this import to the export offered by the system bundle. Note that the importing bundle doesn’t care that the package comes from the system bundle, it could just as easily import from an ordinary bundle! This gives us the flexibility to refactor our application. In the future we might want to do more in OSGi, so the domain model packages would be inside an ordinary bundle in OSGi rather than outside it.

The technique works for any package that is on the classpath of the launcher class. I once had a requirement to embed OSGi in a J2EE application server and expose some EJBs to the OSGi bundles: so javax.ejb and a few other packages had to go on the system bundle exports. The EJB instances were published into OSGi as services.

Other Configuration

There are other configuration changes that can be made in a standard way using the properties you pass to newFramework. Here are a couple of the more useful ones:

// Control where OSGi stores its persistent data:
config.put(Constants.FRAMEWORK_STORAGE, "/Users/neil/osgidata");

// Request OSGi to clean its storage area on startup
config.put(Constants.FRAMEWORK_STORAGE_CLEAN, "true");

// Provide the Java 1.5 execution environment
config.put(Constants.FRAMEWORK_EXECUTIONENVIRONMENT, "J2SE-1.5");

Of course there are a few framework-specific properties as well, for example:

// Turn on the Equinox console on port 1234 (Equinox only)
config.put("osgi.console", "1234");

Using EMF in OSGi

2011-07-02T14:30:00+00:00

It’s official – I ♥ EMF!

This is a surprise to me. I previously regarded modelling in general and EMF specifically as a sort of cult… obscure jargon, terrible documentation, unbearably aloof and patronising talk abstracts. but all that’s behind me now. I see the light: it’s all about the industrialisation of software, which requires both componentisation and automation. OSGi provides admirably for the first, and perhaps EMF can provide for the second.

However I wouldn’t say I’m an entirely orthodox EMFite. I’ve had to hack it about a bit, primarily to get it to work properly OSGi. At first blush using EMF in OSGi may seem trivial, after all, Eclipse is an OSGi application and EMF is designed to work in Eclipse, so EMF must work well in OSGi. Also: all cats have four legs; my dog has four legs; therefore my dog is a cat. Ahem.

EMF supports running in two distinct environments, neither of which matches my requirements. The first is in a “full-fat” Eclipse SDK or Eclipse RCP application. The second is in a traditional non-OSGi Java runtime, which the EMF docs refer to as “standalone”. However the middle ground – an OSGi environment that is not Eclipse – is not supported. That is, you cannot drop EMF into a lightweight OSGi runtime, even allowing for a few dependencies. It depends on the following eight bundles as an absolute minimum:

org.eclipse.core.runtime
org.eclipse.equinox.common
org.eclipse.core.jobs
org.eclipse.equinox.registry
org.eclipse.equinox.preferences
org.eclipse.core.contenttype
org.eclipse.equinox.app
org.eclipse.osgi

The last one is the killer: Equinox itself, meaning that EMF will run only on Equinox, not any other OSGi framework implementation such as Apache Felix or Knopflerfish.

There’s something fishy going on, though. How can EMF run in a “standalone” non-OSGi environment – with no other JARs on the classpath – but not run in Felix??

The explanation is that the EMF bundles have poor internal cohesion. In fact only the activators have a hard dependency on Eclipse classes; other parts of EMF reference them only by reflection. Since bundle activators are ignored in non-OSGi runtimes, EMF works fine there without having to put big chunks of Eclipse on the classpath. However it fails in an OSGi environment where the activator is not ignored – actually it fails during resolution, long before the activators are triggered. As far as I can tell the activators do nothing that is useful outside of Eclipse.

Hopefully the diagram clarifies the problem. We want the green bits but not the red bits; since the green is packaged into the same bundle as part of the red, we have to get all of the red.

The worst consequence is that if you use EMF to generate code for your model then your model API – even the interfaces! – will be inextricably linked to EMF and will therefore be encumbered with all those dependencies as well. As Alex discovered, the links to EMF can be minimised but not completely eliminated, and anyway to remove them would be to throw away some of the most useful functionality of EMF.

Frankly this is the kind of screw-up that makes me want to strangle Eclipse, if only it didn’t have so many necks. But can it be fixed?

If backwards compatibility were not a concern then the fix would be very simple: first, refactor the activator out into a separate bundle; then mark the dependencies optional. However, backwards compatibility is always a concern, so the EMF project is unlikely ever to do this.

The alternative is to repackage EMF into new bundles with better OSGi manifests. So that is what I’ve done, and I’m making them available so that others can enjoy the benefits of EMF without hitching themselves to the Eclipse juggernaut. Bnd makes it so simple to slice n’ dice the JARs into proper bundles. Everything’s available from my GitHub page at http://github.com/njbartlett/emf-osgi.

It’s a work in progress; the bundles available so far are as follows:

name.njbartlett.osgi.emf.minimal contains what appears to be the minimal set of packages required for any EMF-based model. This corresponds to the original org.eclipse.emf.ecore and org.eclipse.emf.common bundles.

name.njbartlett.osgi.emf.xmi contains classes for XMI marshalling/unmarshalling. It has the same content as the original org.eclipse.emf.ecore.xmi bundle.

As the bundle symbolic names are different, you’re going to be out of luck using them if you use Require-Bundle for your dependencies… but if you do use Require-Bundle then you’re not probably not interested in minimising dependencies and being portable, so you might as well use the original bundles.

Be sure not to be misled by the name “minimal” – this is still a fairly chunky bundle, weighing in at over 1Mb. That appears to be the minimum cost of using EMF, and certainly in some scenarios it is too much, but then again in many others it is a drop in the ocean. There is no free lunch, after all.

That’s it for now. I’m only at the start of my EMF journey, and I fully expect to be whacked off course a few more times. Still it seems to be a journey worth taking.

The JDK8 Module System Requirements

2011-05-27T06:30:00+00:00

“There’s something very important I forgot to tell you. Don’t cross the streams… It would be bad… Try to imagine all life as you know it stopping instantaneously and every molecule in your body exploding at the speed of light.” – Egon Spengler (Harold Ramis), Ghostbusters.

I have reviewed the JDK8 Module System Requirements document just published by Mark Reinhold.

The first thing to note is that this is a much more positive and constructive document than anything I have seen previously from Mark on this subject. At last the need to interoperate with the existing standard for Java modularity – that is, OSGi – is recognised and specified as a non-negotiable requirement. So that’s excellent news.

The second point of note is that the requirements for this new module system are really not that far from OSGi itself! Aside from cosmetic differences, such as the location and format of module metadata, the general goals are quite closely aligned and achievable with OSGi. For example, I think that if there were a strong desire to encode metadata in a file named “module-info.java” instead of a file named “MANIFEST.MF” then an agreement could be reached within the auspices of the OSGi Alliance.

In other areas, the proposed module system – which I’ll call “Jigsaw” for now, though it’s unclear if that will be the proper name going forward – is a more permissive superset of OSGi and would be compatible with it. For example it imposes no semantics on versions except that they must be totally ordered, whereas OSGi additionally specifies how many version segments should be used and what are the semantics of an increment in each segment. OSGi’s semantics provide excellent guidance for developing and evolving software, whereas Jigsaw’s are more adaptable to the version schemes used by existing software (actually most existing software uses insane, arbitrary and inconsistent versioning schemes that are driven more by Marketing than by reality). Nevertheless this represents progress, since previous Jigsaw proposals did not even allow for two versions to be compared with each other.

So why create a new module system at all? Why not just implement a few small changes in OSGi to support these new requirements? After all, Oracle and IBM working together probably have the clout within the OSGi Alliance to achieve this, whereas Sun working alone did not.

There is to my mind one big difference that justifies a continued separation. Jigsaw will be used to modularise the Java platform itself, which means it must have features capable of dissecting the tangled mess of packages in the JRE standard library. Even I, a total OSGi bigot, have accepted for some time now that a non-OSGi solution may be necessary for this one specific use-case. That is because, in the Java world, the JRE is unique.

The JRE is an old API that has evolved organically and with very little planning, and as a result it is a tangled mess that cannot be separated cleanly along package lines. In this regard at least, it is not unique at all, but depressingly normal. However when other APIs get into this state we refactor them… which leads to breakages of downstream libraries and applications, but we can cope if we have a functional version system. Unfortunately refactoring the JRE now would literally break every Java application and library in the entire world. So we need a way to modularise without moving classes between packages. In other words we need to split packages across module boundaries.

It’s a common misconception that OSGi does not support split packages; it does. There is a form of module dependency called Require-Bundle in which the depending bundle sees an aggregate view of all the packages exported from its dependencies. The thing is, splitting packages is usually such a bad idea that any OSGi egghead will simply tell you: don’t split the packages.

Don’t split the packages, and don’t cross the streams… except right at the end of the movie, when the Ghostbusters do cross the streams because they’re in the one situation where nothing else will work.

One reason why split packages are bad is because they break Java accessibility rules. Remember that default accessibility in Java is “package private”: a member or a type that can only be accessed by other types in the same package. The current JVM specification (section 5.4.4) says that types or members with default accessibility can be accessed by other members of the same runtime package, where the runtime package is defined as the combination of the package name and the class loader. In OSGi each module has its own class loader, so the runtime package of classes in each module is always different, even if some classes have the same package name. So applying OSGi with Require-Bundle to the JRE would result in IllegalAccessErrors.

Jigsaw’s solution is to provide a mechanism – the details of which are not yet specified – for certain modules to share a class loader. Remarkably, this is really the only major requirement in the Jigsaw document not already supported by OSGi. It would fix the accessibility problem, and also certain other problems where a class expects to be able to load other classes dynamically through its own class loader.

However, as necessary as it may be, Jigsaw’s shared-class-loader solution creates more problems than it solves. OSGi’s class loaders serve to create a barrier between modules, allowing private implementation to be hidden. Jigsaw tears down those boundaries, allowing non-modular code to be repackaged into module-like deployment units. This allows it to be applied to the toughest of legacy code, but at the same time it does nothing to discourage the ill-disciplined development practices that got us into this mess.

It’s like trying to lose weight through liposuction, rather than through diet and exercise.

Also the access rule problem is far from the only issue arising from split packages. See the OSGi core specification, section 3.12.3, for some more good reasons not to do it. Admittedly none are as bad as making your molecules explode at the speed of light, but they’re pretty bad.

Of equal concern is tooling: I believe that tooling to develop and manage modules with fine-grained sub-package splitting will be extremely complex and nigh unusable. In OSGi with our focus on coherent packages, it has still taken us a very long time to develop tools that are easy to use for the majority of developers. This is not due to accidental complexity in OSGi, but inherent complexity in developing modular, reusable software. A module system that encourages split packages will require tools that can manage modularity at the level of types rather than packages. How many types are there in an average Java package… twenty, thirty? That is how many times more difficult it will be to design and assemble coherent modules with Jigsaw.

Sadly when you put a powerful yet dangerous tool in the hands of developers and warn them to never, ever use it except as a last resort, many of them use it as a first resort just to prove what geniuses they are. Lest you think I’m being sanctimonious, I know that I’m guilty of riding this power trip as well, from time to time.

So while I accept the need for Jigsaw in modularising the Java platform, and am happy for certain Jigsaw requirements to migrate into OSGi, I remain opposed to its use anywhere else. OSGi’s Module Layer provides all the power anybody could reasonably need to build highly modular software, and I have not even talked yet about the OSGi Service Layer, where OSGi’s power really shines – something entirely absent from Jigsaw.

OSGi has rules, and those rules provide guidance and a framework for creating modular software. Jigsaw’s permissiveness will be seductive because it will allow painful decisions to be deferred indefinitely. Its additional power will cause great damage if applied to the wrong problems in the wrong way. No wonder it is named after a tool notorious for chopping off fingers.

Concurrent Vaadin

2011-05-25T14:00:00+00:00

As my colleagues and regular readers know, I have been hugely impressed by Vaadin, a web framework for Java, and have been using it in many of my OSGi demos, tutorials and projects.

This blog post is not about OSGi, but the more general problem of dealing with concurrent updates in a Vaadin application. I describe a technique for updating the application display in response to asychronous events. My own use-case was handling service binding/unbinding events in OSGi, but the technique would equally apply to message processing applications, etc.

Vaadin’s programming model feels very much like working with a conventional desktop GUI API such as Swing or SWT/JFace, though it is more elegant than either of those examples. However one aspect where it differs significantly is its lack of an “event thread”, which in desktop APIs is responsible for continuously painting the screen and responding to keyboard and mouse clicks.

Rather, Vaadin controls are backed by a data structure on the server side, which is queried by the HTTP server thread when the browser makes a request – the HTTP server thread is generally owned by the Application Server. Therefore at the first approximation, we merely need to mutate the data structure in response to our async event, and the HTTP request will see the changed value when the next request occurs (this would be when the user clicks a button or refreshes the page… I will ignore “push” technologies for now – i.e., COMET, Web Sockets – since they are not currently supported in core Vaadin).

Unfortunately the first approximation quickly fails because it is not safe. The HTTP request might arrive while we are in the middle of mutating the data structure, so it will see data in an inconsistent state. Or it might see the old value of the data, even if the request occurs after the update in terms of wall-clock time, because of the way caches work on modern CPUs.

To ensure that the changes we make to the data are both visible and consistent when read by the request thread, we have to perform them in a synchronized block, and furthermore we must synchronize on the same object that the HTTP request thread synchronizes on. This topic is not well documented in the Book Of Vaadin, but several messages on the forums indicate that the main Application object is the one to synchronize on. Our life is easiest then if we are the main Application. For example:

public class MyApp extends Application {

  private final Table table = new Table();
  
  @Override
  public void init() {
    setMainWindow(new Window("Demo App", table));
    table.setFullSize();
  }

  // This is our async handler
  public synchronized void onMessage(Message message) {
    Item item = table.addItem(message.getId());
    // ...
  }
}

Making the onMessage method synchronized is enough, because it locks the Application. Be careful with synchronized methods though… it is usually better to reduce the scope of the lock by doing the minimal UI changes in a synchronized block, rather than locking for the entire method.

Unfortunately things are not so easy when we are writing a CustomComponent, which we commonly do in order to implement a reusable “chunk” of the overall Application. The problem is, we may start receiving async events before the component has been bound into the Application, so we cannot synchronize on the Application because we don’t have visibility of it yet (that is, getApplication() will return null). The solution is to synchronize on the application if it is bound, but if it is not yet bound then synchronize on the component itself. There are a number of tricky details to take care of though: what if the application becomes bound while we are handling the async event? We also need to make the setParent() method synchronized, since that is the method where we are notified of the application being bound.

To avoid repeating ourselves, we can encapsulate these details into an abstract base class ConcurrentComponent, as follows:

public abstract class ConcurrentComponent extends CustomComponent {
  @Override
  public synchronized void setParent(Component parent) {
    super.setParent(parent);
  }  
  @Override
  public synchronized Component getParent() {
    return super.getParent();
  }
  
  protected void executeUpdate(Runnable update) {
    Application application = null;
    synchronized (this) {
      application = getApplication();
      if (application == null)
        update.run();
    }
    if (application != null) {
      synchronized (application) {
        update.run();
      }
    }
  }
}

This allows us to implement our own component as follows:

public class MyComponent extends ConcurrentComponent {

  private final Table table = new Table();

  // Async handler
  public void onMessage(Message message) {
    executeUpdate(new Runnable() {
      public void run() {
        Item item = table.addItem(message.getId());
        // ...
      }
    });
  }
}

This code should be strongly reminiscent of SwingUtilities.invokeLater() or SWT’s Display.asyncExec(), which is intentional. Note how the lock on the Component is used directly to execute the operation if the application is unbound, but that lock is released and an Application lock acquired instead when the application is bound.

I must admit that I am not entirely happy with the necessity of taking two locks in succession for the common case, and worry that it may have a performance impact. However I can’t see an easy fix that retains correctness; perhaps somebody else will be able to suggest one.

On the chance that any of the above code proves useful, I hereby place it in the Public Domain – use it however you wish.

Bndtools in the Java Tech Journal

2011-05-19T03:00:00+00:00

:http://jaxenter.com/java-tech-journal/JTJ-2011-04

The Java Tech Journal issue 8 is now available for free download, and is dedicated to OSGi.

The JTJ is an impressive new publication from the folks at JAXenter, with each issue covering a different technology in depth. The current issue contains excellent articles by some of the leading names in OSGi:

A Gentle Introduction to OSGi, by Jerome Moliere from Mentor/J;
What’s New in Core 4.3? by BJ Hargrave and Peter Kriens from the OSGi Alliance;
Developing OSGi with Bndtools, by Marcel Offermans and Alexander Broekhuis from Luminis;
OSGi A La Carte, by Valentin Mahrwald and Holly Cummins from IBM Hursley labs;
OSGi and JPA, by Dmytro Pishchukhin;
The Cure for Complexity, by Richard Nicholson from Paremus (and President of the OSGi Alliance);
The Enterprise OSGi Specification, by Tim Diekmann from TIBCO and David Bosschaert from JBoss/Red Hat.

From a personal point of view I’m very pleased to see multiple references to Bndtools as a tool for developing OSGi bundles, not just in Marcel and Alexander’s article but also in Jerome’s tutorial and in Holly and Valentin’s piece about the tooling ecosystem. Thanks to all the authors for your kind words — your cheques are in the post ;-)

Stay tuned for some exciting announcements about Bndtools, coming up very soon…

OSGi and IKEA

2011-05-11T11:00:00+00:00

Mario Fusco wrote a thought-provoking tweet this morning:

Ikea / furnishing = OSGi / software : They allow to have very modular things, but trying to put them together generally drives you crazy

This is an interesting analogy, but I obviously take issue with the implied criticism of OSGi. So let’s break it down.

The first thing to say is that I actually find IKEA furniture quite easy to build, and I doubt I am alone in this. Especially once you have built a few, you find there are common patterns… similar tools, similar kinds of connectors and so on. The instructions are so simple that they don’t even need any text, and in many cases you can still complete the build even if you’ve lost the intructions.

The main sources of frustration are errors: missing parts, or poorly machined connectors that don’t fit they way they should. Fortunately these problems are rare – when a part IS missing, it’s nearly always because I have lost it myself.

Is there an alternative to this kind of modular construction? Of course. The previous generation of manufactured furniture consisted of completely whole, inflexible objects, produced in huge quantities. A desk, for example, was available in specific dimensions and with a specific number of drawers. If that desk didn’t fit the corner where you wanted to put it, then you had two choices: hope somebody else did sell a desk of the right size… or move house.

Before mass production, furniture was assembled by craftsmen. It was gorgeous, high quality, fit precisely to your requirements… and cost literally thousands of times more than an equivalent IKEA product.

Today most software is produced according to one of these two models. They are packaged COTS software and bespoke in-house applications, respectively. There is nothing wrong with either approach: if your requirements are conventional then a COTS solution is best; if your requirements are unique (and you have deep pockets) then building bespoke is the way to go. But there is a huge gulf between these models, and if your requirements fall somewhere between the two then you face either making do with software that isn’t quite what you need, or spending a fortune building your own thing from scratch.

There has to be a better way… an IKEA way… and that’s exactly what OSGi is. It empowers the end customer to assemble modules into a finished article that meets their needs, going way beyond the limited customisation abilities of COTS products. At the same time it offers a cost model closer to that of mass production.

Can OSGi modules drive you crazy when you try to put them together? Yes that happens, and for the same reasons: something is missing, or something is bad quality. IKEA has excellent quality controls over the parts that it sells… third party software authors, well, not so much.

[Incidentally where does open source fit into this analogy? In my opionion, open source libraries are usually neither comprehensive nor robust enough to correspond to a “component” or “module” in the IKEA sense. I believe they are more like the raw materials: the laminated MDF boards, the screws and connectors and clippy things. These are vital productivity enhancers for the craftsmen who design the components, but they are not components themselves that can be assembled and deployed by end users.]

Apache Felix Diagnostics for "Uses" Conflicts

2011-03-31T09:00:00+00:00

Last month I wrote a blog post explaining how “uses constraint” violations come about in OSGi, and gave some heuristic shortcuts for solving them.

Sometimes though, the shortcuts just don’t work. In the worst case, you have to trace through all of the package dependencies manually in order to find the culprit. Usually the culprit is a duplicated package somewhere in the system, but finding it can be a headache.

Or, so I thought…

Yesterday I ran into just such a problem with a client’s application. Equinox was printing the following error (package names have been changed to protect the innocent):

!MESSAGE Bundle org.example.foo.xydata [17] was not resolved.
!MESSAGE Package uses conflict: Import-Package:
org.example.foo.util.service

None of the shortcuts I talked about in my last post worked, so I began the laborious process of tracing all the package dependencies. I did this for about 30 minutes before remembering that the latest versions of Apache Felix have some great new diagnostics for uses constraint violations. After a little work to switch the framework used by the launcher, I managed to get Felix running and was immediately rewarded with the following message:

Unable to resolve module org.example.foo.xydata [17] because it
is exposed to package 'org.example.foo.util.verify' from
'org.example.foo.framework' and 'org.example.foo.platform.util'
via two dependency chains.

Chain 1:
  org.example.foo.xydata [17]
    import: (package=org.example.foo.util.verify)
     |
    export: package=org.example.foo.util.verify
  org.example.foo.framework [12]

Chain 2:
  org.example.foo.xydata [17]
    import: (package=org.example.foo.bar.datastore)
     |
    export: package=org.example.foo.bar.datastore;
                    uses:=org.example.foo.util.verify
  org.example.foo.bar.datastore [13]
    import: (package=org.example.foo.util.verify)
     |
    export: package=org.example.foo.util.verify
  org.example.foo.platform.util [8]

This gets pretty close – as close as one could get in pure text, I believe – to a graphical representation, which I have attempted to reproduce as follows (omitting the common prefix):

Bundle 17 cannot resolve because of the two dashed lines. These two wires cannot both exist at the same time, because that would lead to an inconsistent view of the util.verify package.

There is some information missing from Felix’s message and my diagram… why has OSGi chosen to try to wire bundle 17’s import of util.verify to bundle 12? Just looking at the diagram, it seems like all would be well if it were wired to bundle 8. But of course the diagram is only a piece of the puzzle, and there are other constraints present that prevent the desired wiring (specifically, another import exists that must be resolved from bundle 12 and also uses util.verify). Nevertheless, there is enough information here to immediately diagnose the real problem: util.verify has been incorrectly duplicated in multiple bundles. Removing the duplication fixed the problem.

Granted, the analysis still requires somebody with some high level understanding of OSGi to work out what the underlying problem is. But at least Felix gives a substantial amount of information to work with, rather than leaving us in the dark.

My client was so impressed that they are considering switching their development to Felix. But even if you remain a committed Equinox user, consider the possibility of using Felix temporarily to obtain diagnostics when you run into such problems. Another reason to use a standard, framework-agnostic launcher architecture.

Towards Maven Support in Bndtools

2011-03-02T14:00:00+00:00

A commonly requested enhancement for Bndtools is to enable integration with Maven. Although I don’t tend to use Maven myself, I do recognise that it’s a very popular build tool, especially in big companies with lots of developers.

That’s why I’ve been working on Maven support for Bndtools, which is now available to try out as an “Alpha” quality pre-release. While not yet particularly stable, I need to get some feedback from real Maven users to guide the rest of the implementation.

Before going into the details, first a word of reassurance to the non-Maven-users: you absolutely will not have to adopt Maven in order to use Bndtools, and I will never make it a requirement. Bndtools projects can still be built with Bnd, which is a very capable tool: for example, the OSGi Alliance uses Bnd to build over 1300 bundles every night.

On the other hand if you are a committed Maven user already, then Bndtools’ support will make it possible for you to take advantage of powerful OSGi-specific development features in your IDE, without changing your build processes.

It’s easiest to explain what I have done in a Q&A style. Please add your own questions in the comments.

How Does it Work?

An important design goal is to avoid duplicating any configuration data. Dependencies and bundle contents/configuration are each defined in exactly one place only.

Bndtools should be installed alongside M2Eclipse in the same Eclipse instance. There is a clear division of responsibilities between the two tools:

Maven and M2Eclipse are responsible for build-time dependencies and performing the canonical “offline” build. This in turn uses the Maven Bundle Plugin, which is the most popular way to build OSGi bundles with Maven; for example, Apache Felix and all its subprojects are built this way. The Bundle Plugin uses Bnd internally, just like Bndtools.
Bndtools is responsible for “on-the-fly” developer builds, launching OSGi frameworks and test runs, and configuring bundle contents.

Why Should I Use This?

If you are a current Maven user, then the approach outlined here allows you to easily adapt your existing Maven builds to develop OSGi bundles, using the most popular approach for building bundles with Maven. It does not require major changes to your project structure.

At the same time you will gain the OSGi specific ease-of-use features of Bndtools (see next question) while still allowing users of other Java IDEs such as NetBeans to easily work on your project.

What Does Bndtools Offer Over M2Eclipse?

M2Eclipse is a fully-featured development environment for working with any kind of Maven-built project. However it offers few ease-of-use features for working with specific project types.

Bndtools offers a rich and easy-to-use environment specifically for developing OSGi bundles. You can take advantage of the following features:

On-the-fly generation of the OSGi bundle, and deployment to a running OSGi container, immediately upon saving the source of a Java file.
Powerful tools to visualise and manipulate the package-level dependencies of bundles.
Tools to configure, run and test an OSGi application, which also work in exactly the same way from the command line.
… and more!

For example, when using the raw Bundle Plugin, the contents of a bundle are controlled via a rather arcane set of instructions embedded in the POM, such as the following:

<instructions>
    <_versionpolicy>
        [$(version;==;$(@)),$(version;+;$(@)))
    </_versionpolicy>
    <Export-Package>
        org.apache.felix.fileinstall;version=${project.version}
    </Export-Package>
    <Import-Package>
        org.osgi.service.log;resolution:=optional,
        org.osgi.service.cm;resolution:=optional,
        *
    </Import-Package>
    <Private-Package>
        org.apache.felix.fileinstall.internal,
        org.apache.felix.utils.properties,
    </Private-Package>
    <Bundle-Activator>
        org.apache.felix.fileinstall.internal.FileInstall
    </Bundle-Activator>
    ...

In contrast Bndtools gives us the following editor (click for full size). Note the Calculated Imports list in the top right, which is recalculated on the fly as we adjust the bundle contents and customise imports; with Maven Bundle Plugin we must do a full build to obtain this information.

:/images/bndtools-editor.png

Do I Have to Use M2Eclipse?

Yes, for now: to enable Maven support in Bndtools, M2Eclipse is required.

M2Eclipse provides the build-time classpath that allows Eclipse to compile the Java sources, before Bnd takes over to build the OSGi bundles.

I am investigating the possibility of directly processing the POM and local repository from within Bndtools. This could provide a lightweight alternative for users that do not need the full suite of functionality offered by M2Eclipse. However, this feature is not yet available.

Why Not Use Tycho?

There is nothing wrong with Tycho, but it is a tool for a different job: it is intended for building projects that have been developed using the Eclipse Plug-in Development Environment (PDE). Projects developed in PDE are very inflexible and require specialised build tools – as a result they have never worked properly with “vanilla” Maven. Tycho works by adapting Maven to the PDE project structure.

In contrast, Bndtools is more flexible and adapts to your existing build system, whether it is vanilla Maven or something else. To summarise the advantages of using Bndtools with Maven versus Tycho:

Bndtools uses the standard Maven project structure; Tycho uses the PDE project structure.
Tycho projects are Eclipse PDE projects, which are difficult (though not impossible) to work on using IDEs other than Eclipse. Bndtools/M2Eclipse projects can be easily worked on by developers using other IDEs, though of course they will miss the ease-of-use features offered by Bndtools.
Tycho requires Maven 3, whereas Bndtools works with both Maven 2 and 3.

These points are distinct from the advantages of Bndtools versus Eclipse PDE, which is a subject for another blog post.

To be clear: if you are currently using Eclipse PDE to develop your OSGi bundles then you should definitely use Tycho to build them! On the other hand if you are:

a Maven user interested in trying OSGi, or;
looking to “OSGify" (or “bundleize") an existing Maven build, or;
already using the Bundle Plugin

…then I believe Bndtools is more likely to fit your needs.

How are Dependencies Specified?

In Bndtools/Maven projects the build dependencies are specified entirely in the POM. I.e., the POM is the sole source of build-time dependencies, and Bndtools uses the build classpath as defined by M2Eclipse to perform its on-the-fly builds.

The -buildpath feature of Bnd, which is ordinarily specified in bnd.bnd, should not used when building with Maven.

How Are Bundle Contents Specified?

The instructions required by Bnd to specify bundle contents and calculate imports are given in the bnd.bnd file, just as in any other Bndtools project. These instructions are referenced from the POM using the following instruction to the Bundle Plugin:

<instructions>
    <_include>bnd.bnd</_include>
</instructions>

This means that bnd.bnd is the sole source of bundle content and configuration instructions.

Note that some parameters in the output bundle can be inherited from the POM. For example, we typically want to use the Artifact ID as the symbolic name of the bundle and the POM version as the bundle version. This can be achieved by referencing these properties from bnd.bnd as follows:

-include: pom.xml

Bundle-SymbolicName: ${pom.artifactId}
Bundle-Version: ${pom.version}

Can I Build Multi-Bundle Projects?

Bnd supports a feature called “sub-bundles”, in which multiple bundles are built from a single project.

While this feature does still work with a Maven-based build, it is not recommended because it works against the Maven philosophy of one output artifact per project. Downstream tools and Maven plugins, such as those that assemble the output bundles into a packaging artifact, are unlikely to work correctly with multi-bundle projects.

How Do On-the-Fly Builds Work?

Bnd uses the project classpath – as created by M2Eclipse from the POM – to perform on-the-fly builds of bundles. As ever when using Bndtools, your bundles are always built, and if you are running an OSGi framework from Bndtools they will be immediately deployed into it.

Since both Bndtools and the Maven Bundle Plugin use Bnd internally, the on-the-fly bundles are exactly the same as those built by Maven in a full offline build.

What Is The Bnd OSGi Repository Used For?

When using Bndtools without Maven, Bnd Repository is used to supply bundles for both building and running. However when using Maven, Bndtools is no longer responsible for build dependencies but the Bnd Repository is still used to supply bundles for running and testing OSGi applications.

This is important because the Maven repositories contain a mixture of different artifact types, and most of the JARs it contains are not OSGi bundles. The Bnd Repository contains exclusively OSGi bundles and so is very useful for actually running OSGi applications.

Having said that, a subset of JARs in the Maven repositories are OSGi bundles, and work is in progress to build a Bnd Repository that wraps the Maven local repository to offer those bundles for use at runtime.

How Do I Use It?

First ensure that both M2Eclipse and Bndtools Alpha are installed in your Eclipse IDE. Here is the update site for Bndtools Alpha:

http://bndtools-alpha-updates.s3.amazonaws.com/

The next step depends on whether you are creating a new project, or adapting an existing one.

Creating a New Maven/Bndtools Project

To create a new Maven project with Bndtools support, use bndtools-component-archetype. This archetype corresponds to Bndtools’ “Component Template”, and creates a project with support for Declarative Services annotations. The OSGi Core and Compendium APIs are added as dependencies, in addition to JUnit.

As soon as the Bndtools Maven support is stable, this archetype will be submitted to Maven Central so that it can be directly accessed by Maven/M2Eclipse. For now, you need to click the link to download it, then install into your local repository as follows:

mvn install:install-file \
    -DgroupId=org.bndtools.archetypes \
    -DartifactId=bndtools-component-archetype \
    -Dversion=0.0.2-SNAPSHOT \
    -Dpackaging=jar \
    -Dfile=./bndtools-component-archetype-0.0.2-SNAPSHOT.jar \
    -DgeneratePom=true

You can now create a new project from the archetype as follows:

mvn archetype:create \
    -DarchetypeGroupId=org.bndtools.archetypes \
    -DarchetypeArtifactId=bndtools-component-archetype \
    -DarchetypeVersion=0.0.2-SNAPSHOT \
    -DgroupId=org.example \
    -DartifactId=comp1 \
    -Dversion=0.0.1-SNAPSHOT

Then you can import the project into Eclipse using File > Import > Existing Projects into Workspace. You can also use M2Eclipse to create a project from the archetype without leaving Eclipse, but due to an apparent bug in M2Eclipse this throws an error (NullPointerException). In fact the project is still created, so the error can be safely ignored.

Adapting an Existing Maven Project

If you have an existing “vanilla” Java Maven project, you can adapt it for use with Bndtools by right-clicking on the pom.xml and selecting Add OSGi/Bnd Support.

This starts a refactoring wizard that performs the following changes:

Creates a bnd.bnd file and adds the Bnd “nature” to the Eclipse project;
Adds the Maven Bundle Plugin to the build plugins section of the POM, if not already present;
Changes the Maven packaging type to “bundle” (required by the Bundle Plugin).

Note that the changes to the POM will be previewed in the wizard, so you can make sure you are happy with them before clicking Finish.

:/images/bndtools-maven2.png

Is That It?

Yes, for now.

Please give it a try, but don’t expect everything to work. If something that seems like it should work doesn’t, then drop me a line. If you think my whole approach is wrong, then tell me how I should be doing this. My goal is to make Bndtools the easiest and fastest way to develop OSGi applications – irrespective of how they are built.

The JCP... Weep for the Experts

2010-12-09T00:00:00+00:00

Recently I have preferred to stay away from political blog posts, but the occasion of Apache’s departure from the Java Community Process (JCP) seems a fitting time to fulminate, just this once.

As Bill Joy once said:

“Most smart people don’t work here… for any definition of ‘here’.”

A deep understanding of that principle, and of the value of diversity, led Sun to create the JCP. Java became strong not as a barbershop quartet singing in perfect harmony, but as a massive, heaving soccer crowd baying for their team. The neutering of the JCP threatens to turn that crowd into a bunch of North Korean rent-a-fans.

There will inevitably be a loss of diversity of brain-power feeding into the Java specification. Java has never been the product of just one company: Sun made a major contribution, yes, but so did IBM, BEA, Ericsson and countless others. They contributed their time, effort and in many cases intellectual property to improving Java because the JCP offered a legal framework for their work to be distributed as an open platform, which benefited everybody.

That framework was called the Java Specification Participation Agreement (JSPA), and it meant that intellectual property from the Experts on an Expert Group flowed via the Spec Lead and into the final specification, so that independent implementations could be built without fear of copyright or patent claims – so long as they passed the TCK. Stephen Colebourne has a nice pictorial guide to this process in one of his blog posts.

It was the JSPA that Sun and later Oracle broke by refusing to allow Apache to test Harmony with the TCK without “Field of Use” restrictions. The refusal was all the more egregious because it was not entirely their property to withold.

So what motivation would an expert now have to serve on a JSR Expert Group?

Take for example Bob “Crazy Bob” Lee, author of Guice, major contributor to the Google Collections API and spec lead of JSR 330, the fastest ever JSR. He was invited to join the expert groups of both the Java 7 and Java 8 JSRs. He is undoubtedly an expert and would have made an excellent contribution to those JSRs… but he declined to participate. After all, why on Earth should Bob spend his valuable time working on a JSR when the fruits of his labour would be taken by Oracle?

Doug Lea, who probably knows more about concurrency than anybody on the planet, resigned from the JCP in disgust for similar reasons. Oddly he continues to work on the OpenJDK, but this is because he is a researcher and it is still possible for him to perform novel research in a closed platform, and Java is still the most advanced managed runtime environment. Wouldn’t he jump ship though if a similarly advanced but more open platform were to emerge? Admittedly I’m speculating about that because I don’t know Doug at all.

Other resignations include Tim Peierls and of course Apache. Google will probably follow shortly. They have all realised that they were giving their work and IP to improve a proprietary Oracle product… for nothing.

The primary remaining motivator is money. Experts will be drawn from the set of people paid by Oracle, directly or indirectly. Oracle employees, obviously. IBM employees too, since it’s clear that Oracle wrote IBM a fat cheque to get them to abandon Harmony and join OpenJDK… the details of the deal are private but I sincerely hope they emerge on WikiLeaks some day.

The result will be a dreadful monoculture. Many smart people work at Oracle and IBM, but even more don’t. End users will not have their say, except through official Oracle or IBM sales channels, hence the joke that JCP now stands for “Just Customers Please”. Vendors of small, alternative frameworks will either not contribute or their contributions will be appropriated and assimilated.

Meanwhile, less pessimistic souls are congratulating Oracle for finally getting Java moving again. Well, Il Duce made the trains run on time. Enough said.

Sigh. What’s a poor Java developer to do? I’m not certainly giving up the platform just yet because there still aren’t any better games in town. All the same, it seems wiser than ever to transfer at least a few eggs to alternative baskets. Just in case.

OSGi Masterclass is Coming to America

2010-11-02T00:00:00+00:00

:http://www.aqute.biz/MasterClass/201101-02

The Masterclass on OSGi is coming to the USA!

After a successful first go in our back yard – i.e., Europe – Peter Kriens and I have fixed a date and location for the USA Edition: 18th to 21st January 2011 in Pleasanton California.

I’ve never been to Pleasanton but it sounds… well, pleasant. And it’s convenient too, being accessible by BART from San Francisco city and airport. But then again it’s not too close to all those distracting highlights of the big city, and this is something we learned from the first class: a little bit of isolation helps to create a “pressure cooker” in which the ideas bounce around at high speed. It’s exhausting, but definitely extracts a lot of value for the attendees.

As for the format of the class, it is very different from your usual structured training. The masterclass is about achieving mastery of OSGi, and that cannot be done with a fixed slide deck. Instead we will look at the subjects that interest you, and work out how to apply OSGi to the challenges you see in your environment. Peter and I won’t always have all the answers, but we usually know the right questions and have the experience to reach towards a solution together. But we also expect to have our own opinions and assumptions challenged.

Needless to say though, we believe that our knowledge of OSGi is second to none. And I don’t mean to imply that we don’t have any slides, or that we have not prepared anything. I have a vast slide deck from my normal training courses, but this will serve as back-up material rather than driving the class.

The price for the 4-day class is a very reasonable $2200, or $2000 if you register before 1st December. Also if you are attending ApacheCon in Atlanta this week, we have a special discount code for you which will be advertised at the Felix/OSGi meet-up on Thursday evening, and the OSGi Track on Friday.

Eclipse on SoyLatte without X11

2010-10-24T00:00:00+00:00

An article by Cade Metz in The Register suggests that Apple’s deprecation of Java is an attack on Android developers:

"If you don't have Java on the Mac, you can't run Java development tools such as Eclipse, IntelliJ, and NetBeans. And though there is a port of OpenJDK for the Mac, it's not quite complete. It's limited to the X11 windows system, and there's no support for native Mac GUIs. ... "The other salient point here is that if you don't have Java on the Mac, you can't use Google's Android SDK. Most Android development is done on Eclipse, and though Eclipse isn't required by the SDK, the Java environment is."

Don’t believe everything you read in El Reg, folks. Eclipse runs just fine on that OpenJDK port (a.k.a. SoyLatte) without X11. Here’s a screenshot:

No, I can’t tell a difference from normal Eclipse either, at least visually. It does feel a bit sluggish, so I assume that SoyLatte is not heavily optimised. But that’s a fixable problem.

SWT is the key. All the UI customisation work that Apple has put into its JVM was required to support Swing/AWT, which Eclipse does not use. Therefore Eclipse – and Google’s Android tools – will work fine, even if Apple eradicates its JVM from future Mac OS releases and refuses to give its UI code to OpenJDK. The same goes for Eclipse RCP applications such as Vuze and RSSOwl.

Sadly the same may not be true of NetBeans and IntelliJ, since they build on a GUI technology that is both tightly coupled to the JVM and requires extensive customisation by the operating system vendor. I wouldn’t miss those other IDEs myself, but it’s good to have choice in the market, so I hope the situation improves for them soon.

One more screenshot, here is the Eclipse 4.0 early access release:

UPDATE: David asked in the comments how to actually run Eclipse on SoyLatte. It’s quite simple. You need to open up the Eclipse.app launcher and edit the file Contents/MacOS/eclipse.ini. Assuming you have SoyLatte installed at /usr/local/soylatte16-amd64, insert the following two lines:

-vm
/usr/local/soylatte16-amd64/bin/java

These two lines must go above the -vmargs line.

A Credible Specification and Standards Body

2010-10-23T00:00:00+00:00

Doug Lea says that “the JCP is no longer a credible specification and standards body”. He’s probably right.

However, a credible specification and standards body does exist in the Java universe. I will leave its identity as an exercise, but let me just point out a couple of its nicer characteristics.

It produces fantastic quality specifications.
It has worked effectively for over ten years as a collaboration forum for competing companies.
Oracle is a very active member, but Oracle doesn’t own it. Indeed, all the full members have exactly the same rights.
All its specifications are open, and come with a non-assert patent pledge from the members.
TCKs are freely available to open source projects, and the majority of the RIs are in open source already.

Of course, it’s not without problems. The biggest is that membership dues are too high for small companies or academic institutes, and there is not yet a useful class of membership for individuals. But to be fair, if you pay your money then you can have the same rights (and responsibilities) as Oracle, IBM, SAP, VMWare and the rest.

To quote my Californian friends: I’m just sayin’.

Bndtools is Growing

2010-10-21T00:00:00+00:00

Bndtools is starting to gather momentum and attract a real community. The best thing about this from my point of view is that I have started to receive significant contributions.

The biggest so far has come from Per Kristian Søreide of ComActivity. Per and his team are hugely experienced in OSGi and Eclipse, and they have developed a tool for releasing bundles into a repository, with in-depth analysis of compatibility and versioning. Essentially, it helps you to comply with the OSGi Semantic Versioning (PDF) guidelines.

When you make changes to an API, the tool calculates whether those changes are backwards compatible for consumers, producers, both, or neither. It then adjusts the version numbers of each package accordingly. It also bumps the version number of your source .bnd and packageinfo files.

For example, suppose I make my first release of a new API, version 1.0.0. Here is what the tool displays:

Now I add a method to the AuctionService interface and release again:

This tells me that I have made a change that is non-backwards compatible for producers, and therefore the “minor” version segment needs to be incremented. That is, I need to update from 1.0.0 to 1.1.0. Of course it also highlights the reason. If I were to make an even more drastic change such as deleting a method, the tool would tell me that I need to bump to version 2.0.0, indicating a breaking change for both consumers and providers.

Using this tool, alongside bnd’s existing support for generating import ranges, means that (in theory) you should never have to manually maintain version numbers! Now, that’s a pretty bold statement and we need to see how things works out in practice, but I strongly feel that version management is one of those extremely important — but also extremely tedious — tasks that can be performed far better by machines than by humans..

The release feature is now included in the main Bndtools update site, so if you have already installed Bndtools, please update it via the Eclipse update manager.

Many thanks to Per and his team! And thanks also to all the others who reported bugs and contributed fixes… please keep them coming.

OSGi Readiness — Loading Classes

2010-08-30T00:00:00+00:00

In my previous post on “OSGi Compliance”, I discussed the idea of making libraries ready for OSGi without depending on OSGi directly. In light of BJ’s comment I will refer from now on to OSGi “Readiness” rather than Compliance, as the latter term is easily confused with the concept of a framework implementation that complies with the OSGi specification.

In each of these blog posts, I will state a requirement and then look at options for satisfying that requirement in an OSGi Ready way.

Problem Statement

You are developing a Java library, “Framework X”. Occasionally you must load and instantiate a class dynamically by name, where the name may originate from a configuration file or be passed into the framework as a parameter to an API call. You would like the framework to work cleanly both in OSGi and other environments, i.e. there must be no dependency on OSGi.

For example, suppose we are implementing an O/R mapping framework similar to Hibernate, and the user has supplied a configuration file such as the following:

<or-mapping>
   <class name="org.example.domain.Event" table="EVENTS">
      <id name="id" column="EVENT_ID"/>
   </class>
</or-mapping>

This tells our framework to create an instance of org.example.domain.Event for each row of the EVENTS table. Therefore we must dynamically load and instantiate the org.example.domain.Event class.

Discussion

To dynamically load a class in Java we require two things: the fully qualified name of the class, and a ClassLoader from which to request it. If we have both of these then we can trivially load a class irrespective of whether we are in OSGi or not.

To reiterate: OSGi does not interfere with the standard Java class loading API. If we have the fully qualified name of a class and a ClassLoader that knows of that class, then we can always load and instantiate the class.

Problems arise only when we do not know the correct ClassLoader, and make the wrong guess about which one to use.

First Wrong Guess: Class.forName()

Java provides us with a way to load a class without explicitly specifying a ClassLoader, using the single-argument version of the Class.forName() method:

Class clazz = Class.forName("org.example.domain.Event");

Since we don’t specify a ClassLoader, this method uses the loader that loaded the class in which the line of code appears. In other words, it uses the same loader that loaded the framework. In a traditional Java SE environment this usually works just fine, because both the framework and domain classes are probably installed as JARs on the classpath, and the classpath is loaded by a single ClassLoader, the so-called “system” loader.

But in OSGi, the domain classes are likely to be supplied in a different bundle, because there is a clear modular separation between the O/R framework code and the domain model classes, and therefore they will be loaded by a different ClassLoader. We can still make it work if the package org.example.domain is listed as an import of the framework bundle like so:

Import-Package: org.example.domain, ...

However this is unsatisfactory because Import-Package is statically defined, and a general-purpose O/R framework does not know in advance which domain packages it will need to import.

(NB there are a couple of other workarounds for this problem in the case that the framework code cannot be changed. These include: DynamicImport-Package; shipping the domain classes as a fragment hosted by the O/R framework; and using the Equinox-only “buddy loading” feature. However all of these workarounds have their own severe problems, which I shall not discuss in depth here because the purpose of the post is to document how to best implement a framework such that hackish workarounds are not required.)

Second Wrong Guess: Thread Context Class Loader

The Class.forName() approach fails not only in OSGi but also in other environments such as Java Enterprise Edition, for similar reasons. Because of this, another common pattern is to use the Thread Context Class Loader (TCCL), which was introduced relatively recently (i.e. Java 1.2):

ClassLoader tccl = Thread.currentThread().getContextClassLoader();
Class clazz = Class.forName("org.example.domain.Event", true, tccl);

Some libraries use the TCCL exclusively, others use it as a last resort after failing to load from other candidate class loaders. Almost every library follows a different strategy because Java has provided almost no specification or guidance on when to use the TCCL… to quote a classic JavaWorld article:

Those class and resource loading strategies must be the most poorly documented and least specified area of J2SE.

Even in J (2)EE the specifications do not lay out precisely which set of classes should be visible through the TCCL. In practice, a J (2)EE application server is able to provide a sensible set of classes via the TCCL because the programming model is highly constrained: each application runs in an isolated silo, is not allowed directly to create threads or network sockets, and threads cannot cross application boundaries. Because it controls all the entry points, the application server can provide a TCCL that is appropriate for the running code.

OSGi’s programming model is far less constrained. Bundles are free to create threads, to open network sockets, and to call APIs exposed by any other bundle. There is no way in Java to intercept a direct method call that happens to cross a bundle boundary, so the OSGi framework cannot ensure the TCCL is relevant to the current code. As a result, the TCCL is not defined by the OSGi specification, and it may be null. Therefore, we cannot rely on the TCCL for anything.

Solutions

Now let’s discuss possible solutions. To be clear, all of the following suggestions are intended to supplement rather than replace existing approaches. If your framework is working in J2SE/J2EE with Class.forName() or TCCLs then you should continue to use those approaches. But, please provide additional options for runtime environments such as OSGi where those approaches fail.

If you feel these suggestions are trivial or obvious — that is exactly the point. There is nothing complicated about making frameworks that work well in OSGi. Indeed, you may find you prefer using one of these patterns as they reduce the complexity and testing burden of your code.

Option 1 — Instance Factory

The first option is to avoid dynamic by-name class loading altogether. Although not always practical, it may be possible in some cases to allow clients to supply their own callback or factory to create objects as required.

For example our O/R mapper could define the following interface:

public interface DomainObjectFactory {
    Object createInstance(String tableName);
}

Clients could supply an instance of this factory when initialising a session, or we could wire it in with our favourite Dependency Injector. This has the advantage that clients can construct objects however they like, rather than placing restrictions on the permitted constructors in the domain classes.

On the other hand it may be risky since a client factory implementation could return shared instances, whereas the framework expects new objects on each call to createInstance(). So long as such requirements are documented, the framework should permit clients to do what they like.

Option 2 — Register Classes

Another option is to add API for clients to register loaded classes in advance. This is best illustrated in client code:

session.registerClassForTable("EVENTS", Event.class);
List events = session.createQuery("from Event").list();

Here the client code has direct visibility of the Event domain class, so it can use a class-literal to pass a Class to the framework API. The framework can now use the supplied class whenever it reads from the EVENTS table.

We can even register classes directly against their names:

session.registerClass("org.example.domain.Person", Person.class);

The framework can use the registered class whenever it needs to create an instance of org.example.domain.Person, but fall back on Class.forName()/TCCL for unregistered names. No other changes are required in the framework.

Note that with this option, the client loses some flexibility and control. By passing a class rather than instances, the client has no opportunity to construct objects in the way it wants them, but instead relies on the framework to construct them.

This is not so bad in the running example of domain objects, but suppose we are talking about some kind of “service” object. Without control over instantiation, clients cannot make the service aware of its “environment” or “context”. Sometimes this forces us to use static fields, thread-local variables or other hack to get hold of our “context” after having been instantiated by a framework.

If the framework allows pre-instantiated objects, then those objects can be pulled out of the OSGi service registry, a Dependency Injection container, or even generated on the fly. So when designing an API, please consider whether the framework really needs to take control of instantiation by requiring Class objects — if not, at least allow the option of passing pre-instantiated objects.

Option 3 — Pass the Loader

This option is the conceptually the simplest, and also has the least impact on existing code: allow clients to pass a ClassLoader into your API. The appropriate time to do this is highly dependent on the nature of the API, but in the O/R mapper it might be done during initialisation of a session:

SessionFactory.createSession(MyClass.class.getClassLoader());
// ... OR ...
session.setDomainClassLoader(MyClass.class.getClassLoader());

To minimise the impact on existing code, this method can be overloaded with another version that takes no argument. Alternatively, clients could pass null to indicate that the framework should attempt to find the right ClassLoader.

Again there is a loss of flexibility for clients since this solution assumes all classes can be loaded from one class loader, which may not actually be the case, but in practice it doesn’t create any real problems.

Summary

Although this post has been rather long-winded, the message is very simple: stop guessing.

If you are writing a library that performs dynamic by-name class loading, do not make assumptions about which ClassLoader to use. Allow client code to specify a class loader, or perform its own class loading. By all means retain your Class.forName() or TCCL-based lookups, but please provide an override for environments in which they will not work.

OSGi Compliance Levels

2010-08-24T00:00:00+00:00

Almost every week, I hear about new and existing Java libraries or frameworks becoming “OSGi Compliant”. This makes me happy as it indicates the traction that OSGi is getting as a platform. The only problem is that “OSGi Compliant” could mean almost anything… and therefore it means precisely nothing.

For example, many libraries can be easily used in OSGi even if they don’t provide an OSGi descriptor (i.e., a MANIFEST.MF containing Bundle-SymbolicName etc.), so long as they steer clear of certain problematic coding patterns such as classpath scanning, unconstrained use of Class.forName() and so on. Such libraries could reasonably be termed “compliant”. But another framework may have been deeply integrated into OSGi so that it can make use of services, receive configuration through Config Admin, and so on.

It seems that we need a vocabulary that will help us distinguish these two cases, and those in between. We need to define, at least informally, some kind of Beaufort Scale for levels of OSGi compliance (if you’re feeling uncharitable towards OSGi you may prefer to call it a Schmidt Pain Index).

The purpose of the exercise would certainly not be to admonish authors of libraries that rate near the bottom of the scale, nor to congratulate those at the top. Rather, it would help authors to communicate the depth of their support for OSGi in a more precise way, and it would help the OSGi community communicate to library authors how they can make their products more useful to us.

To get the ball rolling, here are my initial thoughts. I do this merely to invite feedback rather than to attempt to impose my own classification.

Level	Name	Notes
-1	Non-compliant.	Difficult or impossible to use in OSGi. May be possible to use via an adaption layer or patching of the sources or binaries. For example, the library may rely on classpath scanning or invalid assumptions about the classloading hierarchy.
0	Neutral.	Ships as a plain (i.e., non-bundle) JAR, but can be used straightforwardly in OSGi if wrapped as a bundle or added to `Bundle-ClassPath`.
1	Compliant	Shipped as one or more OSGi bundles with correctly defined `MANIFEST.MF`s. All dependencies are also Compliant and either shipped with the library or available separately from a repository. Package versions follow the Semantic Versioning guidelines (pdf warning).

For many libraries, there is no need to go any higher than level 1. The higher levels are applicable to frameworks, which are more complex and may feature extensibility through plug-ins, custom configuration, logging, etc. Obviously I would like to see all libraries reach at least level 1, but the OSGi community needs to do more to make this easier.

Level	Name	Notes
2	OSGi-Assisted	The framework is aware of OSGi, either in its core or via optional modules. Areas of the framework that are open to extensibility can use services for this, in addition to other mechanisms. The framework can cope with service dynamics without restarting the JVM.
3	OSGi-Powered	The framework is based on OSGi. It publishes services that may be used by other parts of our system. Its own services may be replaced by our own service implementations. It takes advantage of standard OSGi services such as Log Service, Config Admin, Event Admin, HTTP Service. However, despite all this it may still provide a way to run in non-OSGi environments.

A key concern for many frameworks as they seek to rise this pyramid is to remain usable outside OSGi, which is a goal I support. There appears to be very little information available on how to develop frameworks that are able to take advantage of OSGi without introducing a hard dependency on it. This is something I hope to remedy in future posts.

What do you think of this taxonomy? Is it useful, or a waste of time? Should there more levels, or fewer? Please comment below, or link to your own blog.

UPDATE Christian Baranowski blogged some thoughts about independent verification of bundle compliance. That’s a great idea, but my initial goal is less ambitious, i.e. simply to allow library authors to assert something meaningful about their level of OSGi compliance. I think that is a prerequisite before looking to verify those assertions.

UPDATE 2 BJ makes some excellent points in the comments below, which I entirely agree with. Be sure to scroll down and take a look.

OSGi Masterclass with Peter Kriens and Neil Bartlett

2010-08-16T00:00:00+00:00

Peter Kriens and I will be running an OSGi Masterclass in Girona, Spain from the 13th to 15th October 2010.

The course is intended to deepen your knowledge of all aspects of OSGi. It is very hands-on and will cover the following areas:

How to think in Services
Distributed OSGi
Configuration
Persistence
GUIs
etc.

The masterclass is preceded by an optional one-day introductory course on 12th October, for developers who need a refresher on some of the basics of OSGi. Both the introductory day and the masterclass are located in a beautiful hotel near Girona.

Girona airport (GRO) is a RyanAir hub and is therefore cheaply accessible from all over Europe.

Generic OSGi

2010-08-10T00:00:00+00:00

The next version of the OSGi specification — Release 4.3 — will finally support Java Generics. Neither the specification nor the RFC working document are publicly available yet, but the new API has recently been checked into Eclipse Equinox for all to see.

The most noticeable changes are to do with services, naturally. Both the ServiceReference and ServiceRegistration classes now have a single parameter for the service type, and there are new methods on BundleContext that take Class<S> to produce objects of those types. These new methods augment the old String-based methods that now return ServiceReference<?> and ServiceRegistration<?>.

But the biggest single set of changes is on the ServiceTracker class, which now takes two type parameters, S and T, where S is the type of service being tracked and T is the type of objects being produced.

This split has always been an important aspect of ServiceTracker, but it was not obvious before. When a tracker is used in its default form (i.e. without overriding or customising) the two type parameters are the same: we track a service of type S and then typically call getService() to get the current instance of type S. But the call to getService() in fact gives us whatever the tracker’s addingService() method returns, and we can override that.

So in a sense, ServiceTracker is a transformer for services.

For example, the first code sample in my previous post was a ServiceTracker that transformed a service of type IA into a ServiceRegistration<IC>… but you would have had to read the code very carefully to realise that. The ServiceRegistration type wasn’t even mentioned in the addingService() method, so you needed to know that was the return type of registerService().

Here is what the same code will look like in OSGi R4.3 (N.B. I haven’t actually compiled and tested this, but it should work):

public class ForeachA
             extends ServiceTracker<IA, ServiceRegistration<IC>> {

    public ForeachA(BundleContext ctx) {
        super(ctx, IA.class, null);
    }

    @Override
    public ServiceRegistration<IC> addingService(
	                               ServiceReference<IA> ref) {
        IA a = context.getService(ref);
        return context.registerService(IC.class, new C(a), null);
    }

    @Override
    public void removedService(ServiceReference<IA> ref,
                               ServiceRegistration<IC> reg) {
        reg.unregister();
        context.ungetService(ref);
    }
}

Unfortunately this a little bit more verbose than last time but it is safer due to the lack of casts, and more importantly the purpose of the class is clearly stated on the first line, thanks to the type parameters.

When ServiceTracker Trumps Declarative Services

2010-08-05T00:00:00+00:00

I normally recommend using the Declarative Services (DS) specification for working with OSGi services, and in general I agree with the recommendation voiced by others such as Martin and Chris that one should avoid programming against the OSGi APIs.

However I’d like to highlight a use case in which the good old OSGi ServiceTracker API can produce code that is more readable and robust than anything we could achieve with DS.

In DS, we declare components. A component can bind to services with either a unary reference, where the component binds to a single selected instance, or a multiple reference, where the component binds to all available instances. However the component itself remains a singleton.

For example, suppose we declare a component as follows:

@Component
public class C implements IC {
	@Reference
	public void setA(IA a) { /* ... */ }
	public void unsetA(IA a) { /* ... */ }
}

This component binds to the service “IA” with a unary reference. If there are three instances of IA available then the component will bind as follows:

Alternatively if we declare the component like this:

@Component
public class C {
	@Reference (multiple = true)
	public void setA(IA a) { /* ... */ }
	public void unsetA(IA a) { /* ... */ }
}

Then the bind will look like this:

But here is a binding that we cannot achieve in DS (at least, not declaratively):

Fortunately the ServiceTracker code for this is actually quite simple. Suppose we want to create one instance of C for each available instance of IA, and also register those C instances as services under the IC interface. We assume that the C class has a public constructor that takes a parameter of type IA. Here is the complete code (excluding imports and package declaration):

public class ForeachA extends ServiceTracker {
    public ForeachA(BundleContext ctx) {
        super(ctx, IA.class.getName(), null);
    }
    
    @Override
    public Object addingService(ServiceReference ref) {
        IA a = (IA) context.getService(ref);
        return context.registerService(IC.class.getName(),
                                       new C(a), null);
    }
    
    @Override
    public void removedService(ServiceReference ref, Object svc) {
        ServiceRegistration reg = (ServiceRegistration) svc;
        reg.unregister();
        context.ungetService(ref);
    }
}

Of course, we could have achieved this with DS, by writing a component to manage C instances. We would have had to keep track of the instances of IA and the corresponding Cs and ServiceRegistrations, remembering to watch out for concurrency issues. If we were to go that route then we would essentially have reinvented ServiceTracker inside a DS component. The code above is much easier and less error-prone to write.

Things get even more interesting if we need to reference two service types, IA and IB, such that one instance of C is created for each combination of an IA and an IB, i.e.:

To do this in DS is very hard — think about all the things we must keep track of to ensure that we create a C for every IA and every IB irrespective of whether an IA or an IB appears first… and now think about making it thread-safe!

However we can implement this pattern quite easily by chaining together a pair of ServiceTrackers. Here is the code, note that we are reusing the ForeachA class from above but adding to it a constructor parameter of type IB:

public class ForeachB extends ServiceTracker {
    public ForeachB(BundleContext ctx) {
        super(ctx, IB.class.getName(), null);
    }
    
    @Override
    public Object addingService(ServiceReference ref) {
        IB b = (IB) context.getService(ref);
        ForeachA foreachA = new ForeachA(context, b);
        foreachA.open();
        return foreachA;
    }
    
    @Override
    public void removedService(ServiceReference ref, Object service) {
        ForeachA foreachA = (ForeachA) service;
        foreachA.close();
        context.ungetService(ref);
    }
}

Note that while using ServiceTracker necessarily means having a dependency on the OSGi APIs, it’s possible to completely isolate our usage of those APIs from our business logic. The class C is a POJO and receives instances of IA and IB through its constructor, so it remains useable and testable outside of an OSGi environment.

In summary, there are still use cases where the low-level OSGi APIs such as ServiceTracker have an advantage over abstractions such as Declarative Services. I would like to see future versions of the DS specification support this specific use case, but I anticipate that will be difficult because of other things in the spec like Component Factories and factory configurations.

UPDATED: Heiko Seeberger pointed out that the null-checks in my addingService methods were not needed. This makes the code even more concise… thanks Heiko!

The Eclipse/Java 6u21 Blame Game

2010-07-29T00:00:00+00:00

Ed Burnette published an informative article on ZDNet about the recent incompatibility problem with Eclipse and Java 6 update 21. Unfortunately the comments contain a lot of nasty accusations and finger-pointing, with people variously blaming Oracle or Eclipse or both for the mess… and even Ed’s article title seems a trifle unfair on Oracle.

To address those who accuse Eclipse of inappropriately using an internal variable, consider for a moment why this was done. Here are the facts:

The Sun JVM as shipped does not allow enough PermGen memory for a large application such as Eclipse to run. Incidentally many server applications also need more PermGen than is available by default.
There is no standard option to expand the PermGen. The option on the Sun JVM is -XX:MaxPermSize but the “XX” part indicates this option is specific only to the Sun JVM and is not available on other JVMs such as JRockit, IBM J9 etc. In fact using it with those JVMs can cause them to simply emit an error and exit.
Therefore Eclipse needs to detect whether it is on the Sun JVM specifically before it adds the -XX:MaxPermSize option. This cannot be done in Java because by the time the JVM has started it is too late, so it has to be done in the native launcher executable, i.e. eclipse.exe, which is coded in C. At this point the Java system properties are not yet available, but the properties in the exe/dll file are visible.

So I don’t think Eclipse has done anything particularly wrong here. Neither has Oracle, there was no reasonable way for them to know that a simple rebranding would cause such a problem, and they responded very quickly to fix it.

Of course the fix is only a temporary one, as Oracle are going to want to change that variable eventually. So rather than pointing fingers, perhaps the Java community could try to come up with a real long-term solution. Such as a way to manage the JVM’s memory from within the JVM… or getting rid of the cursed PermGen altogether?

Bndtools Now Available in Eclipse Marketplace

2010-07-20T00:00:00+00:00

Bndtools — the OSGi development environment for Eclipse based on Bnd — can now be installed from the Eclipse Marketplace. This is substantially easier than the traditional install procedure for Eclipse plug-ins, which involved copying and pasting an update site URL ino the Update Manager.

Eclipse Marketplace client is available in all of the Eclipse 3.6 distribution packages except for the so-called “Eclipse Classic” distribution… if you are running Eclipse Classic then I strongly recommend installing the Marketplace client – this can be done in the usual way from the Helios update site.

To install Bndtools, open the Help menu in Eclipse and select Eclipse Marketplace. If this is the first time you have used it you may have to select your solution catalog as Eclipse Marketplace (rather than Yoxos Marketplace or any of the other catalogs that may appear in the future) and click Next. Then in the search field type “bndtools” (just “bnd” will work also) and click Go:

</img>

Now click the Install button to the right of the Bndtools entry and follow the rest of the on-screen instructions. These include accepting the licence (of course) and accepting the fact that the plug-in is unsigned (I would love to get rid of this warning, but the price for a code-signing certificate is quite outrageous).

The really neat thing about Marketplace Client is how easy it is to _un_install a plug-in. While I hope you find Bndtools useful enough to keep it around, it’s good to know that it can be removed painlessly if things don’t work out. Just reopen the Marketplace Client, select the Installed tab and click the Uninstall button next to Bndtools (and any other plug-ins that might be cluttering up your SDK):

</img>

Multiple Component Instances with OSGi Declarative Services

2010-07-19T00:00:00+00:00

When we define a component using OSGi Declarative Services (DS), by default we are defining a singleton. For example using the Bnd annotation syntax we can define a component as follows:

@Component
public class MyComponent implements MyInterface {
	
	public void doSomething() { /* ... */ }
	
}

This produces the following XML declaration to be used by the DS runtime:

<component name='org.example.MyComponent'>
	<implementation class='org.example.MyComponent'/>
	<service>
		<provide interface='org.example.MyInterface'/>
	</service>
</component>

At runtime one instance of the org.example.MyComponent class is created and registered as a service under the org.example.MyInterface interface. So far so good. But what if we want more than one of them?

Well, this is possible but it begs the obvious question: “how many do you want?”. We need a way to communicate to the DS runtime how many instances to create. Typically this information is not available to use at build time, so we cannot simply declare it in the annotations or XML. We need a dynamic way to control the number of instances.

If you have ever read the DS specification then you may be thinking that so-called “factory components” are the answer to this conundrum. Sorry but they are not… factory components are mostly a red herring… while they do have their uses, these are a lot more limited than they at first appear.

The real answer is to take advantage of DS’s built-in integration with the Configuration Admin service. Recall that if we use Config Admin to create a configuration record with a “Persistent Identity” (or “PID”) matching the name of our component – in this case org.example.MyComponent – then the component can receive the data in that configuration record simply by declaring an activation method like so:

@Component
public class MyComponent implements MyInterface {

    @Activate
    public void activate(Map<String, String> config) {
        String url = config.get("url");
        // ...
    }

    public void doSomething() { /* ... */ }

}

A potential problem with this component is that it will always be created, even if no configuration record is available. In many cases it’s fine to create a component in the absence of configuration data – the component should simply use defaults for the configuration values. However in other cases this won’t work because no reasonable default values exist. To address this problem, Declarative Services version 1.1 (included in OSGi Release 4.2) added a new “configuration policy” attribute:

@Component(configurationPolicy = require)
public class MyComponent implements MyInterface {

    @Activate
    public void activate(Map<String, String> config) {
        String url = config.get("url");
        // ...

Now the component will only be created if a configuration record with the PID org.example.MyComponent exists.

We have successfully tweaked the cardinality of the component from “always exactly one” to “zero or one”… and in fact this component now already supports “zero to many”. Instead of creating a single configuration record with a PID of org.example.MyComponent, we can create multiple records with a “factory PID” of org.example.MyComponent.

If we do this, and if the component has a configuration policy of “required” as shown above, then we will get multiple instances of the component, with each instance potentially receiving different configuration data. Perfect! DS will continue to maintain all the service references in our component of course.

If you have not used Configuration Admin before and are unsure how to create configuration records, then take a look at Apache Felix FileInstall, which polls a directory for configuration files with the .cfg extension. It works on Equinox and Knopflerfish in addition to Felix.

OSGi and How It Got That Way

2010-03-07T00:00:00+00:00

…some of you may have had occasion to run into mathematicians and to wonder therefore how they got that way. Here in partial explanation perhaps is the story of the great Russian mathematician Nikolai Ivanovich Lobachevsky. — Tom Lehrer.

Probably the most common complaint against OSGi is that it is too complex. For example, quoting Reza Rahman of Caucho:

From a personal standpoint, I did explore OSGi with an open mind. To my dismay, what I found is a very complex, low-level specification that needs a lot of evolution/abstraction to be adapted to be digestible to most enterprise Java environments. It also seems quite overkill for most realistic enterprise needs. Jigsaw in comparison feels much more “clean room”, Java-centric, compact and easy to understand.

Honestly, this complaint always makes me scratch my head a little. If I imagine for a moment that OSGi does not exist and I have been entrusted with the task of designing a new module system for the Java Platform, then the most sensible set of requirements for such a module system lead directly to OSGi as pretty much the most simple possible solution that satisfies those requirements!

Is this just a failure of my imagination? Or is it perhaps a case where Einstein’s Razor (“things should be as simple as possible, but no simpler”) trumps Ockham’s?

On the other hand, I accept that OSGi may not appear particularly simple at first glance, especially if you are not aware of how it got that way. Therefore in this blog post I have attempted to walk through the above thought-experiment of designing OSGi from scratch. I will gloss over a lot of detail of course. I had also intended to perform a similar analysis on Jigsaw, but this post is already a screed of nearly Yeggian proportions, so I will save Jigsaw for a subsequent post.

So without further ceremony, just how did OSGi get that way?

Module Separation

Our first requirement is to cleanly separate modules so that classes from one module do not have the uncontrolled ability to see and obscure classes from other modules. In traditional Java the so-called “classpath” is an enormous list of classes, and if multiple classes happen to have the same fully-qualified name then the first will always be found and the second and all others will be ignored. This is not as unusual a problem as it may appear; it’s quite common if you have a lot of libraries that depend on other libraries. Obscuration is an absolute killer because it can lead to weird errors such as LinkageError, IncompatibleClassChangeError, etc. In fact if we see those errors, we are lucky! If we are unlucky then our system will just quietly do the wrong thing, irrespective of all the up-front testing we may have done before deployment.

The way to prevent uncontrolled visibility and obscuring of classes is to create a class loader for each module. A class loader is able to load only the classes it knows about directly, which in our system would be the contents of a single module (though it can also choose on a class-by-class basis to ask another class loader to supply the class instead — this is known as delegation). Once we do this, each module contains the code that it needs to work, and classes developed together are guaranteed to obtain the classes they expect, even if another module in the system has heavily overlapping fully-qualified class names.

Getting Back Together

If we stop here then modules will be completely isolated and unable to communicate with each other. To make the system practical we need to add back in the ability to see classes in other modules, but we do it in a careful and constrained way. At this point we input another requirement: modules would like the ability to hide some of their implementation details.

In Java there is a missing access modifier between protected/default and public. If I am writing a library and I wish one of my classes to be used by other packages in my library, I must make the class public. But then the class is visible to everybody, including clients external to my library; those clients can now use my internal parts directly. We would like to have a “module” access level, but the problem today is that the javac compiler has no idea where the module boundaries lie, so it cannot perform any checks against such a access modifier. In fact the existing “default” access modifier is also broken because it is supposed to offer access only to members of the same “runtime package” (i.e., a package as loaded by a particular class loader), but again javac has no idea what class loaders might exist at runtime. In this situation, javac punts: it allows the access even though it may result in IllegalAccessErrors later.

The solution we choose in our module system is to allow modules to “export” only portions of their contents. If some part of a module is non-exported then it simply cannot be seen by other modules. But what should be the default? Should we export everything except for some explicitly hidden parts, or should we hide everything except for some explicitly exported parts? Choosing the latter seems to result in more clarity: we can easily look at the export list to determine the visible parts, i.e. the “surface area”, of the module.

Notice that I have not yet specified what exactly is exported… that is deliberate, please bear with me.

What’s the opposite of exporting? Importing, of course. A module that wants to use code from another module can import from it. Now we have another choice… should we import everything that is exported by the other module, or should we import just the parts of it that we need? Again we choose the latter since it results in more clarity: the important thing is what we are importing, not whom we import from.

A Strained Shopping Analogy

This point about imports is very important, so I will digress for a moment into a hopefully humourous, if somewhat exaggerated, analogy about shopping.

My wife and I have different approaches to shopping. I regard it as an onerous chore. When I must purchase something, I find a shop (or group of shops) that sells the items I need and I buy just those items and take them home. I don’t care which shop(s) I visit so long as I get the items I need.

In contrast, my wife goes to a shop she likes and basically buys everything in that shop.

Obviously I believe my approach is superior, because my wife has no control over what she brings home. If her favourite shop du jour changes its inventory then she ends up with different stuff… certainly lots of stuff she doesn’t need and perhaps missing some of the stuff she really does need.

It gets even worse… sometimes she ends up with an item that will not work on its own, because it has a dependency on something else, like batteries perhaps. So she has to go shopping again for batteries, and of course ends up buying the entire contents of the battery shop. Straining the analogy a little now, we could imagine that something else acquired from the battery shop also depends on something else, so she makes yet another trip to another shop, and another, purely to satisfy the requirements of a bunch of items that were not even needed to begin with! This problem is known as “fan-out”.

Hopefully the analogy with a module system is clear. The pathological shopping behaviour is equivalent to a system that forces us to import everything from the modules we declare a dependency on. When importing, we should import what we actually need to use, irrespective of where it comes from and ignoring all the things that happen to be packaged alongside it. We see the fan-out problem acutely when using the Maven build tool, which supports only whole-module (i.e. “buying-the-whole-shop”) dependencies, and as a result must download the entire Internet before it can compile a 200-byte source file.

Granularity of Exports and Imports

What should be the granularity of the things that we import and export from modules? There are many levels of granularity in Java due to the various nesting levels: methods and fields are nested in classes, which are nested in packages, which are nested in modules or JAR files.

It shouldn’t be too hard to see that the sharing level should not be methods and fields. It would be simply insane to export some methods of a class but not others. Not only this, but we could write some methods/fields of a class in one module and some methods/fields in another module! Imagine writing the lists of exports and imports for every method that we wished to share across a module boundary. The complexity of checking this and diagnosing problems at runtime would be horrible, and lots of things would break because classes are just not designed to be split apart at runtime.

At the other extreme, the sharing level should not be whole modules because then modules would not be able to hide parts of their implementation, and importers would routinely suffer from bought-the-whole-shop syndrome.

So the only sane choices are classes and packages… and frankly, choosing classes just isn’t all that sane. Though not as bad as methods/fields, classes are too numerous and too heavily dependent on other classes in the same package for it to be sensible either to list classes as our imports and exports or to split some classes of a package into one module and other classes of the same package into another module.

As a result, OSGi chooses packages. The contents of a Java package are intended to be somewhat coherent, but it is not too onerous to list packages as imports and exports, and it doesn’t break anything to put some packages in one module and other packages in another module. Code that is supposed to be internal to our module can be placed in one or more non-exported packages.

What we lose is the ability to handle so-called “split-packages” cleanly. In OSGi, packages are the fundamental unit of sharing: when you import a package, you get the whole of a package as exported by one module but NO other. There are some strategies for dealing with legacy libraries that insist on smearing the contents of a package across many modules, but it is far better to adjust the modules so that each package is exported in its entirely by only one module.

Package Wiring

Now that we have a model for how modules isolate themselves and then reconnect, we can imagine building a framework that constructs concrete runtime instances of these modules. It would be responsible for installing modules and constructing class loaders that know about the contents of their respective modules.

Then it would look at the imports of newly installed modules and trying to find matching exports. Suppose module A exports package com.foo and module B imports the same package. The framework would notify B that it can obtain classes for the com.foo from module A — this is called wiring. Now when the class loader for B tries to load the class com.foo.Bar, it will delegate to the class loader for A. The process of wiring up imports for a whole module is known as resolution and if all imports are successfully wired then the bundle is resolved, which makes it fully available for use.

An unexpected benefit from this is we can dynamically install, update and uninstall modules. Installing a new module has no effect on those modules that are already resolved, though it may enable some previously unresolvable modules to be resolved. When uninstalling or updating, the framework knows exactly which modules are affected and it will change their state if necessary. There are some additional details to make this work smoothly, for example if a module is doing something important before it is uninstalled or unresolved then we need to send it a notification so it can shutdown cleanly. So, dynamic modules in OSGi do not come for free, since there is no magic involved, but OSGi at least makes it possible.

Some OSGi users prefer to avoid dynamic loading, and that is fine. It is not the most important feature of OSGi, but because it is unique to OSGi it tends to get a disproportionate degree of attention. Nevertheless, nobody will force you to use it, and you will still derive plenty of benefits from OSGi even if you never take advantage of dynamics.

Versions

Our module system is looking good, but we cannot yet handle the changes that inevitably occur in modules over time. We need to support versions.

How do we do this? First, an exporter can simply state some useful information about the packages it is exporting: “this is version 1.0.0 of the API”. An importer can now import only the version that is compatible with what it expects and has been compiled/tested against, and refuse to accept e.g. version 3.0.0. But what if the importer wants version 1.0.0 and only version 1.0.1 is available… the slightly higher version doesn’t sound like it should be a breaking change, so the importer should probably accept 1.0.1 as well as 1.0.0. In fact the importer should specify a range of versions that it will accept, which could be something like “version 1.0.0 through to 2.0.0 but excluding 2.0.0 itself“. The package wiring process supports this and will only wire up an import to an export where the exported version falls within the range specified by the import. For this to work, version numbers need to be ordered and comparable.

How do we know that version 1.0.1 is not a breaking change against 1.0.0? Unfortunately we cannot for sure. OSGi strongly encourages, but cannot enforce, the following semantics on version numbers:

Increments to the Major segment (the first) should be made for non-backwards-compatible changes
Increments to the Minor segment (the middle) should be made for feature enhancements that are backwards compatible.
Increments to the Micro segment (the last) should be made for bug-fixes that do not produce visible feature changes.

If everybody stuck to these semantics then specifying import ranges for any module would be a breeze. Unfortunately the real world is not so simple, so we have to look carefully at compatibility issues when using any external library.

Packaging Modules and Metadata

Our module system will need a way to package the contents of a module along with metadata describing the imports and exports into a deployable unit.

Java already has a standard unit of deployment: the JAR file. JAR files may not be fully-fledged modules but they work just fine for moving around chunks of compiled code, so we have no need to invent something new. So the only question is, where should we put the metadata, i.e. the lists of imports and exports, versions and so on?

Configuration formats seem to be strongly influenced by fads; if we designed our module system between 2000 and 2006 we would probably choose to put the metadata in an XML file somewhere in the JAR file. This would work but it has a number of issues: XML is not particularly efficient to process, especially when we must find it somewhere inside a JAR file and decompress it before parsing. A JAR is just a ZIP, so finding a particular file means reading to the end to find the trailing central directory, then jumping back to the offset indicated by the directory. In other words, we always have to read the whole JAR file, which is a pain for tools that might want to scan large directories containing many modules, for example to search for available modules to satisfy a dependency.

XML is also barely editable by human beings; we need custom editing tools to do it reliably.

On the other hand if we were designing after 2006 then our first impulse might be to use Java annotations. I’m a big fan of annotations when used appropriately, and it’s easy to see the attraction of putting something like @Export(version=”1.0.0”) on a package declaration in the Java source rather than maintaining it in a separate file. But wait a minute… the package declaration is repeated in every source file in the package; must we put the annotation on all of them??

To address this problem, the Java Language Specification (JLS) suggests the use of a special source file named package-info.java. Now what about the metadata that does not belong to any specific package, e.g. the list of imported packages or the name and version of the module itself? This suggests we need another special source file called something like module-info.java.

So far so good, but now consider how the module will be processed. Those special source files will be compiled to bytecode in package-info.class and module-info.class, so you can forget about opening up the JAR file in WinZip to take a squint at the metadata! Any module-scanning tool will not only have to read whole JAR files but also have to be able to process bytecode. And the runtime module system itself will have to create a class loader for the module immediately, just to read its metadata; as it turns out this eliminates a large class of optimisations that are possible if we can defer class loader creation until a class is actually loaded from the module.

As it happens OSGi was designed before 2000, so it did choose either of these solutions. Instead it looked back at the JAR File Specification, where the answer is spelled out: META-INF/MANIFEST.MF is the standard location for arbitrary application-specific metadata. In the words of the specification: “Attributes which are not understood are ignored. Such attributes may include implementation specific information used by applications.”

MANIFEST.MF is designed to be efficient to process, and it is faster than XML, at least. It is somewhat readable; at least as readable as XML, and obviously more readable than compiled Java bytecode. Also the standard jar command line tool always places MANIFEST.MF as the first entry in the JAR file, so tools can scan just the first few hundred bytes of the file in order to obtain the metadata.

Unfortunately MANIFEST.MF is not perfect. For one thing it is quite hard to write by hand due to a rule requiring each line to be no longer than 72 bytes, which becomes problematic if one considers that a single UTF-8 character can be between 1 and 6 bytes. A better approach is to generate MANIFEST.MF using a template in an alternative format; the Bnd tool does this, and so do Maven’s Bundle Plugin and SpringSource’s Bundlor.

In fact, Bnd even includes experimental support for processing annotations such as @Export on source code annotations. These may give us the best of both worlds: the convenience of annotations, combined with the efficiency and runtime readability/toolability of MANIFEST.MF.

Late Binding

The final piece of modularity puzzle is late binding of implementations to interfaces. I would argue that it is a crucial feature of modularity, even though some module systems ignore it entirely, or at least consider it out of scope.

It is well known that interfaces in Java can break the coupling between consumers and producers of functionality. By defining an interface that serves as the contract between a consumer and a producer, neither of them require direct knowledge of the other, so we can put them in different modules that do not depend on each other. Instead they will each have a dependency on the interface, which can live in a third module if we choose. The only question is how to supply instances of the interface to the consumer class, and the most common answer these days is with a Dependency Injection (DI) framework such as Spring or Guice.

Therefore to complete our module system we could simply defer to an existing DI framework. We are seeking simplicity after all, and there can be nothing simpler than declaring a problem to be out of scope, to be solved by somebody else. However this is not quite satisfactory because the DI framework really needs to be aware of the module boundaries. The problem with a traditional approach to DI is that it tends to create huge, centralised configurations which cut across all modules. Peter Kriens calls this the “God Class” problem, in which one component knows everything about every module and requires them all to do its bidding (as an atheist I find this unsatisfactory, but even if you are a theist then I’m sure you agree that we should not create any more Gods than currently exist). These God Classes — or XML configurations — are fragile, difficult to maintain, and negate most of the benefits of separating our code into modules.

Instead we should look for a decentralised approach. Rather than being told what to do by the God Class, let us suppose that each module can simply create objects and publish them somewhere that the other modules can find them. We call these published objects “services”, and the place where they are published the “service registry”. The most important information about a service is the interface (or interfaces) that it implements, so we can use that as the primary registration key. Now a module needing to find instances of a particular interface can simply query the registry and find out what services are available at that time. The registry itself is still a central component existing outside of any module, but it is not “God”… rather, it is like a shared whiteboard.

We do not need to abandon DI, indeed it is still very useful: existing DI frameworks can be used to inject services into other services and to declaratively publish certain objects as services. The DI framework no longer directs the whole system, instead it is a utility employed within individual modules. We could even use multiple DI frameworks, e.g. Spring and Guice together in the same application, which might be useful if we want to integrate a third-party component that uses a different framework from our own preferred one. Finally, the service registry will have a programmatic API for publishing and lookups, but it will only be used for low-level work such as implementing a new DI framework.

Conclusion

Hopefully this explains in broad stokes the reasons why OSGi is the way it is. OSGi will still be accused of being complex, but I believe that any complexity that does exist is there to handle the corner cases in what I have described.

Of course it has its warts. For example, versioning could stand to be improved, especially in the face of third-party libraries with weird versioning schemes. It’s still the right choice to assign meaning to version numbers, but more assistance needs to be available for managing versions and compatibility of APIs. Then there are the legacy libraries that make dodgy assumptions about the existence of a flat system classpath. In my opinion, any library that takes in a class name as a String and calls Class.forName() to obtain an object is broken, since it assumes that all classes are visible to all modules, which is simply not true in any kind of modular system. Unfortunately they cannot all be fixed overnight, so we need strategies to deal with such broken libraries. However they need to be dealt with in exceptional ways that do not compromise the rules of modularity for everybody else.

Coming up next: the motivation and design of Jigsaw, along with a technical comparison of its features alongside those of OSGi. I also intend to point out where it may be appropriate for OSGi to borrow some ideas from Jigsaw.