In the recent days and weeks, we've seen an increasing amount of interest in the future of build solutions for applications made up of OSGi bundles. Due to our heavy involvement with OSGi, this is something that is near and dear to our hearts and we've spent a long time looking at customer requirements and solutions for those requirements. In this blog entry, I will outline the requirements that we have identified and present the solutions that we see to these requirements.

I'm very interested in hearing from anyone who has extra requirements, thinks the requirements we have are bogus or has better ideas for solutions.

Requirements for OSGi Build

No duplication of dependency metadata

A real problem with current OSGi builds is that dependency metadata can be duplicated in many places. For dm Server, we have dependencies described in ivy.xml, .classpath and MANIFEST.MF files. Any long-term solution for OSGi builds must require only one location for dependency metadata.

Work with existing build solutions

Given the unique nature of OSGi applications it is tempting to try and start from scratch with the build system. We entertained this idea for some time, but it became apparent that this wasn't workable technically nor was it the best solution for our users.

Our users are heavily vested in the build solution they have chosen, and making them move doesn't seem like the pragmatic approach. Instead, users want a solution that will slot nicely into their Maven or Ant builds.

Generate manifest files automatically

Probably the easiest way to move to OSGi is to start generating manifest files automatically from the associated metadata available for an application module. Ideally, this approach will lead to minimal changes to application code or to the way developers work with that code.

Derive build dependencies from OSGi manifest

Although manifest generation is a valid approach, many users are interested in treating their OSGi manifests as the canonical description of dependency metadata. These users want to drive their builds from the metadata they have written in their manifests. To support easy manifest creation, many users want to see support for manifest templates. A manifest template allows for boilerplate manifest content to be generated automatically, whilst giving users full control over important information such as versions.

This approach uses the manifest as the central dependency descriptor in the IDE and in the offline build. Developers write the OSGi manifest by hand, or with help from tooling, and the manifest is used by the build tooling in a similar way to how Maven uses the pom.xml and Ivy uses ivy.xml.

Using OSGi metadata for resolution allows for the full set of OSGi resolution rules to be applied during build. This will allow for more accurate matching of the build-time dependency resolution to the resolution that will occur at runtime.

A potential drawback of this approach is that developers will need to specify test dependencies in their manifests. Both Maven and Ivy have ways of marking dependencies for use in the test environment only, and a similar solution is needed when using manifests for dependency resolution.

Scenarios

From these requirements, we have derived four main scenarios. Interestingly, each of these scenarios is represented by one or more of the users I interact with frequently for dm Server.

1. Maven and Eclipse - pom.xml-driven

Along with the next scenario, this is the most frequent one we encounter, and it is the one we are currently spending the most time working on. With this approach, developers are carrying out their development tasks as they would for any normal Maven/Eclipse project.

The OSGi manifest file is generated by plugins for Maven and for Eclipse. The developer can choose to have the Eclipse plugin regenerate the manifest as part of the incremental build or on-demand.

Most of the metadata needed to generate the OSGi manifest can be found in the Java code (source/bytecode). However, it is not possible to derive sensible version numbers for dependencies from bytecode. In this case, versions can be extracted from the pom.xml file or from a customisable manifest template.

The manifest template can carry more information than just version numbers, allowing developers to add OSGi attributes and directives along with any custom headers.

2. Ant, Ivy and Eclipse - ivy.xml-driven

This approach is similar to the Maven, pom-driven approach except that dependency metadata will be extracted from the ivy.xml file and not from the pom.xml file.

Developers will have access to Ant tasks to perform manifest generation at the appropriate stage in the build.

The biggest drawback of this approach is the lack of compelling Ivy integration in Eclipse. For Ivy 2, the Eclipse plugin support is sadly lacking. One approach we are considering is to create the Eclipse classpath from the generated manifest. This is tooling that we already support and will be extending for the next two scenarios. Developers will write ivy.xml files, generate their manifest using the Eclipse plugin and have the Eclipse classpath updated automatically.

3. Maven and Eclipse - MANIFEST.MF-driven

In this approach, the OSGi manifest file is the definitive declaration of project dependency metadata. Developers may choose to write the entire manifest by hand, but most likely they will be writing a manifest template and letting our plugins do the rest.

The dm Server tools already come with an Eclipse classpath container that is driven by the manifest and this will be extended to support accurate transitive dependency resolution.

To support testing is this environment, the plugins will allow for both MANIFEST.MF and TEST.MF files to be written, with TEST.MF describing dependencies that apply only for testing and MANIFEST.MF describing dependencies that apply during test and production.

The last remaining piece for this scenario is how to plug into Maven. We are currently investigating two approaches: replacing/integrating with the Maven dependency resolver or generating a pom.xml file. I'm interested in hearing your opinions on which approach you think has most merit and would be most beneficial to you.

4. Ant, Ivy and Eclipse - MANIFEST.MF-driven

This approach is very similar to using Maven in a manifest-driven way. Unlike scenario 2, this scenario does not suffer from the lack of decent Ivy/Eclipse integration because we will using the same manifest-driven Eclipse classpath container as in scenario 3.

Tools for OSGi build

To address the requirements outlined we are building a suite of tools that work alongside Maven, Ant and Ivy to provide an integrated solution for building OSGi applications.

Bundlor

Bundlor is a tool that we have been using internally for some time to generate OSGi manifests. We used it predominantly to create all the bundles that you see in our Enterprise Bundle Repository. Bundlor can be used to generate manifests for legacy libraries that were created without much consideration for OSGi. Bundlor can also be used for generating manifests for bundles that you are creating specifially with OSGi in mind. Bundlor removes most of the donkey-work involved in creating high quality bundle manifests.

Bundlor supports the notion of a manifest template that allows you to customize the generated manifests. Using the template you can add versions and attributes to package imports and exports, exclude packages from exports, add imports when automatic detection is impossible and also add extra manifest headers. The code snippet below shows a manifest from one of our sample applications.

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: GreenPages Service
Bundle-SymbolicName: greenpages
Bundle-Vendor: SpringSource Inc.
Bundle-Version: 1.0
Import-Template: org.springframework.*;version="[2.5.6.A,3.0)"
Excluded-Exports: greenpages.internal

In this sample, the Excluded-Exports header is being used to prevent greenpages.internal from being exposed. The Import-Template header is used to set a version for all package names that match the org.springframework.* pattern. The project code is scanned to determine the set of possible imports and exports. The template file is then used to control which of the imports/exports in this set actually make it into the manifest and what versions they have. Importantly, the generation process will correctly generate uses metadata for you, removing one of the most tedious jobs.

Bundlor will actually support the creation of two manifest files: MANIFEST.MF and TEST.MF. The TEST.MF file will be used to describe dependencies that are needed only during test. This is important for developers who want to use manifest-driven builds but don't want to pollute production manifests with test dependencies.

In the next few weeks we will be releasing our first public milestone of Bundlor. In this release, you will be able to generate manifests for projects from within Ant and Maven, and you'll be able to customise those manifests using a manifest template. In the next few milestones after that, we'll be adding support for automatically detecting versions for dependencies from pom.xml and ivy.xml files.

We'll be switching most of our samples from hand-written manifests to use Bundlor and manifest templates in time for Bundlor 1.0 final. In addition, we will be moving dm Server to Bundlor as well, discarding all our hand-written manifests.

BundlorEclipse

BundlorEclipse is an Eclipse plugin that makes the Bundlor features available directly in your Eclipse IDE. Using this plugin, you can have fully-incremental manifest generation performed as part of your normal Eclipse build lifecycle. If, like me, you are a bit too free and easy with the save button, you can configure BundlorEclipse to only generate manifests on demand.

When combining this with the forthcoming dm Server in-container test framework, you'll be able to launch full dm Server test environments, complete with manifest files for all your bundles, all from within Eclipse.

Offline dependency resolver

OSGi has its own rules for how dependencies between modules are resolved and how these are validated and turned into a consistent class space. Outside of OSGi, build tools work on a flat classpath which lists the JARs and directories to search for classes.

To support a manifest-driven build, we'll be creating an offline dependency resolver that performs OSGi resolution without starting Equinox or dm Server. We'll be pairing this with a tool that will flatten on OSGi resolution graph into a simple classpath for compile and test. This flattening process will be fully configurable by users who will be able to supply policy information to govern how multiple versions in the OSGi graph are mapped to a single version in the classpath.

Manifest classpath container

Using the dependency resolver, we'll be able to improve our manifest classpath container to create a full, transitive classpath for each Eclipse project.

Maven/Ivy dependency resolver/pom.xml/ivy.xml generator

The dependency resolver will also form the core part of our dependency resolver/pom.xml/ivy.xml generator tool. The critical observation here being that as a developer you should expect to get consistent results inside Eclipse and in your build tool.

Why Bundlor and not Bnd?

As you may be aware, Peter Kriens, OSGi Technical Director, makes available a tool called bnd that serves a similar purpose to Bundlor. Our original position was to use bnd for our own projects and to use it as part of our tooling. Indeed, the Spring Dynamic Modules was using bnd to create its bundles long before Bundlor came along. So why did we decide to create Bundlor?

The main reason for creating for creating Bundlor was that we had to support some additional features that weren't present in bnd such as:

• JDT-based source code scanning
• Partial code handling
• Incremental manifest creation

These features are used to support BundlorEclipse. We use JDT-based scanning in place of bytecode scanning in Eclipse to improve the user experience and to get much better integration into the Eclipse build lifecycle. Eclipse handles partially correct code quite well, and we wanted Bundlor to be able to support partial code in much the same way.

In BundlorEclipse, it is possible to configure Bundlor to run every time you save changes in the IDE. To make this performant, we support incremental manifest generation. When you make changes to your code, Bundlor can track what changes have to be made to manifest files and avoid having to recreate all manifests.

Working with IDEs other than Eclipse

We are aware that, although Eclipse is the dominant IDE for Java developers, there is still a significant number of developers running IDEs such as IntelliJ and NetBeans.

Whilst we won't be committing to build any plugins for these IDEs ourselves we will be making all the tools described available as open source enabling other people to create plugins for their favorite IDEs.

Our approach to building our tools has been to try and encapsulate as much functionality as possible in general-purpose libraries. We are committed to working with the NetBeans and IntelliJ teams to help them integrate our tool libraries into their IDEs.

The ideal end result is that no matter what IDE you are using, you can still get access to the same underlying OSGi tools.

Further requirements

The requirements for OSGi build don't stop with the 4 scenarios that I've outlined. There are many more features and enhancements that can improve the experience for developers working with OSGi applications.

Bulk bundle generation

Many shops will have a large backlog of JAR artifacts that are not OSGi bundles. We plan to extend Bundlor to support bulk bundle generation. In this mode, you will be able to save time by pointing Bundlor at a directory of JARs and having Bundlor create bundles for all of them. In addition, to this, Bundlor will be able to use the relationships between these artifacts to work out what the version ranges for dependencies should be. This will reduce the amount of template code you have to write whilst still resulting in high-quality bundles.

Repository-driven code completion

When building dm Server applications with manifest-driven builds, you have to write the initial manifest imports by hand. With support for repository-driven code completion, you will be able to get Eclipse code completion against the dm Server repository that will generate both Java imports and manifest imports. Using this feature, you'll be able to build up your manifest dependencies as needed, without having to switch to the manifest file.