Managing OSGi Dependency Hell

Question

UPDATE 2: Since my blog is a bit dead the links got degraded so you can view the articles here:

https://www.dropbox.com/s/xvobgzqnl43kcda/Managing_OSGi_Transitive_Dependencies__Part_1____CitizenRandom.pdf?dl=0

https://www.dropbox.com/s/0bdooux4yhrf8lf/Managing%20OSGi%20Transitive%20Dependencies%20%28...pdf?dl=0

https://www.dropbox.com/s/km3mxqah6oy23iq/Why%20using%20Require-Bundle%20is%20a%20bad%20pract...pdf?dl=0

https://www.dropbox.com/s/mtenchtjopcrmr8/How%20many%20ways%20can%20we%20import%20bundles%20in%20OSGi_%20_%20CitizenRandom.pdf?dl=0

https://www.dropbox.com/s/sldxynx3fl8vn61/Managing%20OSGi%20Transitive%20Dependencies%20%282...pdf?dl=0

I have a maven project, using the very famous felix maven bundle plugin configured in my POM.XML in the following way:

<packaging>bundle</packaging>

(...)

<plugin>
            <groupId>org.apache.felix</groupId>
            <artifactId>maven-bundle-plugin</artifactId>
            <extensions>true</extensions>
            <configuration>
                <instructions>
                    <Bundle-SymbolicName>${project.artifactId};singleton:=true</Bundle-SymbolicName>
                    <Bundle-Version>${project.version}</Bundle-Version>
                    <Export-Package>jlifx.*</Export-Package>
                    <!-- <Embed-Dependency>*</Embed-Dependency> -->
                </instructions>
            </configuration>
        </plugin>

And then I have some 1st degree/level dependencies included in my POM as well:

<dependencies>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.11</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.1</version>
    </dependency>
    <dependency>
        <groupId>commons-io</groupId>
        <artifactId>commons-io</artifactId>
        <version>2.4</version>
    </dependency>
    <dependency>
        <groupId>commons-logging</groupId>
        <artifactId>commons-logging</artifactId>
        <version>1.1</version>
    </dependency>
    <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty-all</artifactId>
        <version>5.0.0.Alpha1</version>
    </dependency>
</dependencies>

And now my problem begins... If I do mvn install I'll get my bundle well built with its very great MANIFEST.MF and all, but I won't get the other dependencies bundles, meaning that if I grab my bundle file and drop it on my OSGi framework instance I'll get something like "Unable to resolve 1.0: missing requirement [1.0] osgi.wiring.package; (&(osgi.wiring.package= etc..."

So one way I've found to create my 1st level dependencies' bundles was by creating a profile in my POM like this:

<profiles>
    <!-- http://www.lucamasini.net/Home/osgi-with-felix/creating-osgi-bundles-of-your-maven-dependencies -->
    <!-- -Pcreate-osgi-bundles-from-dependencies bundle:wrap -->
    <profile>
        <id>create-osgi-bundles-from-dependencies</id>
        <build>
            <directory>${basedir}/bundles</directory>
            <plugins>
                <plugin>
                    <groupId>org.apache.felix</groupId>
                    <artifactId>maven-bundle-plugin</artifactId>
                    <version>2.0.1</version>
                    <extensions>true</extensions>
                    <executions>
                        <execution>
                            <id>wrap-my-dependency</id>
                            <goals>
                                <goal>wrap</goal>
                            </goals>
                            <configuration>
                                <wrapImportPackage>;</wrapImportPackage>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>
    </profile>
</profiles>

And this way when I execute mvn -Pcreate-osgi-bundles-from-dependencies bundle:wrap I'll get the bundles, well constructed and working. But, here comes the real deal. Those bundles also have dependencies on their own, so they'll need to have their dependencies wrapped around as bundles. According to numerous webpages, long time ago we had the mvn org.apache.felix:maven-bundle-plugin:bundleall goal to do that for us, but I've tried, it is buggy and return exceptions and it is marked as deprecated, according to Stuart it will be removed in maven 2.4.1 and later (ref: https://issues.apache.org/jira/browse/FELIX-4145).

So my only solution now is to, manually, check each manifest of my 1st level dependencies and go google for the jars containing the required packages, add them to my POM.XML as maven dependencies, and then run mvn -Pcreate-osgi-bundles-from-dependencies bundle:wrap to wrap them as bundles.

This is what is known as the dependency hell...

Is there any way to automate this task of solving nth level dependencies for a maven-bundle osgi project? I.e. make maven study each of my 1st level dependencies' manifest file, read the import packages, look in the central repo for jar's that provide such packages, download them and wrap them as bundles?

Note: Please provide detailed instructions on how to achieve this, don't just link to this tool or that tool that may solve this issue. The major problem with these tools is their lack of examples and documentation. For example bundleall is deprecated, but there seems to be no tool to replace it, at least in their official documentation of maven bundle plugin, which is deprecated so far... I'm certain that I may have crossed with tools capable of doing this, but the lack of documentation forbids the newbie user from knowing that...

Thanks!

---

---

---

---

EDIT-1:

Thank you for your answers so far :) I think I didn't explain my situation in the most proper way and I'm feeling some difficulties doing it just by plain text. Or maybe I din't understand your answer. I'm quite "fresh-new" in OSGi and the only knowledge I have comes from books (OSGi in Action and alike) and from Google.

Imagine that my bundle imports packages A and B. However package A imports package C, and package B imports C also. But now C imports packages D, E, F and G. On the other hand package D imports a ton of other packages and so does E, F and G.

The only bundles I have in my computer are my own bundle and the bundles that provide the packages A and B because they are my 1st level dependencies. However I don't have any of the other required bundles, even if they are installed as jars in my JDK installation folder, I don't have them as bundles, nor even do I know where can I get the jars to wrap them (actually I know but lets imagine I don't).

What I would expect the build tool to do was to run an algorithm similar to the following:

1) Go to my bundle MANIFEST.MF and read the Import-Package field. Enumerate all the required packages and their resp. versions.

2) Search somewhere on the Internet for the jars or bundles of my required libraries.

3) Download each one and check if they are just plain jars or have a valid osgi manifest file (i.e. they are a bundle)

3.1) if they are a bundle, copy them to my bundles/ folder.

3.2) else wrap the jar into a bundle using whatever tool to do that and copy the bundle to my bundle/ folder.

4) Now, for each new bundle downloaded/created repeat the steps 1),2),3) and 4).

What I want as the final result: A bundle for each library to which I have a direct or indirect dependency, so that I can install them on-the-fly in my OSGi Framework instance, such as felix or equinox.

What I don't want:

1) Have to do this manually, because if I try to solve each dependency of dependency I may spent hours or days collecting and wrapping jars.

2) Embed all dependencies into a ubber/mega bundle. That's a bad practice according to several books I've read, it is harder to maintain each dependency individually and also, it corrupts modularity.

Note: my problem is not about the particular task of wrapping a jar into a bundle, but about doing it recursively to each bundle's imports, even if they need to be downloaded from an online repository such as maven's central.

Is there a way to do this automatically or I'm missing something very big about OSGi? So big that I would never need to do this that I'm asking for?

EDIT-2:

Some, if not all, Import-Package dependencies could be solved at run time. Imagine the OSGi Framework trying to start a bundle, but instead of showing the error message of "Unable to resolve 8.0: missing requirement [8.0] osgi.wiring.package;" it would search for that package online, download it and install it on-the-fly. Life would be so much easier.

Mattias

If i understand you correctly, you want to build bundles with a lot of dependency jars inside? Is it possible you understood osgi wrong? If you want to package your full application into an archive you can create an EBA, and use useTransitiveDependencies=true to include all 3. party libs. – Utopian

Hi, no I don't want to package them all in one archive (archive = .jar file right?). I just want maven to generate a bundle archive for each transitive dependency my project has. Ideally he would read my bundle's import-package field, list the package names and versions, go to maven's central repository, fetch the required libraries from there, wrap the ones which are not bundles into bundles, and paste them into my project folder. Then do the same for each transitive dependency. – Mattias

I think you are missunderstanding what a bundle is. A bundle is a container containing all your specific classes, import-statements for other packages (incl versions or version ranges) and export-statements. You can apply all your 3. party libs like commons-lang, commons-io etc to your application and import the necessary statements. Even if 3. party libs have static fields which you need to overwrite in your bundles, you're fine, because in OSGi every bundle is has its own classloader. – Utopian

wrap/bundleall goals are deprecated because they got replaced by Embed-Dependency #10104166 – Utopian

Hi, thanks for your answers. I've added some information to my opening post, if you don't mind reading it. Thanks! – Mattias

Whats your application server, if you use Karaf or websphere you should really take a look at aries.apache.org/modules/applications.html and aries.apache.org/modules/ebamavenpluginproject.html . If you have multiple libs which aren't "osgi-enabled" (which is the only reason for embedding a dependency i think), check the springsource enterprise bundle repository ebr.springsource.com and the apache-servicemix bundles at search.maven.org/… – Utopian

@Mattias I wound up with your exact question. Your blog post link is dead. Curious what you came up with to accomplish your goal. Thanks! – Scent

EDIT: looks like it can be found here now. Thanks for the writeup. web.ist.utl.pt/ist162500/?p=1 – Scent

Hi! I have had almost no time for that blog as you can see... that article is already missing the images, take a look at this backup of it: dropbox.com/s/xvobgzqnl43kcda/… – Mattias

I added more links in the opening post. – Mattias

All the Dropbox links are dead... – Incantatory

Answer 1

Use Conditional-Package from maven-bundle-plugin to recursively inline all required packages in your JAR bundle and activate the tag true.

Talkingto

Could you please post your solution @PedroD, i'm interested :) – Quillet

Answer 2

If you want to wrap jars that not are bundles but just need these jars to be bundles as a librarys in OSGi framework you can wrap it with BND tool. See the link How to create/generate OSGi bundles from existing third-party jars?

But I can see you use very common libaries than already has been converted as osgi bundles. You can find a lot of libraries converted to bundles in the SpringSource Enterprise Bundle Repository.

But if you can't find any library that is not converted to a good OSGi bundle, you can use WRAP protocol from PAX-URL to install these dependencies in OSGi framework. To use these protocol depends on the OSGi framework that you are using but in apache karaf is installed by default. For example, to install a library from maven repository:

root@karaf> osgi:install -s 'wrap:mvn:commons-lang/commons-lang/2.4$Bundle-SymbolicName=commons-lang&Bundle-Version=2.4'

This instruction install the commons-lang library from maven repository into OSGi framework and wrap it as OSGi bundle with the headers appear in the line.

For automatic install dependencies like you say in the second edit, there are several solutions but is a little work. There are two main solutions for provisioning automatically bundles to OSGi framewor, the "Felix OBR repository" and "Felix Provisioning" bundles and Equinox p2 repository. Both of them have console commands to automatically install bundles and features. The problem is that actually I can't find a good public repository of bundles. You need to build your own repository with all bundles than you need.

If you use the maven-bundle-plugin, when you install artifacts into your local maven repository, the plugin update the file "repository.xml" located in the root of your repository to reflect the requires and capabilities of your bundles. This file is a OBR repository file.

Obsolescent

Hi, thanks for helping! Yes I know several tools that aid me wrapping jar files into bundles. My problem is bigger than that. I've edited my opening post (see the bottom text of it) to try to explain in other words what's my problem. Thanks again! – Mattias

Answer 3

What I want as the final result: A bundle for each library to which I have a direct or indirect dependency, so that I can install them on-the-fly in my OSGi Framework instance, such as felix or equinox.

Ok, if I understand your question correctly, you want something like the maven-dependency-plugin?

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <executions>
                <execution>
                    <id>copy-dependencies</id>
                    <phase>package</phase>
                    <goals>
                        <goal>copy-dependencies</goal>
                    </goals>
                    <configuration>
                        <outputDirectory>${project.build.directory}/thridparty-libs</outputDirectory>
                        <overWriteIfNewer>true</overWriteIfNewer>
                        <includeScope>runtime</includeScope>
                        <excludeGroupIds>${project.groupId}</excludeGroupIds>
                        <excludeArtifactIds>...</excludeArtifactIds>
                    </configuration>
                </execution>
            </executions>
        </plugin>

If you face the problem, that you have lots of dependencies without an osgi-Manifest, you should check if they are available in the springsource enterprise bundle repository http://ebr.springsource.com/repository/ and the apache-servicemix bundles at http://search.maven.org/#search|ga|1|g%3A%22org.apache.servicemix.bundles%22 (as mentioned in my comment)

Utopian

This is close, but still doesn't automate most of the job. I think there's a way to configure OBR repositories to do this job, I'll post here any update from my side to share with anyone who's interested the solutions I can find meanwhile. Thanks! – Mattias

Answer 4

try p2-maven-plugin it does exactly what you have described from 1 to 4. its actually using tycho to build p2 repository but you may also use it to collect and bundle your dependencies and transitive dependencies. Im not writing example here because in the link above every think is well documented with examples in an easy way for understanding.

hope this helps

Opsonize