How to express inter project dependencies in Eclipse PDE

Question

I am looking for the best practice of handling inter project dependencies between mixed project types where some of the projects are eclipse plug-in/OSGI bundle projects (an RCP application) and others are just plain old java projects (web services modules). Few of the eclipse plug-ins have dependencies on Java projects.

My problem is that at least as far as I've looked, there is no way of cleanly expressing such a dependency in Eclipse PDE environment. I can have plug-in projects depend on other plug-in projects (via Import-Package or Require-Bundle manifest headers), but not of the plain java projects.

I seem to be able to have project declare a dependency on a jar from another project in a workspace, but these jar files do not get picked up by neither export nor launch configuration (although, java code editing sees the libraries just fine).

The "Java projects" are used for building services to be deployed on an J2EE container (JBoss 4.2.2 for the moment) and produce in some cases multiple jar's - one for deploying to the JBoss ear and another for use by client code (an RCP application).

The way we've "solved" this problem for now is that we have 2 more external tools launcher configurations - one for building all the jar's and another for copying these jar's to the plug-in projects. This works (sort of), but the "whole build" and "copy jars" targets incur quite a large build step, bypassing the whole eclipse incremental build feature and by copying the jars instead of just referencing the projects I am decoupling the dependency information and requesting quite a massive workspace refresh that eats up the development time like it was candy.

What I would like to have is a much more "natural" workspace setup that would manage dependencies between projects and request incremental rebuilds only as they are needed, be able to use client code from service libraries in an RCP application plug-ins and be able to launch the RCP application with all the necessary classes where they are needed.

So can I have my cake and eat it too ;)

NOTE

To be clear, this is not so much about dependency management and module management at the moment as it is about Eclipse PDE configuration.

I am well aware of products like [Maven], [Ivy] and [Buckminster] and they solve a quite different problem (once I've solved the workspace configuration issue, these products can actually come in handy for materializing the workspace and building the product)

Triphibious

Answer 1

I never did it so this is a theoretical approach. But I'd try a dependency management system like ivy or maven2.

Since maven2 does much more, then just dependency management, I'd recommend ivy in this case.

Verdun

After getting used to Maven at work, I use it at home for all of my projects, even if they are just little toy projects for my own personal use. The dependency management is wonderful and indispensible. I have not used ivy personally, so I cannot speak to it directly. – Halakah

Answer 2

Eclipse projects depend on each other by virtue of the checkbox in the project's properties (dependent projects?) which is how Eclipse decides which to build. You can set this yourself, but it's usually set when you change your Java build path. It stores the data in the .project file IIRC so once you've gone through the GUI and seen what changes, you can be more flexible in how you apply the others.

However, it sounds like you want to mix and match Jars and Bundles. The easy way to do that is just treat all projects as Java projects. In the PDE project, you can actually go in and tweak the Java build path; it'll complain and say that it isn't the right way to do it, but it will allow you to have a PDE project depend on a Java project without all that fluffy JARing up. Having said that, it wouldn't surprise me if there were runtime problems with this approach - the PDE runtime is likely to not see it that way.

The other approach is to make your JARs themselves PDE/OSGi bundles. After all, an OSGi bundle is nothing more than a JAR with a bit of extra cruft in the Manifest, and it will let you develop and assemble your projects trivially using automatic dependency management. That's probably the easiest one to go for, even if you don't really need the manifest to be present in your bundles. But doing this will mean your PDE app can be shipped with a more modular approach instead of embedding the libraries in each plugin as necessary.

So, PDE can generate OSGi bundles, and that's just another name for JAR + Manifest stuff. You can use a JAR in exactly the same way in other environments (e.g. for your EAR or other client uses) and you can take advantage of the OSGi layer in your app. There's really no reason not to do this given the type of hybrid bundle that you're talking about.

Afrit

I completely agree. We were doing some development with RCP and ran into much the same issue. In the end, we simply bundled up all of our libraries and a simple fashion (depending on how many projects needed them) and then had our plugins depend on them. Much easier to manage and the headless PDE build worked out quite well with this approach. – Carrycarryall

I've tried that and there are indeed runtime problems attached to this approach – Triphibious

Answer 3

Our solution uses an Ant builder to copy the "classes" directories of the plain Java projects directly into the top directory of the plugin project. We skip the JAR building step to save time, and it works pretty well. If the plugin project depends on external JARs that're already built, we copy those in too.

Here's exactly how to set it up in Eclipse 3.5 (sorry for the odd formatting, but that's the only way I could find to preserve indentation):

Create empty "classes" dir in plugin project
Select plugin project, hit F5 to refresh resources
Create new Ant build file in plugin project to copy dependencies (ours is shown below)

Right-click plugin project, select Properties
  Select Builders
  Click "New..." (brings up Edit Configuration dialog)
    Select Ant Builder and click "OK"
    Name your builder (ours is called "PluginProject externals")
    Browse workspace for Buildfile (ours is ${workspace_loc:/PluginProject/copyDependencies.xml})
    click Refresh tab
      check "Refresh resources upon completion", click "Specific resources"
      click "Specify Resources...", check box for the classes dir, click "Finish"
  Click "OK" (closes Edit Configuration dialog)
  Click "Up" to move "PluginProject externals" to top of builder list
Click "OK" (closes Properties dialog)

Open your plugin project's MANIFEST.MF
Click "Runtime" tab
Click "Add..." under "Classpath", select your the "classes" dir and JARs and click "OK"

The manual creation of the empty "classes" directory in the plugin project is so you can tell your new builder to refresh that resource (which doesn't exist yet before the new builder is run). Here's what's in our copyDependencies.xml file:

<project name="Copy dependencies" default="copyDependencies" basedir=".">
  <!--
    This copying is needed because it appears that Eclipse plugins can't
    depend directly on external Eclipse projects.
  -->
  <description>
    Copies external dependency class andd JAR files into this plugin's directory.
  </description>

  <target name="copyDependencies">
    <copy file="../External/JDOM/jdom-1.0/build/jdom.jar" todir="." preservelastmodified="true"/>
    <copy file="../External/Xalan/xalan-j_2_6_0/bin/xalan.jar" todir="." preservelastmodified="true"/>
    <copy file="../External/Xalan/xalan-j_2_6_0/bin/xercesImpl.jar" todir="." preservelastmodified="true"/>
    <copy file="../External/Xalan/xalan-j_2_6_0/bin/xml-apis.jar" todir="." preservelastmodified="true"/>
    <copy todir="./classes/com/arm" preservelastmodified="true">
      <fileset dir="../Utilities/src/com/arm" excludes="**/*.java"/>
    </copy>
  </target>
  <target name="clean" description="Deletes local copies of external classes and JARs.">
    <delete file="jdom.jar" quiet="true"/>
    <delete file="xalan.jar" quiet="true"/>
    <delete file="xercesImpl.jar" quiet="true"/>
    <delete file="xml-apis.jar" quiet="true"/>
    <delete dir="./classes/com/arm/utilities" quiet="true"/>
  </target>
</project>

The only downside to this method seems to be that Eclipse isn't 100% perfect about invoking the external builder when it needs to be invoked, so occasionally you have to do a "Project > Clean..." in Eclipse to force it along.

Balbo

Answer 4

You have my sympathies. I too have battled with this issue and the Wall-of-Silence from the Eclipse devs on such a simple, obvious question: how to declare a dependency from a plugin to a normal Java project (such that it works at runtime)?

I dont think they support it. The only way Ive got around the problem is to create folders inside my plugin projects that are actually links to the bin/ folders of the java projects, and then include these folders into the plugin. That at least works, but its brittle due to the absolute filesystem paths required.

Craving

They are actually trying to improve the situation now and are working on a completely new build system that will (afaik) replace current clunky PDE build with a somewhat more elegant architectural design called b3. So far the development looks promising, but as of yet it is still deep in the initial implementation phase, so it's far too early to have any informed opinion on that... – Triphibious

Unfortunately active work on b3 build system has been all but stopped at this point – Triphibious

Don't use filesystem links, use the link-source option from @ctr0 . At least that lets you use variables like WORKSPACE_LOC/something to avoid absolute paths. – Dx

Basically, you can't have a plug-in depend on something that isn't a plug-in or a package supplied by a plug-in. It's a matter of the plug-in runtime not supporting it because everything in that runtime is supposed to be a plug-in. – Coitus

Answer 5

maybe you can use "project properties" -> "deployment assembly" on Eclipse, and add other projects to your main project. The other projects are seeing like "jars" that automatically are added to your deployment file (war, ear or whatever). Maybe this can be works. At least it works for me. Good luck !!

Ariesandes.

Gibbs

this is nice but the "deployment assembly" feature is available for web/ear projects only and does not work for plug-in projects. – Triphibious

Answer 6

There is a "Link source" option in Build path properties that allows you to define additional source folders for a project. You can selected the "src" folder of another workspace project and rename it to "src2" or whatever you want. This way classes are compiled and deployed into the plug-in project output folder and can be loaded at runtime.

Brooklet

Would you care to elaborate on this. The answer in it's current form is little too terse to make out what you actually did and how or why it helped... – Triphibious

Two projects, A and B. A is our plugin. B is a regular Java project. You want to include code from B when you build A. The link source option lets you tell project A this: there are folders somewhere else that contain source code. I want you to compile the code in those folders just like the other source folders in project A. Make sense? – Dx

Answer 7

With a complex set of build dependencies, I've found Maven2 and Hudson (for CI) to be a pretty nice combination. It took a while to set up the infrastructure and get my head around configuration, but after that, it just worked.

Of course, you are dependent then on Maven2 (or Hudson) support for your build mechanism. I'm not sure how well Eclipse Headless builds are supported. But if the only reason you're using Eclipse headless is to allow the dependencies to be expressed in a single place, do yourself a favour and switch.

Mccutchen

My problem is not so much about managing dependencies as it is about massageing the Eclipse IDE to become aware of the dependencies between these (somewhat incompatible) project types. My goal is to have the entire workspace behave as it does if you were developing only plain java projects with project-to-project dependencies set up (e.g. automatic rebuilds, code completion hints and all the stuff just working). – Triphibious

Answer 8

I am having the exact same problems. We have a set of multiple normal java projects that can be built by Maven and should (currently) all share the same classpath to work properly. These projects can be used to start a server in a non-OSGi environment. Then we also have an eclipse RCP client which uses these projects as one bundle. I am perfectly able to build this one-big-bundle with Maven using the Apache Felix Maven Bundle Plugin and everything works fine. But whenever I change one class in the normal project I have to rebuild the entire bundle. Incremental build and resource linking does not work.

I tried the "solution" with linking binaries/source directories from these projects into the Manifest classpath of the one-big-bundle and it seems to work but it would be a real maintenance nightmare because I even have to link some individual files.

So ironically BECAUSE of the use of OSGi in the client I am actually thinking of collapsing our nice and modular structure with Maven modules into subpackages of only one plugin project. This is a better alternative than having extremely slow development.

The other and better alternative (according to the OSGi guys) which I will investigate first is to make all my Maven modules into OSGi bundles. But this may be a real PITA because I depend on classpath scanning and combining multiple configuration files from several bundles (with sometimes the same name) to form one configuration.

(Specifically we use the Spring framework and we merge multiple persistence.xml files into one persistence context. We also merge several spring XML context files from different modules (all providing a different ASPECT) that should together form one Spring context to be used by Spring-DM.)

Hackler

Yeah, Spring is a b**ch to use in OSGi environment. So is Hibernate. If it was up for me, I would forego using them at all in OSGi altogether – Triphibious