Apache Ivy. Transitive dependencies not retrieved

Question

I have 3 projects of the following structure:

App
|  |
  ...
|  |
|  +--lib
|  |    |
|  |    +--...
|  |
|  +--dist
|
Lib
|  |
   ...
|  |
|  +--lib
|  |    |
|  |    +--sublib-1.0.jar
|  |
|  +--dist
|       |
|       +--lib-1.0.jar
|
SubLib
   |
  ... 
   |
   +--dist
        |
        +--sublib-1.0.jar

Which have the following relation:

App <-- Lib <-- SubLib

I am using apache ivy to retrieve the dependencies for both App and Lib. The dependencies are described as follows: ivy.xml of Lib:

<ivy-module version = "2.0">
    <info organisation = "com.test.lib" module = "lib"/>
    <dependencies>
        <dependency org = "com.test.sub.lib" name = "sublib" rev = "1.0" conf = "compile->default"/>
    </dependencies>
</ivy-module>

ivy.xml of App:

<ivy-module version = "2.0">
    <info organisation = "com.test.app" module = "App"/>
    <dependencies>
        <dependency org = "com.test.lib" name = "lib" rev = "1.0" conf = "compile->default"/>
    </dependencies>
</ivy-module>

ivysettings.xml:

<ivysettings>
    <settings defaultResolver = "local"/>    
    <resolvers>
        <filesystem name = "local">
            <artifact pattern = "${ivy.settings.dir}/SubLib/dist/[artifact]-[revision].[ext]"/>
            <artifact pattern = "${ivy.settings.dir}/Lib/dist/[artifact]-[revision].[ext]"/>
        </filesystem>
    </resolvers>    
    <modules>
        <module organisation = "com.test.ivytest" resolver = "local"/>
    </modules>
</ivysettings>

Expected result: after executing ivy:retrieve, both sublib-1.0.jar and lib-1.0.jar to be present in App/lib

Actual result: only lib-1.0.jar is present in App/lib. The generated ivy-report for App does not contain any mention of sublib being a dependency of lib. Nothing of a sort is in ant + ivy logs during build as well.

Note: lib-1.0.jar is not being built as a fat-jar.

What am I missing in this configuration?

---

Update

I've done some thinking, and the only conclusion I came with is that this problem is indeed misconfiguration. Judging by the fact, that transitive dependency is not retrieved, we can positively say that ivy does not have any information of a sort when it resolves lib. And that makes sense, because Lib/dist folder could be anywhere in the file system. The only way to have information about transitive dependency would be having respective ivy.xml somewhere close to that jar. Which is not. This is slightly confirmed by the message in logs [ivy:retrieve] local: no ivy file found for com.test.lib#lib;1.0: using default data. The only way that information is preserved is cache data in %user%/.ivy/cache. There the generated [org]-[artifact]-[conf].xml files do contain the dependency information. So I'm guessing for that to work properly, I will have to use cache on App's resolution level.

Does that make any sense or am I plainly wrong again?

Bodhisattva

ivy:resolve should be called before retrieve. What ere the errors that you get? IS there something like "configuration missing"? – Boeotian

From what I was able to understand is that it's not necessary. I see for [ivy:retrieve] sections in logs that it does perform a resolve. I get no errors, it's just transitive jar dependencies missing. – Bodhisattva

Ok (reading the updated part also) could please do clear ivy cache and run it again. Also add to the question the ant script(s) that you are using to publish lib and sublib. – Boeotian

ivy:cleancache is explicitly called during App.clean ant task. It's just not mentioned. I'm testing one idea at the moment, if it works properly, I'll post it here. – Bodhisattva

Can you show the full ant command you use ? – Calabar

Answer 1

Ok, so the problem was indeed the bad configuration and my lack of understanding it. Here is the detailed explanation how to make it work. I'm not really good with terminology, so I may have misused some words here. Let's look at project configuration and what's what.

Sublib is going to be a runtime dependency for Lib that has a compile-time dependency guava.

SubLib
   |  `lib
   |      `guava-19.0.jar
   |
    `dist
   |    `--sublib-1.0.jar
   |
    `src
        `...

So, we need to make appropriate configurations in SubLib's ivy.xml:

<ivy-module version="2.0">
    <info organisation="com.test.sub.lib" module="sublib"/>

    <configurations>
        <conf name="runtime" visibility="public"/>
    </configurations>

    <dependencies>
        <dependency org="com.google" name="guava" rev="19.0" conf="runtime->default"/>
    </dependencies>
</ivy-module>

Here, by declaring a runtime configuration we state that this ivy.xml describes a module that is a runtime-time dependency. And since guava has no such file we describe it as default. Pretty standard here. Now, for others to know that sublib-1.0.jar actually has a dependency from guava-19.0.jar we need to publish it to a repository, so that such information exists in form of file ivy-[version].xml next to the jar. I chose to publish into the build folder. To do so, ivysettings.xml needs to contain a resolver that helps to match file patterns for publishing and later on on retrieval when we'll resolve from Lib.

<ivysettings>
    <settings defaultResolver="filesystem-resolver"/>

    <resolvers>
        <filesystem name="sublib-resolver">
            <ivy pattern="${ivy.settings.dir}/SubLib/dist/repo/ivy-[revision].xml"/>
            <artifact pattern="${ivy.settings.dir}/SubLib/dist/repo/[artifact]-[revision].[ext]"/>
        </filesystem>

        <filesystem name="filesystem-resolver">
            <artifact pattern="${ivy.settings.dir}/SubLib/lib/[artifact]-[revision].[ext]"/>
        </filesystem>
    </resolvers>

    <modules>
        <module name="sublib" organisation="com.test.sub.lib" resolver="sublib-resolver"/>
    </modules>
</ivysettings>

sublib-resolver will allow to find the corresponding ivy-[revision].xml that has information about the dependencies and the location of jar. Whereas filesystem-resolver will find our guava dependency. Now we just publish sublib with ant by invoking our resolver:

<target name="publish">
        <ivy:publish artifactspattern="${dist.dir}/[artifact]-[revision].[ext]"
                     resolver="sublib-resolver"
                     overwrite="true"
                     pubrevision="${revision}"
        />
</target>

Now to Lib. Lib is going to be a compile-time dependency for App and we describe it as such in ivy.xml and declare SubLib as a runtime dependency for it:

<ivy-module version="2.0">
    <info organisation="com.test.lib" module="lib"/>

    <configurations>
        <conf name="compile" visibility="public"/>
        <conf name="runtime" extends="compile" visibility="public"/>
    </configurations>

    <dependencies>
        <dependency org="com.test.sub.lib" name="sublib" rev="2.0" conf="runtime->compile"/>
    </dependencies>
</ivy-module>

Here's where configuration comes in play and what I didn't understand at first. runtime->compile Left hand side is understandable: sublib was declared as a runtime dependency and we assigned it to runtime conf in it's ivy file. On the arrow's right hand side we state that we want sublib's compile-time dependencies as well. And the one that was configured as such is guava. It will be found by the resolver and retrieved as well. So, we need a resolver for Lib as well, so complete ivysettings.xml file will look like this:

<ivysettings>
    <properties file="${ivy.settings.dir}/ivysettings.properties"/>

    <settings defaultResolver="filesystem-resolver"/>

    <resolvers>
        <filesystem name="sublib-resolver">
            <ivy pattern="${ivy.settings.dir}/SubLib/dist/repo/ivy-[revision].xml"/>
            <artifact pattern="${ivy.settings.dir}/SubLib/dist/repo/[artifact]-[revision].[ext]"/>
        </filesystem>

        <filesystem name="lib-resolver">
            <ivy pattern="${ivy.settings.dir}/Lib/dist/repo/ivy-[revision].xml"/>
            <artifact pattern="${ivy.settings.dir}/Lib/dist/repo/[artifact]-[revision].[ext]"/>
        </filesystem>

        <filesystem name="filesystem-resolver">
            <artifact pattern="${ivy.settings.dir}/SubLib/lib/[artifact]-[revision].[ext]"/>
        </filesystem>
    </resolvers>

    <modules>
        <module name="sublib" organisation="com.test.sub.lib" resolver="sublib-resolver"/>
        <module name="lib" organisation="com.test.lib" resolver="lib-resolver"/>
    </modules>
</ivysettings>

And publish of Lib in Lib's build.xml:

<target name="publish">
        <ivy:publish artifactspattern="${dist.dir}/[artifact]-[revision].[ext]"
                     resolver="lib-resolver"
                     overwrite="true"
                     pubrevision="${revision}"
        />
</target>

Now to the main problem: transitive retrieval. Configurations. In App's ivy.xml we need to specify exactly that we want transitive dependencies. The information that they exist stored in repos is not enough. One must specify it in App's ivy.xml:

<configurations>
    <conf name="compile" visibility="public"/>
</configurations>

<dependencies>
    <dependency org="com.test.lib" name="lib" rev="1.0" conf="compile->compile; compile->runtime"/>
</dependencies>

What happens here is the following: by declaring compile conf we state that App has a compile configuration. The first arrow chain, like before, states that we (compile-configured module App) want to get compile-configured dependency called lib. Left and right arrow sides respectively. And the second arrow set states that we (compile-configured module App) want to get runtime-configured dependencies of lib! Which is sublib. And since it comes together with guava it is retrieved as well.

---

A little messy explanation, and may very not be the most elegant one for a solutioin, but it's the only way I managed to make this work properly at all. If anyone knows any better ways of doing so, any help would be appreciated.

Bodhisattva