Improve performance of ivy resolve

Question

After migrating from pure ant to ant+ivy my project build times increased from 7s to 26s while incremental rebuilds are now 7s instead of just under 1s (nearly instant).

Most of this time seems to be spent in ivy:resolve which I need to generate classpath using ivy:cachepath.

Is there some way to speed this up, especially rebuilds ?

Saith

Increasing cache ttl helps greatly - rebuild times are down to 2s. I guess that extra second is just ivy inefficiency. Full builds (with populated cache) are 20s for some reason though. Please create an answer that I can mark. – Saith

I thought you don't have to run ivy:resolve every build, just when the dependencies change? I separate resolve out to a install build target that is run by hand rarely. cachepath is still 4s though – Chinchilla

Related: #53623140 – Chinchilla

Answer 1

Another option is to switch off network based resolution and force ivy to use cached data.

See the following answer for more details:

• Resolving Apache Ivy dependencies when offline/disconnected?

Whisky

Answer 2

Are you using <ivy:cleancache>? This is why your rebuilds are so short, but your initial builds are so long.

Ivy is going to be slower than using Ant without Ivy. Ivy has to download each and every jar you specify, plus all dependencies into the Ivy cache (which by default is $HOME/.ivy2/cache on Unix/Mac/Linux and %USERPROFILE%\.ivy2\cache on Windows) before it can begin. You might specify 4 or 5 jars, but these could depend upon others which might depend upon even more.

There really isn't a way to speed up the Ivy downloading of jars¹, but once jars are downloaded, there really isn't a reason to constantly clean the Ivy cache each and every time you do a new project, or when you do a clean.

What you can do is setup your clean, so you can avoid cleaning the Ivy cache unless you specify it:

<taskdef uri="http://ant.apache.org/ivy"
    resource="org/apache/ivy/ant/antlib.xml">
    <classpath>
        <fileset dir="${ivy.dir}">
            <include name="ivy*.jar"/>
        </fileset>
    </classpath>
</taskdef>

<property name="ivy.cleancache"   value="false"/>
<target name="clean">
    <if>
        <istrue value="${ivy.cleancache}"/>
        <then>
           <ivy:cleancache/>
        </then>
    <if>
    <delete dir="${target.dir}"/>
</target>

This way, running ant clean won't scrub your Ivy cache every time, and you can reuse it over and over again. If you want to clean the Ivy cache you need to do this:

ant -Divy.cleancache=true clean

Yes, this is using antcontrib. I use my special Ivy directory configuration to configure Ivy for everyone and while I'm at it, to include definitions for ant-contrib, plus Findbugs, PMD, and other useful tools.

However, it might be possible in Ant 1.9 not to have to do this:

<property name="ivy.cleancache"   value="false"/>
<target name="clean">
    <ivy:cleancache if:true="ivy.cleancache/>
    <delete dir="${target.dir}"/>
</target>

I haven't tried this, but if it works, you don't have to use antcontrib. Of course, you could do this too:

<target name="ivy.cleancache"
   if="ivy.cleancache">
   <ivy:cleancache/>
</target>

<target name="clean"
    depends="ivy.cleancache">
    <delete dir="${target.dir}"
</target>

Then you could specify:

$ ant -Divy.cleancache clean

to clean your Ivy cache and simply put ant clean to clean your build without cleaning the Ivy cache.

---

^1. You might be able to speed up the downloading of jars if you use your own Maven repository like Nexus or Artifactory. These will have their own cache which will store the downloaded third party jars locally. This is a bit faster than going outside your network to find these third party jars, but they all still have to be downloaded. Maybe instead of taking 26 seconds, it might only take 20 seconds.

Ahrens

I normally create two clean targets "clean" and "clean-all", the latter depends on the former and runs the ivy cleancache task. Simpler, I think, and doesn't require the 3rd party "ant-contrib" library. Otherwise I endorse this answer. Caching (ivy and Nexus) improves performance. – Bridgetbridgetown

@MarkO'Connor As I said, I have a special Ivy Directory project which developers include and it configures everything they need for Ivy. TAs a bonus, I add a whole slew of useful third party plugins, including ant-contrib. Also as part and parcel, a javac.macro that displays all warnings, and a jar.macro that automatically adds in a pom.xml to the jar like Maven likes it. However, the last two ways I give do show you how to do this without Ant-contrib. – Ahrens

I stand corrected. I must look at you github project in more detail, thx! – Bridgetbridgetown

@MarkO'Connor A slight confession. We setup our build.xml to automatically do an <ivy:cleancache> on a build, unless the developer sets up a build.properties with ivy;cleancache=false. Our default is to clean the ivy cache, and it's up to the developer to set things up not to do it. The idea is that Jenkins can do a fresh checkout, no other configuration, hit the all target and do everything for the build. The idea is that everything is configured in the project. Any PC with JDK, And, and SVN can build any project without further setup. – Ahrens

I don't use cleancache by default as it would defeat the point of caching. I'm interested where did you find about task level if:true ? What's the if namespace ? – Saith

@MarcinWisnicki <if:true>. Note this is Ant 1.9.1 only! I assumed you used <ivy:cleancache> because you said my project build times increased from 7s to 26s while incremental rebuilds are now 7s instead of just under 1s (nearly instant). When the ivy cache is downloading, it will take a while for all the jars to download, but once done, it shouldn't take more than a few seconds. That's what I'm assuming about rebuilds vs. non-rebuilds. Do you have log set as download-only or default on <ivy:resolve>? – Ahrens

With empty cache it's more like 2 minutes but that's understandable. I did set log to 'download-only'. Thanks for pointing me to if:true - I somehow missed that whole chapter of ant documentation. – Saith

Answer 3

Try this -Divy.checkmodified=false / -Divy.skip=true

Piecework