Ivy, Ant, Jenkins - Is it good idea to to a <ivy:cleancache> on Jenkins builds?

Question

We are going to be using Ivy with Ant, and we'll have Jenkins do our builds. I originally thought that having Jenkins do a <ivy:cleancache/> before running a build would be a good idea. (It would be part of the mandatory "clean" target).

However, I now see that <ivy:cleancache> doesn't simply clean up stuff from <ivy:cachepath>, but really removes the entire $HOME/.ivy/cache directory.

My concern is that if Jenkins does a <ivy:cleancache> on all builds before they start, it will interfere with other builds that Jenkins could be executing.

Is doing an <ivy:cleancache> a good idea, especially if a single user might be doing multiple builds at the same time?

In fact, what happens when you do a <ivy:cachepath pathid="compile.path"/> in multiple projects? Does this also affect something like Jenkins? Will Jenkins become confused if multiple builds are building the compile.cachepath at the same time?

Occurrence

Answer 1

In my opinion running the ivy cleancache task with every build is overkill and does away of one of the main benefits of using ivy, intelligent downloading of 3rd party dependencies.

Having said that as stated in the following related Maven question, all caches can become dirty and should be periodically purged:

When is it safe to delete the local Maven repository?

Couple of recommendations:

Use dedicated Jenkins job(s) to purge ivy cache

My first recommendation is to create a periodic Jenkins job that calls the following clean-all target in your build:

<target name="clean-all" depends="clean">
   <ivy:cleancache/>
</target>

This ensures that Jenkins decides when the cache is purged and you can schedule it to happen outside of the normal build times (for example 2am on 1st of every month)

Isolate each project by using multiple caches

My second recommendation increases the isolation between your project builds. Configure each project to have it's own private cache, using the caches directive. in you ivy settings file.

Miseno

Thanks. Cleaning the cache only adds about 90 seconds to the build (we have a local company Maven repository). It doesn't affect the CI concept, so it wouldn't really hurt for Jenkins to start each build w/ a clean cache. Developers can shut off <ivy:cleancache> by setting ivy.cleancache to false. Ironically, cache cleaning irritates the developers because it lengthens the build, but it doesn't affect them. Jenkins doesn't care, but if Jenkins is running multiple jobs it can be a problem. – Occurrence

Answer 2

Here's what I've decided to do:

I've modified my ivysettings.xml file to have the following:

<ivysettings>
    <properties environment="env." override="false"/>
    <caches
        defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
        resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
    <settings defaultResolver="default"/>
    <include file="${ivy.dir}/ivysettings-public.xml"/>
    <include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
    <include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
    <include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
    <include url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
</ivysettings>

This does two things:

• It defines the Ivy local cache as $HOME/.ivy/cache-$EXECUTOR_NUMBER where $EXECUTOR_NUMBER is the Jenkins executor. This means that each executor gets their own Ivy cache. Thus, if Jenkins is executing more than a one job at a time, each job will be picked up with a different executor, so it will have its own cache. If a job wants to clean the cache, it can go right ahead.
• I've defined the resolve cache to ${basedir}/target/ivy.cache. This gives each job its own resolver cache which is quite small. But, this way ivy resolving doesn't interfere with other jobs if Jenkins is building multiple revisions of the same Ivy project.

The only drawback is that the user's default cache directory is called $HOME/.ivy/cache-$env.EXECUTOR_NUMBER which is not a pretty site. I'd love to make it a more reasonable $HOME/.ivy/cache-0, but I haven't figured that out. However, it doesn't really affect anything at this point.

Now, a developer has a single Ivy cache which contains all of the jars they've downloaded. This way, jars can be shared between projects which speeds up things for developers.

Meanwhile, Jenkins can clean the Ivy cache as often as it is configured. This could be done for each job, or once per day, or per month. However, since the cache is done per executor, I won't have an issue of the cache being cleaned while another job (which would be running on another executor) is depending upon that cache.

This should solve all of the particular issues. The only thing I'd like to do is figure out how to set a default EXECUTOR_NUMBER variable if one isn't already set. I've tried various things like this:

<ivysettings>
    <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
    <properties environment="env." override="false"/>
    <caches
        defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
        resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
    <settings defaultResolver="default"/>
    <include file="${ivy.dir}/ivysettings-public.xml"/>
    <include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
    <include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
    <include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
    <include url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
</ivysettings>

But, to no avail. I've trued changing the override parameters on both the <property> and <properties> file all different ways, but it doesn't quite do what I want.

Occurrence

Answer 3

Just something I have been doing a lot to solve the last problem you have.

You can add this to the properties of the Jenkins Ant Build Steps

another.less.obtrusive.name=${EXECUTOR_NUMBER}

and add to ivysettings.xml.

So it will be "0" for everybody, except for Jenkins because it will inject this property to ANT.

Something on the main question: On Jenkins, I always start new. CI builds should be robust, thorough. Fast is a welcomed by-product, but not a motivation.

Pitchblende

Good idea. What I've done is set it in my ivy.tasks.xml which everyone has to import anyway. I do a <property environment="env"/> then right after that <property name="env.EXECUTOR_NUMBER" value="0"/>. Then, I do my <ivy:setting> task. We have to import the environment anyway because I require our developers to embed the Jenkins build information into jars, wars, and ears. – Occurrence

Took me longer than I care to admit - the 'EXECUTOR' variable name listed in this answer is misspelled. Should be ${EXECUTOR_NUMBER} – Thorp