What's wrong with this Ivy changingPattern / SNAPSHOT configuration?

Question

I can't get Ivy to update cache when snapshot dependencies are updated. The resolver (to has the following settings:

<url name="xxx" m2compatible="false" 
     checkmodified="true" changingMatcher="regexp" 
     changingPattern=".*-SNAPSHOT.*">

An example artifact filename (in Artifactory) is:

my-jar-1.999-SNAPSHOT.jar

A detailed Ant log of resolve includes:

[NOT REQUIRED] com.myorg#my-module;1.999-SNAPSHOT!my-jar.jar

There is no POM on the artifact.

The resolver is underneath a chain resolver; they both have all the relevant attributes set. I have read https://issues.apache.org/jira/browse/IVY-938 and https://issues.apache.org/jira/browse/IVY-1221, including all the comments, and AFAICT (perhaps incorrectly!) none of the workarounds are relevant.

Should I give up on snapshots and just use explicit versions with "integration.latest" dynamically versioned dependencies? I fear this may end up failing when we have integration builds happening for multiple major versions. At that point we'll need to split the major versions out into separate repositories, or put the major build number in the artifact name, or something equally clunky, just to make "integration.latest" work.

Centaurus

I see you've revised the question... Snapshot revisions are a special Maven feature. You specify 1.2.3-SNAPSHOT and this is magically transformed into the internally stored file which has a timestamped version number: 1.2.3-201301210911. Take a look into your repo and see the URLs required to retrieve the actual files. – Meng

@MarkO'Connor The repo has untimestamped names like I showed above: my-jar-1.999-SNAPSHOT.jar. – Centaurus

I'm sorry to fixate on Maven... It's introduced the concept of snapshots, when I think ivy's approach of "latest.release" and "latest.integration" are far more standard in how people manage their revision numbers. – Meng

Answer 1

After days of struggle...

The problem was that for

checkmodified="true" changingMatcher="regexp"

to work on a <resolver>, it has to be on every resolver in the hierarchy line - all parent <chain> resolvers and the <url>, <local>, or <ibiblio> resolver at the bottom.

Centaurus

Thank you very much for this hint! I found that explicit changing="true" on dependency also brings remedy but this workaround is better. Unfortunately checkmodified="true" on resolver chain increases total retrieve time due to additional network roundtrips for not changing artivacts from other repos. This encouraged me to avoid use of chain resolver at all as https://mcmap.net/q/1329348/-apache-ivy-chain-tag suggests. This made retrieve faster and also effectively eliminated the issue in question and the need to workaround it. – Pilot

Answer 2

I'm not a fan of using the url resolver when talking to Maven repository managers. The problem is Maven has special and rather unique handling for snapshot revisions..... The url resolver is better suited for use against ivy respositories.

I use Nexus, but the following should also apply to Artifactory. The following settings file sets up Maven Central and my two hosted repositories (Maven repositories come in two flavours, release or snapshot):

<ivysettings>
    <settings defaultResolver="repos" />
    <resolvers>
        <chain name="repos">
            <ibiblio name="central" m2compatible="true"/>   
            <ibiblio name="my-releases" m2compatible="true" root="https://myhost/releases"/>   
            <ibiblio name="my-snapshots" m2compatible="true" root="https://myhost/snapshots"/>   
        </chain>
    </resolvers>
</ivysettings>

You'll notice I'm using the ibilio resolver which has internal logic to decipher Maven's special Snapshot handling.

When I require a snapshot revision I think declare it explicitly as follows:

<ivy-module version="2.0">
    <info organisation="myOrg" module="Demo"/>
    <dependencies>
        <dependency org="myOrg" name="myModule" rev="2.7-SNAPSHOT"/>
        ..
    </dependencies>
</ivy-module>

Under the hood the ibilio resolver is reading the Maven repository meta data files to determine which timestamped artifact should be fetched from the snapshot repository.

Update

I would suggest reading the following presentation:

• Continuous delivery with Maven

It outlines pretty well the Maven philosophy separating releases from dev builds (or snapshots). It also explains one of the very clunky aspects of Maven... Two different ways to publish artifacts...

I suspect what you're trying to do is along the lines of the author which is setup a CD pipe-line. In that case every build is a potential release and should be treated as such (No dynamic dependencies which are allowed by snapshots).

I would suggest limiting snapshots to developer only builds. Only deploy release candidates. The problems with this approach will be in managing lots and lots of releases. Some of the repository managers (Nexus, Artifactory, Archiva) offer "staging" features which make it possible to certify or discard releases that don't pass your quality toll-gates.

Update 2

If you are using ivy to publish snapshots into a Maven repository then there are a couple of issues:

• Ivy doesn't support the publication of snapshots with timestamps
• Ivy doesn't update the Maven module's metadata.xml file

In my opinion time-stamped files is one of the killer features of using snapshots in the first place. With ivy it's only possible to provide the latest file (overwriting the previous latest file).

There are work-arounds to address these issues:

1. As suggested in the second link you can ignore metadata completely (setting the "useMavenMetadata" attribute to false) and default back to ivy's older mechanism of comparing file names. This only fixes the problem for ivy clients.
2. The repository manager should be able to regenerate the metadata files (Nexus at least has a task to do this).
3. Use the Maven ANT task.

The last suggestion is not as crazy as it seems. Firstly it's the only way I know to support timestamped snapshots and secondly the Maven client appears to do a lot of extra processing (updating the module metadata) that is largely undocumented.

Leven

Thanks, but we're not using Maven at all. There are no poms, no significant metadata at all other than the ivy.xml file. – Centaurus

@EdStaub Nexus, Artifactory, Archiva are products originally designed to host Maven repositories. Ivy understands the Maven 2 (or M2) repository format which these repos use to store their data. Ivy is capable of talking natively to these repositories. So yes, client-side there is no POM (functionally replaced by an ivy.xml file) server-side everything is Maven based. Take a look into the repository used to store the snapshots. You'll discover special "maven-metadata.xml" files that list the time-stamped revisions. It's these files which the ibilio resolver understands how to read. – Meng

Thanks much. The metadata files aren't there, but it's probably because I have m2compatible="false" set and/or because I'm using a url resolver (because we need some naming-pattern flexibility that Maven doesn't support). I'll see if I can get that to work. – Centaurus

@EdStaub One other thing to check... Artifactory is more flexible in the way it manages repository layouts. Perhaps the repository you're accessing is not configured to have the M2 layout? It's worth checking.... Personally I'd recommend standardizing on what is the defacto industry standard. Enables Maven based builds to access your artifacts. – Meng

Mostly I've been relying on the Ivy and Artifactory documentation and Jira's, which often reflect differences from Maven and Nexus. No, we're not trying to implement a continuous-delivery pipeline. Snapshot builds are used by developers and by Jenkins. I'm not sure what you mean by "deploy" - if you mean "publish outside of dev", yes, we don't do that with snapshots. This is all pretty way off mark - I just want to get snapshots working as documented in the Ivy docs, which may imply it works only with ibiblio. – Centaurus

AFAICT this is a dry hole. I can't find anything suggesting that Ivy publishes maven-metadata.xml, and lots of email (and my experimenting with an ibiblio resolver, configured as you suggested) to the contrary. My guess is that you use ivy to read and not to write snapshots. – Centaurus

@EdStaub This is the first mention of using ivy to publish snapshots into Artifactory... You stated issue was in resolving or reading the files. There are several known issues properly supporting snapshots when you don't use the standard Maven client.... I'll update my answer with some links. – Meng

Well, I said we weren't using Maven at all, and that there were no poms, no metadata... I guess that wasn't clear enough. Re Tim O'Brien's post: yes, Ivy doesn't support publishing snapshots with timestamps. According to the doc, it's supposed to be using the last-modified date to check for staleness. Re useMavenMetadata: it's not even a supported tag on URL resolvers, because they don't look for maven metadata at all. I suppose I could try an ibiblio resolver with it. – Centaurus

@EdStaub I give up... My parting advice, don't use snapshots. As I keep pointing out they're a Maven repository (server-side) feature. If you do want to use them from ivy then you have to understand how Maven repositories structure their data. – Meng