OpenShift 3.1 - Prevent Docker from caching curl resource

Question

I have this curl command in my Dockerfile:

RUN curl -H 'Cache-Control: no-cache' -f ${NEXUS_URL}${ARTIFACT_PATH}-${ARTIFACT_VERSION}.war?nocache=true -o $JBOSS_HOME/standalone/deployments/ROOT.war

The first time I ran it I could see the download information. However after that it seems to be caching the remote resource and thus, not updating it anymore:

Step 6 : RUN curl -H 'Cache-Control: no-cache' -f ${NEXUS_URL}${ARTIFACT_PATH}-${ARTIFACT_VERSION}.war?nocache=true -o $JBOSS_HOME/standalone/deployments/ROOT.war
30   ---> Using cache
31   ---> be50412bf6c3

How could I prevent this?

Austenite

Answer 1

According to the OpenShift docs (https://docs.openshift.com/enterprise/3.1/dev_guide/builds.html#no-cache) you can force builds to not be cached using the following syntax:

strategy:
  type: "Docker"
  dockerStrategy:
    noCache: true

This will mean that no steps are cached, which will make your builds slower but will mean you have the correct artifact version in your build.

Salerno

Answer 2

You can bust the cache at a specific Dockerfile instruction using ARG.

In the Dockerfile:

ARG CACHEBUST=1
RUN git clone https://github.com/octocat/Hello-World.git 

On the command line:

docker build -t your-image --build-arg CACHEBUST=$(date +%s) . 

Setting CACHEBUST to the current time means that it will always be unique, and instructions after the ARG declaration in the Dockerfile won't be cached. Note that you can also build without specifying the CACHEBUST build-arg, which will cause it to use the default value of 1 and preserve the cache. This can be used to always check out fresh copies of git repos, pull latest SNAPSHOT dependencies, etc.

Source

Also:

You can use --no-cache, or --build-arg to invalidate the cache. You can minimize the effect of --no-cache by having a base image with all the cacheable commands.

Source

Gyniatrics

I can't control the way docker is run, I mean I am running in a PaaS that calls my dockerfile so I can't pass arguments in docker build ... – Austenite

I'm giving you a +1 as the best pure Docker answer – Austenite

So when run from a shell is there a difference between using CACHEBUST vs --no-cache? My dockerfile (indirectly) clones a branch of a separate git repo that I'm actively changing. (To be completely accurate, it's not directly cloning the second repo but pip installing it.) I'd like to conditionally bust cache, ideally only for the layer that adds that separate repo, whenever HEAD of its remote branch changes. I've thought of dynamically fetching the HEAD commit hash from GitHub and storing it in a file that's added above the ADD command for the repo but it feels hacky for a common scenario. – Rapscallion

Answer 3

I can't control the way docker is run, I mean I am running in a PaaS that calls my dockerfile so I can't pass arguments in docker build

You could try and control the content of your Dockerfile though.

If you can re-generate the Dockerfile before letting the PaaS calling it, that would help making sure the cache is invalidated:

sed -i 's/ROOT.war.*/ROOT.war && echo $(date)'/g' Dockerfile

Armed

Unfortunately I can't do that either. The PaaS server gets my Dockerfile from a Git repository – Austenite

I can definitely modify the Dockerfile pushing a new version to the repo. However that didn't work either. The PaaS is OpenShift 3.x. I start a new build setting the envars that will get to Docker. Whenever I start a build with the same ARTIFACT_VERSION it gets a cached artifact. – Austenite

@Austenite is the ARTIFACT_VERSION linked to the git repo tags? – Armed

Nope, it is used to download the war file from a Nexus server: ${NEXUS_URL}${ARTIFACT_PATH}-${ARTIFACT_VERSION}.war – Austenite

@Austenite So it does not detect the new version from the git repo? – Armed

Anyway, the idea is that the Dockerfile is never modified, that was the point of using envars inside it. – Austenite

@Austenite then the only variable left is ARTIFACT_VERSION. If you cannot change this one, Docker will keep using the cache. – Armed

I'm afraid you're right. If there were a way of generating a random number in a Dockerfile I could concatenate it to the download URL... – Austenite

@Austenite Then why not concatenate the date (YYYMMDDHHMM) and change a symbolic link (based on the same date and minute) to the war? – Armed

How can I do that in the Dockerfile? – Austenite

@Austenite include a $(date +%Y%m%d%H%M) in your curl command: curl -H 'Cache-Control: no-cache' -f ${NEXUS_URL}${ARTIFACT_PATH}-${ARTIFACT_VERSION}.war.$(date +%Y%m%d%H%M) ... – Armed

@Austenite just before making the PaaS calling it. (and updating the symlink to point to the actual war) – Armed

Sorry I don't get it. I start a build with oc start-buid my-proy -e=ARTIFACT_VERSION=1.0-SNAPSHOT. Are you suggesting passing another envar called date? That would work but I would have to find an automatic way to get the date. The oc start-build command is gonna be launched in Jenkins. By the way, thanks for helping me out ;-) – Austenite

@Austenite Or modify ARTIFACT-VERSION itself: -e=ARTIFACT_VERSION=1.0-SNAPSHOT-$(date +%Y%m%d%H%M), provided you can update a symlink pointing to the right war. – Armed

But where does the value of $(date...) come from?? – Austenite

@Austenite it is a linux command which will resolve at runtime into year-month-day-hours-minutes. – Armed

Answer 4

I know this is an old question, but I think my solution is nicer than the ones above. Instead of removing caching completely, you can just invalidate the cache at a particular point. The best way seems to me to be something like:

# No caching from now on!
ADD http://worldclockapi.com/api/json/utc/now build_time.json

I.e. get the current time in a file, which will be different every time you run it and thus invalidate the cache.

Le