Work around for Docker layer caching not working in Azure Pipeline

Question

I'm hoping to cache a specific stage in my multi-stage Dockerfile that my test stages use in an effort speed the build process up. Otherwise, it builds for unit testing and then for integration testing.

Here is a bare bones example of one of the Dockerfile:

# creating a node base
FROM node:16-slim as node-base
ENV CI=true


# builder-base is used to build dependencies
FROM node-base as builder-base
COPY ./package-lock.json ./package.json ./
RUN npm ci --production


# 'development' stage installs all dev deps and can be used to develop code.
FROM builder-base as development
WORKDIR /app
COPY . . 
RUN npm ci
EXPOSE 4001
CMD ["npm", "start"]


# 'unit-tests' stage 
FROM development AS unit-tests
RUN npm test -- --coverage --testNamePattern=UT: 


# 'integration-tests' stage 
FROM development AS integration-tests
RUN npm test -- --coverage --testNamePattern=IT:

I'd like to cache the development stage, pull it and just run the unit-tests and integration-tests stages without out building development twice.

I found this question which I'm trying to implement:

How to Enable Docker layer caching in Azure DevOps

The top answer there is this:

- task: Docker@2
  inputs:
    containerRegistry: '$(ContainerRegistryName)'
    command: 'login'

- script: "docker pull $(ACR_ADDRESS)/$(REPOSITORY):latest"
  displayName: Pull latest for layer caching
  continueOnError: true # for first build, no cache

- task: Docker@2
  displayName: build
  inputs:
    containerRegistry: '$(ContainerRegistryName)'
    repository: '$(REPOSITORY)'
    command: 'build'
    Dockerfile: './dockerfile '
    buildContext: '$(BUILDCONTEXT)'
    arguments: '--cache-from=$(ACR_ADDRESS)/$(REPOSITORY):latest' 
    tags: |
      $(Build.BuildNumber)
      latest

- task: Docker@2
  displayName: "push"
  inputs:
    command: push
    containerRegistry: "$(ContainerRegistryName)"
    repository: $(REPOSITORY) 
    tags: |
      $(Build.BuildNumber)
      latest

I've repurposed it for my pipeline like so:

# pr.yaml
# # This is triggered by the PR and branch policies
trigger: none

# Specify this to run on the app repo
resources:
  repositories:
  - repository:app
    type: git
    name: app

# Read in the base variable template
variables:
- template: templates/variables.yaml

# Use the ubuntu-latest image
pool:
  vmIMage: $(vmImageName)

# Stages and their templates for the PR pipeline
stages:
# Checks to see what services in the mono repo have changed by comparing
# the PR code to trunk
- template: templates/changed.yaml
  parameters:
    comparedTo: origin/trunk

# Run unit tests for each changed service
- template: templates/services.yaml
  parameters:
    stageName: BuildDev
    stageDisplayName: Build dev stage for services...
    dockerCommand: build
    phrase: build dev
    target: development
    tag: latest

# Run unit tests for each changed service
- template: templates/services.yaml
  parameters:
    stageName: UnitTests
    stageDisplayName: Run unit tests for services...
    dockerCommand: build
    phrase: unit test
    target: unit-tests
    tag: ut-$(Build.BuildNumber)

# Run integration tests for each changed service
- template: templates/services.yaml
  parameters:
    stageName: IntegrationTests
    stageDisplayName: Run integration tests for services...
    dockerCommand: build
    phrase: integration test
    target: integration-tests
    tag: it-$(Build.BuildNumber)

# services.yaml
parameters:
- name: stageName
  default: ''
- name: stageDisplayName
  default: ''
- name: phrase
  default: ''
- name: dockerCommand
  default: ''
- name: target
  default: ''
- name: tag
  default: ''
- name: services
  type: object
  default:
  - admin-v2
  - api
  - portal

stages:
- stage: ${{ parameters.stageName }}
  displayName: ${{ parameters.stageDisplayName }}
  # Run if detectChanges ran successfully
  dependsOn: 
  - Changed  
  - ${{ if eq(parameters.stageName, 'UnitTests') }}:
    - BuildDev
  - ${{ if eq(parameters.stageName, 'IntegrationTests') }}:
    - BuildDev
    - UnitTests
  condition: succeeded()
  jobs:
  # Runs for all other stages

  - ${{ each service in parameters.services }}:
    - template: docker.yaml
      parameters:
        service: ${{ service }}
        stageName: ${{ parameters.stageName }}
        jobName: ${{ service }}${{ parameters.stageName }}
        jobDisplayName: Run ${{ parameters.phrase }} for ${{ service }} service...
        taskDisplayName: Run ${{ service }} ${{ parameters.phrase }} tasks...
        dockerCommand: ${{ parameters.dockerCommand }}
        target: ${{ parameters.target }}
        tag: ${{ parameters.tag }}

# docker.yaml
parameters:
- name: stageName
  default: ''
- name: service
  default: ''
- name: jobName
  default: ''
- name: jobDisplayName
  default: ''
- name: taskDisplayName
  default: ''
- name: dockerCommand
  default: ''
- name: target
  default: ''
- name: tag
  default: ''

jobs:
- job: 
  displayName: ${{ parameters.jobDisplayName }}
  # Handle whether to run for service or not
  variables:
    servicesChanged: $[ stageDependencies.Changed.Changes.outputs['detectChanges.servicesChanged'] ]
  condition: or(contains(variables['servicesChanged'], '${{ parameters.service }}'), eq(variables['Build.Reason'], 'Manual'))
  steps: 
  # Set to app repo
  - checkout: app

  # Create mysecrets.txt primarily for Django system check
  - bash: |
      printenv >> $(dockerFilePath)/${{ parameters.service }}/mysecrets.txt
    displayName: Create mysecrets.txt for ${{ parameters.service }}
    env:
      DJANGO_SECRET_KEY: $(DJANGO_SECRET_KEY)

  - ${{ if not(eq(parameters.stageName, 'BuildDev')) }}:
    # Run the Docker task
    - task: Docker@2
      inputs:
        containerRegistry: $(dockerRegistryServiceConnection)
        command: login

    - script: docker pull $(containerRegistry)/$(imageRepository)-${{ parameters.service }}:latest
      displayName: Pull latest for layer caching
      continueOnError: true # for first build, no cache

    - task: Docker@2
      # Run if there have been changes
      displayName: ${{ parameters.taskDisplayName }}
      inputs:
        command: ${{ parameters.dockerCommand }}
        repository: $(imageRepository)-${{ parameters.service }}
        dockerfile: $(dockerFilePath)/${{ parameters.service }}/docker/Dockerfile
        buildContext: $(dockerFilePath)/${{ parameters.service }}
        containerRegistry: $(dockerRegistryServiceConnection)
        arguments: |
          --target ${{ parameters.target }} 
          --cache-from=$(containerRegistry)/$(imageRepository)-${{ parameters.service }}:latest
        tags: |
          ${{ parameters.tag }}-$(Build.BuildNumber)
      env:
        DOCKER_BUILDKIT: 1

  - ${{ if eq(parameters.stageName, 'BuildDev') }}:
    - task: Docker@2
      # Run if there have been changes
      displayName: ${{ parameters.taskDisplayName }}
      inputs:
        command: ${{ parameters.dockerCommand }}
        repository: $(imageRepository)-${{ parameters.service }}
        dockerfile: $(dockerFilePath)/${{ parameters.service }}/docker/Dockerfile
        buildContext: $(dockerFilePath)/${{ parameters.service }}
        containerRegistry: $(dockerRegistryServiceConnection)
        arguments: --target ${{ parameters.target }}
        tags: |
          ${{ parameters.tag }}
      env:
        DOCKER_BUILDKIT: 1
    - task: Docker@2
      displayName: Pushing ${{ parameters.service }} ${{ parameters.tag }} to ACR
      inputs:
        command: push
        repository: $(imageRepository)-${{ parameters.service }}      
        containerRegistry: $(dockerRegistryServiceConnection)
        tags: |
          ${{ parameters.tag }}

The stageName BuildDev is where the development stage would be built.

Everything runs successfully. The task where the image is pulled shows it being pulled, then task where --target unit-tests runs shows importing cache manifest from ***/app-admin:dev-20211030.11, but it still builds the development and preceeding stages. It pulls it, sees it's there, and decides to build it anyways.

Here are those logs:

# Pull Job
2021-11-02T23:29:33.0494768Z ##[section]Starting: Pull latest for layer caching
2021-11-02T23:29:33.0502467Z ==============================================================================
2021-11-02T23:29:33.0502844Z Task         : Command line
2021-11-02T23:29:33.0503204Z Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2021-11-02T23:29:33.0503544Z Version      : 2.182.0
2021-11-02T23:29:33.0504030Z Author       : Microsoft Corporation
2021-11-02T23:29:33.0504416Z Help         : https://learn.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2021-11-02T23:29:33.0504832Z ==============================================================================
2021-11-02T23:29:33.2083789Z Generating script.
2021-11-02T23:29:33.2098563Z Script contents:
2021-11-02T23:29:33.2099499Z docker pull ***/app-admin-v2:latest
2021-11-02T23:29:33.2100315Z ========================== Starting Command Output ===========================
2021-11-02T23:29:33.2155259Z [command]/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/5e7263fa-2853-4e6d-b303-62fe80cfacdc.sh
2021-11-02T23:29:34.5925241Z latest: Pulling from app-admin-v2
2021-11-02T23:29:34.5933140Z b380bbd43752: Pulling fs layer
2021-11-02T23:29:34.5933539Z 8d36a6ce056a: Pulling fs layer
2021-11-02T23:29:34.5933881Z f54546b42be1: Pulling fs layer
2021-11-02T23:29:34.5934203Z f5bd69d20a35: Pulling fs layer
2021-11-02T23:29:34.5934568Z 21494383f180: Pulling fs layer
2021-11-02T23:29:34.5934902Z 87500a3a7192: Pulling fs layer
2021-11-02T23:29:34.5935238Z debc4a9f3725: Pulling fs layer
2021-11-02T23:29:34.5935558Z 1b67e176d924: Pulling fs layer
2021-11-02T23:29:34.5935890Z d603a960b591: Pulling fs layer
2021-11-02T23:29:34.5936223Z 9e85221572ee: Pulling fs layer
2021-11-02T23:29:34.5943277Z f5bd69d20a35: Waiting
2021-11-02T23:29:34.5943992Z 21494383f180: Waiting
2021-11-02T23:29:34.5944321Z 87500a3a7192: Waiting
2021-11-02T23:29:34.5944642Z debc4a9f3725: Waiting
2021-11-02T23:29:34.5944958Z 1b67e176d924: Waiting
2021-11-02T23:29:34.5945246Z d603a960b591: Waiting
2021-11-02T23:29:34.5945573Z 9e85221572ee: Waiting
2021-11-02T23:29:34.9794967Z 8d36a6ce056a: Verifying Checksum
2021-11-02T23:29:34.9799329Z 8d36a6ce056a: Download complete
2021-11-02T23:29:35.6335949Z f5bd69d20a35: Verifying Checksum
2021-11-02T23:29:35.6337075Z f5bd69d20a35: Download complete
2021-11-02T23:29:35.8084539Z b380bbd43752: Verifying Checksum
2021-11-02T23:29:35.8120185Z b380bbd43752: Download complete
2021-11-02T23:29:35.9459756Z 21494383f180: Verifying Checksum
2021-11-02T23:29:35.9460303Z 21494383f180: Download complete
2021-11-02T23:29:35.9926957Z f54546b42be1: Verifying Checksum
2021-11-02T23:29:35.9927391Z f54546b42be1: Download complete
2021-11-02T23:29:36.1820048Z 87500a3a7192: Verifying Checksum
2021-11-02T23:29:36.1820481Z 87500a3a7192: Download complete
2021-11-02T23:29:36.4513965Z 1b67e176d924: Verifying Checksum
2021-11-02T23:29:36.4514451Z 1b67e176d924: Download complete
2021-11-02T23:29:37.1143768Z d603a960b591: Verifying Checksum
2021-11-02T23:29:37.1144264Z d603a960b591: Download complete
2021-11-02T23:29:37.3920871Z b380bbd43752: Pull complete
2021-11-02T23:29:38.0559222Z 9e85221572ee: Verifying Checksum
2021-11-02T23:29:38.0559774Z 9e85221572ee: Download complete
2021-11-02T23:29:38.5139277Z debc4a9f3725: Verifying Checksum
2021-11-02T23:29:38.5140203Z debc4a9f3725: Download complete
2021-11-02T23:29:39.0212051Z 8d36a6ce056a: Pull complete
2021-11-02T23:29:40.9828384Z f54546b42be1: Pull complete
2021-11-02T23:29:41.1410341Z f5bd69d20a35: Pull complete
2021-11-02T23:29:41.2067833Z 21494383f180: Pull complete
2021-11-02T23:29:41.2833611Z 87500a3a7192: Pull complete
2021-11-02T23:29:54.5480084Z debc4a9f3725: Pull complete
2021-11-02T23:29:54.6097840Z 1b67e176d924: Pull complete
2021-11-02T23:29:54.6823771Z d603a960b591: Pull complete
2021-11-02T23:30:04.3756447Z 9e85221572ee: Pull complete
2021-11-02T23:30:04.3801963Z Digest: sha256:64308db1d461a2aff0deaf31b5bb5694becfb2298f0c474366d1d9b695b0a441
2021-11-02T23:30:04.3836112Z Status: Downloaded newer image for ***/app-admin-v2:latest
2021-11-02T23:30:04.3874744Z ***/app-admin-v2:latest
2021-11-02T23:30:04.4026099Z ##[section]Finishing: Pull latest for layer caching

# Build Job
2021-11-02T23:30:08.6626317Z [command]/usr/bin/docker build -f /home/vsts/work/1/s/admin-v2/docker/Dockerfile --label com.azure.dev.image.system.teamfoundationcollectionuri=https://dev.azure.com/thecompany/ --label com.azure.dev.image.system.teamproject=-dev --label com.azure.dev.image.build.repository.name=production-resources --label com.azure.dev.image.build.sourceversion=809969a7bb0880a135c935c5d66ea0e2bba2c65e --label com.azure.dev.image.build.repository.uri=https://[email protected]/thecompany/-dev/_git/production-resources --label com.azure.dev.image.build.sourcebranchname=main --label com.azure.dev.image.build.definitionname= App PR --label com.azure.dev.image.build.buildnumber=20211102.3 --label com.azure.dev.image.build.builduri=vstfs:///Build/Build/1322 --label image.base.ref.name=nginx --label image.base.digest=sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36 --target unit-tests --cache-from=***/app-admin-v2:latest -t ***/app-admin-v2:ut-20211102.3-20211102.3 /home/vsts/work/1/s/admin-v2
2021-11-02T23:30:08.9603622Z #1 [internal] load build definition from Dockerfile
2021-11-02T23:30:08.9604133Z #1 sha256:7818e7e1291667e1af9ce6f8a463d74e3df7b64001596dd00b927ccc12c37515
2021-11-02T23:30:08.9604573Z #1 transferring dockerfile: 1.05kB done
2021-11-02T23:30:08.9604904Z #1 DONE 0.0s
2021-11-02T23:30:08.9605027Z 
2021-11-02T23:30:08.9605303Z #2 [internal] load .dockerignore
2021-11-02T23:30:08.9605713Z #2 sha256:1c18692a923cc3e70a98431aefc897940bf0c36edf7ae0f3b0b525b4d753b7fb
2021-11-02T23:30:08.9606117Z #2 transferring context: 329B done
2021-11-02T23:30:08.9607436Z #2 DONE 0.0s
2021-11-02T23:30:08.9607554Z 
2021-11-02T23:30:08.9608532Z #3 [internal] load metadata for docker.io/library/node:16-slim
2021-11-02T23:30:08.9609018Z #3 sha256:faa605aa367b596b57bbdc1bdcccade69c92d97d03d44e595a34c7e28b8d594e
2021-11-02T23:30:10.1243750Z #3 DONE 1.2s
2021-11-02T23:30:10.1243963Z 
2021-11-02T23:30:10.1245311Z #12 importing cache manifest from ***/app-admin-v2:latest
2021-11-02T23:30:10.1245833Z #12 sha256:91a404777043fddf396dcad59a4fd8b976e0224c77d860e00947ec3630b83eaf
2021-11-02T23:30:10.1246215Z #12 DONE 0.0s
2021-11-02T23:30:10.1246341Z 
2021-11-02T23:30:10.1246618Z #4 [internal] load build context
2021-11-02T23:30:10.1247014Z #4 sha256:e1bc60dd1feb9ca2db3a89f6b76ec616e7b56215986652ad691e1fc4c108a5aa
2021-11-02T23:30:10.1247447Z #4 transferring context: 713.43kB 0.0s done
2021-11-02T23:30:10.1247777Z #4 DONE 0.0s
2021-11-02T23:30:10.1247897Z 
2021-11-02T23:30:10.1254657Z #11 [node-base 1/1] FROM docker.io/library/node:16-slim@sha256:9ec1ff69c844f2de3a6a2180cd49ca75797d9f2a0fc52bb33c8a672fd0fe7e18
2021-11-02T23:30:10.1255382Z #11 sha256:af5e5b9d07d96a94506820483ad64d714f1bf9a0e5ae75b7b4e7265902c9f941
2021-11-02T23:30:10.1256263Z #11 resolve docker.io/library/node:16-slim@sha256:9ec1ff69c844f2de3a6a2180cd49ca75797d9f2a0fc52bb33c8a672fd0fe7e18 done
2021-11-02T23:30:10.1256894Z #11 sha256:9ec1ff69c844f2de3a6a2180cd49ca75797d9f2a0fc52bb33c8a672fd0fe7e18 1.21kB / 1.21kB done
2021-11-02T23:30:10.1257454Z #11 sha256:ed230d53c9d9820caa9b1bea418c1f835d15ec4d1253160e908ff31fe074ac35 1.37kB / 1.37kB done
2021-11-02T23:30:10.1258009Z #11 sha256:dd74f260f56dccc771f512ef5b2a81345e3bcefcac34c248459da36169be36b2 6.89kB / 6.89kB done
2021-11-02T23:30:10.1258404Z #11 DONE 0.1s
2021-11-02T23:30:10.2748114Z 
2021-11-02T23:30:10.2749533Z #5 [builder-base 1/2] COPY ./package-lock.json ./package.json ./
2021-11-02T23:30:10.2774757Z #5 sha256:e5803272aeaf38a29bf5ca34e14cc0c613c673044fdab9a2c27d837bb2de837c
2021-11-02T23:30:10.2775425Z #5 DONE 0.0s
2021-11-02T23:30:10.2775692Z 
2021-11-02T23:30:10.2776552Z #6 [builder-base 2/2] RUN npm ci --production
2021-11-02T23:30:10.2777798Z #6 sha256:30a22a0c38f1eebba7d10ab9f4cd8f24ce6a4599827350e6b42f38836af226ab
2021-11-02T23:30:12.6879927Z #6 2.477 npm WARN old lockfile 
2021-11-02T23:30:12.6881199Z #6 2.478 npm WARN old lockfile The package-lock.json file was created with an old version of npm,
2021-11-02T23:30:12.6881732Z #6 2.478 npm WARN old lockfile so supplemental metadata must be fetched from the registry.
2021-11-02T23:30:12.6882137Z #6 2.478 npm WARN old lockfile 
2021-11-02T23:30:12.6883737Z #6 2.479 npm WARN old lockfile This is a one-time fix-up, please be patient...
2021-11-02T23:30:12.6884195Z #6 2.479 npm WARN old lockfile 
...

Any ideas why this might be happening and how to prevent it? Really is slowing things down.

Synthetic

I'm a little late to the party, sorry about that. Hope you're still interested in a solution. I think the problem is that you've split up your docker commands into different stages. Tasks/Jobs in different stages not necessarily run on the same agent (see learn.microsoft.com/en-us/azure/devops/pipelines/get-started/…). I would try to put everything into one job, and check if this fixes your problem. – Thigmotropism

@SimonLang Hi, took a long time to get back to this... I'm not 100% that is the issue. While that dev build is a different stage, it is pushed to ACR, then the stage where it is pulled from ACR are all tasks of the same job. It is my understanding agents persist for the duration of the job, so while that would affect different stages, it shouldn't affect these tasks. This should be similar to what they are doing here: https://mcmap.net/q/433888/-how-to-enable-docker-layer-caching-in-azure-devops – Synthetic

Answer 1

I managed to get this working after asking a related question here:

Work around for Docker layer caching not working in Azure Pipeline

Synthetic

This link just refers to this page. – Aguiar

Answer 2

I have the same problem.

Simple Docker file:

FROM node:erbium as base
WORKDIR /app
COPY . .

FROM base as test
RUN npm i
RUN npm run test

FROM base as prod
RUN npm ci
EXPOSE 3000
CMD ["node", "app.node.js"]

Example Pipeline:

jobs:
    -   job: test
        displayName: "test before build"
        steps:
            -   task: Docker@2
                displayName: 'test service inside docker'
                inputs:
                    command: build
                    arguments: '--target test'

    -   job: build
        displayName: "build and push to registry"
        dependsOn: test
        steps:
            -   task: Docker@2
                    displayName: 'build and tag'
                    inputs:
                        command: build
                        dockerfile: '**/Dockerfile'
                        buildContext: '**'
                        arguments: '--target prod'
                        repository: 'myName/myImage'
                        tags: |
                            dev
                            $(Build.SourceVersion)

When I have a different job/task setup, my --target is ignored.

Solution:

I ended up, splitting my Dockerfile into 2 files.

Dockerfile-test

FROM node:erbium as base
WORKDIR /app
COPY . .

FROM base as test
RUN npm i
RUN npm run test

Dockerfile

FROM node:erbium as base
WORKDIR /app
COPY . .

FROM base as prod
RUN npm ci
EXPOSE 3000
CMD ["node", "app.node.js"]

Septuagenarian

Answer 3

Azure Pipeline Docker Caching simply does not work - save your time trying to fix it.

I recommend building your images in a docker - which will allow you to run anywhere and not be dependent on the specific cloud vendor and their bugs

Castiron

Welcome to Stackoverflow! Would you mind backing such claims up with a reproducible argument or a link? – Petiole

Answer 4

You need to add the build argument BUILDKIT_INLINE_CACHE=1. Only then, the image can be used as a cache source afterwards.

Add --build-arg BUILDKIT_INLINE_CACHE=1 to your docker build command and you're all settled :-)

With more advanced multi-stage builds, you might need to go even one step further and use registry cache instead of inline cache.

See official Docker documentation on inline cache.
Some articles on build speed: Speeding up Docker builds in CI and Streamline Your Azure DevOps Pipelines: Advanced Docker Optimizations Unveiled.

Aguiar