Azure Pipelines "Cache@2" fails with "##[error]The system cannot find the file specified"

Question

I'm using Azure Pipelines with hosted builds to build a web project. Our build times were hitting 10-15 minutes, with most (5-10 minutes) of the time spent doing npm install. To speed this up, I'm trying to use the Cache task (https://learn.microsoft.com/en-us/azure/devops/pipelines/caching/?view=azure-devops).

However, when the auto-added task Post-job: Cache runs, it always errors out with:

##[error]The system cannot find the file specified

The host server is Windows Server 2017.

Here is my entire build YAML

# Node.js with Vue
# Build a Node.js project that uses Vue.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript

trigger:
- develop

pool:
  name: Default

variables:
  FONTAWESOME_NPM_AUTH_TOKEN: $(FONTAWESOME_NPM_AUTH_TOKEN_VARIABLE)
  npm_config_cache: $(Pipeline.Workspace)/.npm


steps:
- task: DutchWorkzToolsAllVariables@1

- task: NodeTool@0
  inputs:
    versionSpec: '10.x'
  displayName: 'Install Node.js'


- task: Cache@2
  inputs:
    key: 'npm | "$(Agent.OS)" | package-lock.json'
    path: $(npm_config_cache)
    cacheHitVar: NPM_CACHE_RESTORED

- task: Npm@1
  displayName: 'npm install'
  inputs:
    command: 'install'
  condition: ne(variables.NPM_CACHE_RESTORED, 'true')


- task: Npm@1
  displayName: 'npm run build'
  inputs:
    command: 'custom'
    customCommand: 'run build'

- task: CopyFiles@2
  inputs:
    SourceFolder: '$(Build.Repository.LocalPath)\dist'
    Contents: '**'
    TargetFolder: '$(Build.StagingDirectory)'
    CleanTargetFolder: true

- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'drop'
    publishLocation: 'Container'    

Cache task output:

Starting: Cache
==============================================================================
Task         : Cache
Description  : Cache files between runs
Version      : 2.0.0
Author       : Microsoft Corporation
Help         : https://aka.ms/pipeline-caching-docs
==============================================================================
Resolving key:
 - npm               [string]
 - "Windows_NT"      [string]
 - package-lock.json [file] --> F93EFA0B87737CC825F422E1116A9E72DFB5A26F609ADA41CC7F80A039B17299
Resolved to: npm|"Windows_NT"|rbCoKv9PzjbAOWAsH9Pgr3Il2ZhErdZTzV08Qdl3Mz8=
Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session zzzzz
Information, Getting a pipeline cache artifact with one of the following fingerprints:
Information, Fingerprint: `npm|"Windows_NT"|rbCoKv9PzjbAOWAsH9Pgr3Il2ZhErdZTzV08Qdl3Mz8=`
Information, There is a cache miss.
Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session zzzzz
Finishing: Cache

Post-job: Cache output:

Starting: Cache
==============================================================================
Task         : Cache
Description  : Cache files between runs
Version      : 2.0.0
Author       : Microsoft Corporation
Help         : https://aka.ms/pipeline-caching-docs
==============================================================================
Resolving key:
 - npm               [string]
 - "Windows_NT"      [string]
 - package-lock.json [file] --> 2F208E865E6510DE6EEAA6DB0CB7F87B323386881F42EB63E18ED1C0D88CA84E
Resolved to: npm|"Windows_NT"|OQo0ApWAY09wL/ZLr6fxlRIZ5qcoTrNLUv1k6i6GO9Q=
Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session zzzzz
Information, Getting a pipeline cache artifact with one of the following fingerprints:
Information, Fingerprint: `npm|"Windows_NT"|OQo0ApWAY09wL/ZLr6fxlRIZ5qcoTrNLUv1k6i6GO9Q=`
Information, There is a cache miss.
Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session zzzzz
##[error]The system cannot find the file specified
Finishing: Cache

How can I fix my build definition so the caching works?

Nightrider

I think the problem is a chicken-and-egg one. Basically, the cache key is based on the hash of the packages-lock.json file, but those file contents may change during the subsequent npm install. When the post-build task runs, it recalculates the key (?) and fails to find the "new" cache folder. – Nightrider

Hi @Nightrider Did you tried the solution provided by Florian. It seems like the right answer. – Organist

At first pass, the solution by @Florian-Labranche did not seem to work. I've been pulled to other things for a while, but will circle back to this in a few weeks. Ultimately the pipeline DOES work (without the Cache@2 task) ... just very slowly. – Nightrider

Answer 1

@Levi Lu-MSFT was right in his comment but there's a gotcha.

@FLabranche has a working solution in his answer but I believe reasoning is not quite right.

The problem

npm install and @Cache task are looking for the npm cache at different locations. Consider the flow when pipeline runs for the first time:

1. @Cache task: does nothing since there's no cache yet.
2. npm i (or npm ci) task: installs packages in node_modules/ and updates the npm cache at default location. Default location is ~/.npm on Linux/Mac and %AppData%/npm-cache on Windows. On Linux hosted cloud agent the absolute path will be /home/vsts/.npm.
3. (... more tasks from your pipeline)
4. Post-job @Cache task (added implicitly): reads the npm cache found at user-provided location to store it for future reuse. User-provided location is set by the npm_config_cache: $(Pipeline.Workspace)/.npm variable. On Linux hosted cloud agent the absolute path will be /home/vsts/work/1/.npm.

As a result, @Cache task fails with tar: /home/vsts/work/1/.npm: Cannot open: No such file or directory.

Solution

Make npm install and @Cache task use the same npm cache location.

One option suggested by Levi Lu is to update the npm config with npm config set cache $(npm_config_cache) --global but it won't work in the pipeline (at least it didn't work for me in an Azure-hosted Linux agent): Error: EACCES: permission denied, open '/usr/local/etc/npmrc'

npm ci --cache $(npm_config_cache) updates the npm cache location for a single call and it does work in this case. It feels a bit hacky though since --cache option is not even documented on the npm website.

All in all this code worked for me:

variables:
  NPM_CACHE_FOLDER: $(Pipeline.Workspace)/.npm

steps:
- task: Cache@2
  displayName: Cache npm dependencies
  inputs:
    key: 'npm | "$(Agent.OS)" | package-lock.json'
    restoreKeys: |
      npm | "$(Agent.OS)"
      npm
    path: $(NPM_CACHE_FOLDER)

- script: npm ci --cache $(NPM_CACHE_FOLDER)
  displayName: 'Install npm dependencies'

...

Striker

Excellent summary. This level of detail should honestly be included in Microsoft's official documentation. One clarification: does deleting the task "Cache@2" completely prevent subsequent runs from using the cache? I understand that cache cannot be deleted (only relocated), but I still have the power to ignore cache entirely in subsequent runs, right? – Hubby

Answer 2

The keys generated with your package-lock.json differ between the two tasks. It happens when the file is modified. Here, they're modified by your npm install task.

You can use the restoreKeys option when configuring the Cache task to fall back onto the latest cache entry. And I think you don't need the 'npm install' task.

Could you try replacing this :

- task: Cache@2
  inputs:
    key: 'npm | "$(Agent.OS)" | package-lock.json'
    path: $(npm_config_cache)
    cacheHitVar: NPM_CACHE_RESTORED

- task: Npm@1
  displayName: 'npm install'
  inputs:
    command: 'install'
  condition: ne(variables.NPM_CACHE_RESTORED, 'true')

By this definition :

- task: Cache@2
  inputs:
    key: 'npm | "$(Agent.OS)" | package-lock.json'
    restoreKeys: |
       npm | "$(Agent.OS)"
       npm
    path: $(npm_config_cache)
  displayName: Cache npm

- script: npm ci --cache $(npm_config_cache)

Rover

I can't comment on the main thread, but could you try mixing Levi Lu and my solution together by using npm ci --cache $(npm_config_cache) instead of npm ci ? – Rover

npm install is a step that comes after the cache step so how can it affect the lock file's hash? – Variorum

Answer 3

You can log into your Windows Server 2017 server and check if the folder $(Pipeline.Workspace)/.npm is created and the dependencies are stored inside.

I copied and tested your yaml. It worked both on local agent(win2019) and cloud agents. You can try to run your pipeline on the cloud agents or other agents with newer system to check if it is the agent that cause this error.

Organist

I do in fact have a directory in the appropriate place: c:\agent2\_work\40\.npm and it looks like its filled with cache stuff. So, why is the auto-generated task reporting an error, and thus erroring the build? – Nightrider

Could you try adding an additional npm task before npm install to run npm config set cache $(npm_config_cache) --global to override the npm_config_cache environment variable to path $(Pipeline.Workspace)/.npm. Or run npm install --cache $(npm_config_cache) for your npm install task. – Organist

Answer 4

Yesterday, I was able to get it working with no issue at all on a self-hosted machine agent by using this:

- task: Cache@2
  inputs:
    key: '**/package-lock.json, !**/node_modules/**/package-lock.json, !**/.*/**/package-lock.json'
    path: '$(System.DefaultWorkingDirectory)/node_modules'
  displayName: 'Cache Node Modules'

Today, trying to work on a hosted agent today and this doesn't cut it at all. Aggh, Back to the grinding board. Anyhow, maybe could work for you on your self-hosted pipeline

Halcyon

Answer 5

This seems to be related to this open issue.

I have resolved the problem by switching the build agent pool to hosted and using windows-latest image.

pool:
  vmImage: 'windows-latest'

Allanadale

Not acceptable in my case, since I need to do hosted builds. Looking for tar might be an interesting avenue to solution though. – Nightrider

@Nightrider - Did you get the tar to work? I'm trying to get tar on the build server but unable to. – Stackhouse