Azure CLI - "Hard linking failed!" when downloading Azure DevOps Universal Packages into AzureML Compute Instances

Question

We need additional binaries such as tesseract or pretrained models on our ML computes. As those are not preprovisioned in the Azure ML compute instances and we need to restrict external access, we need to load them differently. We try now via Azure artifacts universal packages. Unfortunately this does not work: After having installed the azure devops extension onto the ml compute instance we can download the artifact, but I cannot install it. I get the following error: Encountered an unexpected error.

System.IO.IOException: Hard linking failed! 
 Status: Failed 
 Path: tesseract/tesseract-ocr_4.00~git2288-10f4998a-2_amd64.deb
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClient.DownloadToFileAsync(DedupNode node, String fullPath, Uri proxyUri, EdgeCache edgeCache, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClientWithDataport.DownloadToFileAsync(IDedupDataPort dataport, DedupNode node, String fullPath, Uri proxyUri, EdgeCache edgeCache, CancellationToken cancellationToken)

Salvador

Answer 1

This is due to the usage of hardlinks in a deduplication mechanism which runs as part of ADO's artifact extraction procedure.

An AML Compute Instance's initial directory points at ~/cloudfiles/code, which is hosted on a networked filesystem. Networked filesystems generally lack support for hard-links.

Until a fix is made- as a workaround- you can target a different directory, such as /tmp/ or even ~/. If your files must be on a shared filesystem or one which otherwise lacks hard-link support, you can then move them to your directory of choice.

---

For anyone who may find this page while investigating similar issues with hardlinks, you can use: strace -yy -f -e trace=link,unlink -o /tmp/linktraces.txt -- <command> and read the linktraces.txt file, looking for events which produced an error. That should help identify the destination for which filesystem was attempting a link. If that doesn't help, change to trace=file for a wider scope.

The strace flags above mean the following:

• -yy enables file handle to path conversion.
• -f follows forks of the process, tracing the same events from child processes.
• -e trace=(events) chooses which events to trace based on syscall name or category.
• -o specifies where to write the traces; when omitted, it writes to the console, though it may be a lot of output.

Archiearchiepiscopacy