Migration from packages.config to PackageReferences causes Could not load file or assembly "..." or one of its dependencies. Access is denied

Question

After upgrading our projects to PackageReferences I am facing the following issue on the build server (on-premise AzureDevOps agents)

"C:\Windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.Xaml.targets(193,5): 
Error XC1000: XC1020: Build error occurred in the XAML MSBuild task: 
'Could not load file or assembly 'file:///C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.7.2\Facades\System.Threading.ThreadPool.dll'
 or one of its dependencies. Access is denied.'"

This happens randomly(50% of the time) for different Dlls. Those dlls are in different directories : Nuget restore directory, Reference Assemblies or any other code dll.

On the server, we are using VS 2017.

Now the thing is this started happening when I migrated the project to PackageReferences. We never had any issues with Access Denied before that. And this is exactly the same code base with the same build infrastructure.

What I tried to do :

• I tried to check with the process monitor but there is no "access denied" entry there. I could see that it was read correctly
• Just to be safe I installed the agent on my local machine to see if it's not the build machine problem, and it did happen there as well
• This does not happen when we build locally with MSBuild nor visual studio

Any ideas would be appreciated. At least to point to the direction of the solution.

[Update]

We actually had 3 whole days w/o the issue happening on the server and now it's back.

We also "managed" to get it once while building locally with msbuild(PS script). From the look of that local build and the next one ( passing one triggered right after), we managed to see that nuget was downloading packages that were successfully downloaded in the previous(failed) build. We do not observe that behaviour with consecutive passing builds. Nuget does not download new packages each time unless cleaned.

[Update2] Debug log file for Nuget restore step and build step.

gdrive

Thanks

M

Zimmer

What about your role? Does your account is one of administrator? – Interjection

Yes, it is. When I build an older version of the source code(w/o package ref) it works just fine 100% of the time. Also when tried locally it was installed with my account so admin as well. – Zimmer

Ok. But still think the issue is very wired. Does the VS version in server same with your local build used? If not, please ensure they are same. In addition, do you mind share the completed build pipeline log? Maybe could find something from it. – Interjection

Versions were the same, 2017 on both the server and our local dev env. We will try to update it to vs19 soon just cant do it yet. I will see about build log. Should be possible to share. – Zimmer

Please also try with right click C:\Program Files (x86)\Reference Assemblies folder ->properties-> Add users "Everyone" then check it. After that assign it with the full control. I just reproduce the error message if I modify this file permission. But not sure if it meet yours – Interjection

As mentioned in the post, we are getting this in various different directories. including for example agent\..\s\ directory. But I will still check it, thanks! – Zimmer

I added some updates to the post, and I'm still working on getting logs to share. – Zimmer

Managed to get the log file. If there is anything else that might help with the investigation just let me know. – Zimmer

This is a know bug in .net framwork 4.7.1 and above. As normal, these system files could be loaded from nuget package. But now, in .NET framwork 4.7.1 and above, it will load .NET standard which would load some system files from the local Assembly folder. Also, the security level of Assembly folder has a high security level. So it will cause the error you are facing. You can refer to this github ticket. You can try with the work around listed in that ticket. Then feel free to let me know whether it is work for you. – Interjection

Thanks for the link and for help. I will try it. – Zimmer

Hi, how the things going? Does the error you were facing solved by trying the work around in that thread? If you still facing any issue or error, feel free to leave comment:-) – Interjection

Right now we added the package reference tag in all project files. Issue happens less often now but still happens. I will be doing the second part of the workaround this week. Thx for asking :) – Zimmer

Welcome, feel free to leave comment here if still need any other help about Azuer Devops. – Interjection

Hey, it's been a while. We implemented the workarounds from the link. We are not facing Access denied too often now, but it's still there. We are now looking at limiting the amount of copy to output for dlls. As we suspect it might be one of the causes of the problem. If we find something I will post it here. Mb someone will find it usefull. – Zimmer

@MerlinLiang-MSFT We did try your mentioned steps, however, we are running into another issue. The question is raised here could you also have a look into that and let us know if we are missing something? – Beliabelial

Answer 1

I would suggest running Procmon on the build agents. This will give you a full audit of all filesystem and registry activity during that time. So start it up on the agents, do a couple of builds on them to reproduce the issue, then pause procmon and look at the logs. Filter down by process, file or path to narrow in on your issue. This should give you a much clearer answer on what is happening and the answer will either present itself or it should be very easy to figure out from here.

Macswan