Why does my Service Fabric code lock its own PDBs?

Question

I am working on a Service Fabric application that contains a number of stateless services and a single stateful service. When I publish the first time, everything is fine and it's deployed to my local cluster. After this, if I try to package or publish the app without explicitly stopping it first, I get the following error:

CSC : error CS2012: Cannot open 'C:...\ProjectFolder\obj\x64\Debug\ProjectName.pdb' for writing -- 'The process cannot access the file 'C:...\ProjectFolder\obj\x64\Debug\ProjectName.pdb' because it is being used by another process.'

According to process explorer, the PDB is locked by my own ProjectName.exe. This is the single stateful service in my application.

1. Why would my exe lock its own PDBs? I could understand if it was Visual Studio doing it.
   • I see nothing in my own code that should cause this, so I'm assuming it's something in the Fabric code I'm calling.
2. The PDBs are deployed with the application, but it's the files in the original source directory that are locked - why not the PDBs adjacent to the running code?
3. Why do I only see this error relating to the stateful service, and not the stateless services?
   • I suspect this has something to do with the stateful service generating lots of errors on startup, which Fabric may need symbols to properly display.
4. How can I stop it happening - either using the correct PDBs or not using them at all unless I'm debugging through Visual Studio?

Edit: Raised on github. Current workaround for this:

A current workaround at this point in time would be to restrict the Network Service access to the pdb in the build folder (obj\x64\Debug).

Messaline

It the stateful service asp.net core? Do you have any class library projects in your solution? – Momentary

I've avoided any asp.net core or .net core. I had too many issues getting the platform and configuration right for them with fabric. The direct deps from fabric are 64 bit only services. One or two of these, including the stateful service, reference a class library in the solution that's built in any cpu. Is that likely to cause issues? – Messaline

I have seen an issue where this happens after changing the build configuration of the class library. In that case it resolves by a secondary build. Are you stuck in build or can you just do another build and it's good? – Momentary

I can resolve by publishing a second time because I think it stops the fabric app running. But building or packaging twice still fail. I'll confirm this when I'm back at my desk on Monday though. – Messaline

Confirmed - repeated packaging or building doesn't appear to help. I've also tried making sure all projects in my solution are now building under x64 in case Any CPU was the issue, and that also does not appear to change things. – Messaline

Can I ask you to open an issue here: github.com/Azure/service-fabric-issues/issues and share the project (if possible) and a file with the output of handle (technet.microsoft.com/en-us/sysinternals/handle.aspx)? We would like to investigate further. Thanks – Momentary

Sure - I'll need to create a minimal repro as we have a pretty large codebase, which I'm sure I won't be allowed to share in full. – Messaline

Could it be stack trace resolution? When you throw an exception, or otherwise ask for a stack trace, doesn't .NET automatically look for PDBs to see if it can insert line numbers into the stack frames? – Krilov

@JonathanGilbert It's possible, I'm struggling to find much documentation for this though. If that's the cause it still leave the question of why not lock the pdb in the same folder as the exe. From everything I've found so far, that should be checked first. – Messaline

Answer 1

Based on the issue that was raised, it looks like this is now fixed.

"Going to close the issue as we believe both issues have now been addressed by the 5.6 runtime and the 1.6 tooling."

Seems upgrading is now the answer to point 4. If anyone can answer 1-3 I'd be very happy to move the big green tick.

Messaline