Application Crashes With "Internal Error In The .NET Runtime"

Question

We have an application written against .NET 4.0 which over the weekend crashed, putting the following message into the event log:

Application: PnrRetrieverService.exe Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP 791F9AAA (79140000) with exit code 80131506.

This is on a Windows Server 2003 R2 Standard Edition box. Googling this error hasn't turned up anything pertinent. For example, this isn't occurring in VS Studio, but instead on a production box; when the service was eventually restarted, it experienced no further problems.

How does one go about diagnosing a bug in the .NET Runtime?

Hereabout

If this is the first time this error has happened, then I would look into anything that has changed in the last few days to a week. – Canso

Answer 1

with exit code 80131506

That's a nasty one, ExecutionEngineException. Starting with .NET 4.0, this exception immediately terminates the program. The generic cause is corruption of the state of the garbage collected heap. Which in turn is invariably caused by unmanaged code. The exact location in code at which this exception is raised isn't helpful, the corruption usually occurred well before the damage is detected.

Finding the exact cause for this is going to be difficult. Review any unmanaged code your service might be using. Suspect environmental problems if there is no obvious candidate, misbehaving malware scanners are notorious. If it repeats very poorly then suspect hardware problems like soft RAM errors.

Paul

I've had problems with SQL CE 3.5 corrupting the heap, causing exceptions in ntdll.dll and .NET runtime errors. – Judicature

Fortunately, this was the exact error code I was trying to find; but are all the CLR exit codes and what they mean listed somewhere? (in other words, how did you know what 80131506 meant?) – Spendable

They are listed in the SDK header file CorError.h – Paul

How did you know they were listed in CorError.h?? – Degenerate

Use this Err.exe tool microsoft.com/en-au/download/details.aspx?id=985 to work out what hex error codes like 80131506 mean and which header file contains them. – Cecelia

@HansPassant I think the question intended was 'of all the files that exist in the world, how did you know that CorError.h was a worthwhile file to look at'? – Perspiration

I don't really understand a comment like that, they are all worth a look and the SDK is a very tiny subset of "all files that exist in the world". It is like visiting your local library, why would you intentionally not take a look at a book? Maybe the cover is unattractive, but it doesn't take very long to see the pattern behind the way they named the files. – Paul

Answer 2

A bug in the concurrent implementation of the Garbage Collection on x64 .Net 4 can cause this as stated in the following microsoft KB entry:

ExecutionEngineException occurs during Garbage Collection

You should first make a deep minidump exploration to be sure that the problem occured during a Garbage collection.

The minidump location can usually be found in a Windows Error Reporting entry in the event log following the crash entry. Then, have fun with WinDbg !

The latest documentation on the use of the <gcConcurrent/> configuration element, to disable concurrent or (in .NET 4 and later) background garbage collection, can be found here.

Rohn

thanks for this comment - this was the solution for a problem I've had for a long long time! – Aerophagia

You're a life saver, this was the issue for us. As an aside, you can also open the minidump file in Visual Studio, set up the symbol paths if you need to, and then debug. This told us that the error happens at clr.dll!WKS::gc_heap::mark_object_simple(). I'm sure WinDbg is very powerful but using VS can tell you enough if you're just verifying the source of the error. – Bolitho

The application crashed but I didn't find any mini dumps in C:\Temp\CrashDump folder. There're some other crash dumps there, and we can find the dumps from the crashes days ago. Do you know why there's no crash dumps? The error message and exit code are exactly the same. – Monocoque

This is exactly what I was looking for... the app crash event contained an instruction pointer, which was useless to me without a dump. Never thought to look for later events. Thank you! – Dignity

For others in the same situation, it may be useful to configure Windows Error Reporting to do a full heap dump on crash: msdn.microsoft.com/en-us/library/windows/desktop/… – Dignity

Ok, I've got the same error. After that opened dump with WinDbg, last line in callstack was: clr!JIT_ChkCastClassSpecial+0x10. Any ideas? – Aviatrix

Answer 3

I've experienced "internal errors" in the .NET runtime that turned out to be caused by bugs in my code; don't think that just because it was an "internal error" in the .NET runtime that there isn't a bug in your code as the root cause. Always always always blame your own code before you blame someone else's.

Hopefully you have logging and exception/stack trace information to point you where to start looking, or that you can repeat the state of the system before the crash.

Squeegee

Those bugs in your code, were they in purely managed code as well, or did they always involve unmanaged code? – Deface

Answer 4

For those arriving here from google, I've eventually come across this SO question, and this specific answer solved my problem. I've contacted Microsoft for the hotfix through the live chat on support.microsoft.com and they sent me a link to the hotfix by email.

Titus

Answer 5

Had the same exact error on WinXP box with latest build of my .NET 4 code. Checked previous builds - now they crash too! Ok, so it's not me :). No suggestions here/above helped.

Much more recent (2018-05-09) report of the same problem: Application Crash with exit code 80131506.

A: We were receiving a similar error, but we believe ours was caused by the Citrix memory optimizer.
The resolution was to force a regeneration of the .Net core libraries on the host(s) where the issue was occurring:
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ngen.exe update /force

Root cause is still unknown (machine is not being updated and has little use), but that did it for me!

Confidential

Answer 6

After years of wrestling with this issue in a number of applications, it appears that Microsoft has finally accepted it as a bug in .NET 4 CLR that causes this to occur. http://support.microsoft.com/kb/2640103.

I had previously been "fixing" it by forcing the garbage collector to run in server mode (gcServer enabled="true" in app.config) as described in the Microsoft article linked to by Think Before Coding. This in essence forces all threads in the application to pause during the collection removing the possibility of other threads accessing the memory being manipulated by the GC. I am happy to find that my years of searching in vain for a "bug" in my code or other 3rd party unmanaged libraries were only fruitless because the bug lay in Microsoft's code, not mine.

Reader

What is the version number of the HotFix files you received? The version number listed in the KB is 4.0.30319.526 but I already have 4.0.30319.18052. Is the HotFix still needed or has it been rolled into a Windows Update? – Chlorinate

When I run the HotFix exe I get "KB2640103 does not apply, or is blocked by another condition on your computer." – Chlorinate

Answer 7

Could be a bug with concurrent GC http://support.microsoft.com/kb/2679415

Multipurpose

Answer 8

In my case this exception was occured when disk space was over and .NET can't allocate memory in Windows Virtual Memory.

In event log I saw this error:

Application popup: Windows - Virtual Memory Minimum Too Low : Your system is low on virtual memory. Windows is increasing the size of your virtual memory paging file. During this process, memory requests for some applications may be denied.

And previous error:

The C: disk is at or near capacity. You may need to delete some files.

Decorticate

Answer 9

Every 5-10 minutes my application pool kept crashing with this exit code. I do not want to ruin your trust of the Garbage Collector, but the following solution worked for me.

I added a Job that calls GC.GetTotalMemory(true) every minute.

I suppose that, for some reason, the GC is not automatically inspecting the memory often enough for the high number of disposable objects that I use.

Lipcombe

Answer 10

In my case the problem was a C++/CLI library in which there was a call to the NtQuerySystemInformation; for some kind of reason sometimes (and under mysterious circumstances), when it was called the CLR heap got corrupted and the application crashed.

I've resolved the problem using a "custom heap" created with HeapCreate and allocating there the buffers used by that function.

Jumna

Answer 11

This might be an exception occurring in the finalizer. If you are doing the Pattern of ~Class(){ Dispose(false); } check what are you disposing as an un-managed resource. Just put a try..catch there and you should be fine.

We found the issue as we had this mysterious failure with no logs We did the usual recommended pattern of using a "void Dispose(bool disposing)".

Looking at the answers on this question about the finalizer we found a possible place where the Disposal of the unmanaged resources could throw an exception.

It turns out somewhere we did not dispose the object properly thus the finalizer took over the diposal of unmanaged resources thus behold an exception occurred.

In this case was using the Kafka Rest API to clean up the client from Kafka. Seems that it did threw exception at some point then this issue occurred.

Payday

Answer 12

I am not sure it may help everyone, but I could get around this by running

devenv.exe /ResetSettings 

...in the path {Visual_Studio_root}\Common7\Ide

I had the following errors in the event log and VS was just crashing and restarting all the time:

Faulting application name: devenv.exe, version: 14.0.25123.0, time stamp: 0x56f22f32
Faulting module name: clr.dll, version: 4.7.2115.0, time stamp: 0x59af88f2
Exception code: 0xc0000005
Fault offset: 0x0015f90e
Faulting process id: 0x3a7c
Faulting application start time: 0x01d353463eaf0c36
Faulting application path: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\devenv.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Report Id: a232f984-6e80-4f61-9003-e18a035c8f93
Faulting package full name: 
Faulting package-relative application ID: 

Marshamarshal

This worked for me too. Context: I had converted a medium-sized solution (~25 projects) to the .NET Core SDK, fronted by an almost-empty Web Application Project that replaced the old WAP before the conversion. Apparently some lingering settings were clashing with the expectations of IISExpress in the new project. – Carrero

Answer 13

In my case the issue was due to duplicate binding redirects in my web.config. More info here.

I assume it was because of NuGet modifying the binding redirects, but for example it was looking like this:

  <dependentAssembly>
    <assemblyIdentity name="Lucene.Net" publicKeyToken="85089178b9ac3181"/>
    <bindingRedirect oldVersion="0.0.0.0-2.9.4.0" newVersion="3.0.3.0"/>
  </dependentAssembly>
  <dependentAssembly>
    <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed"/>
    <bindingRedirect oldVersion="0.0.0.0-11.0.0.0" newVersion="11.0.0.0"/>
  </dependentAssembly>
  <dependentAssembly>
    <assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
    <bindingRedirect oldVersion="0.0.0.0-4.2.0.0" newVersion="4.0.0.0"/>
  </dependentAssembly>
  <dependentAssembly>
    <assemblyIdentity name="Lucene.Net" publicKeyToken="85089178b9ac3181"/>
    <bindingRedirect oldVersion="0.0.0.0-2.9.4.0" newVersion="3.0.3.0"/>
  </dependentAssembly>
  <dependentAssembly>
    <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed"/>
    <bindingRedirect oldVersion="0.0.0.0-11.0.0.0" newVersion="11.0.0.0"/>
  </dependentAssembly>
  <dependentAssembly>
    <assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
    <bindingRedirect oldVersion="0.0.0.0-4.2.0.0" newVersion="4.0.0.0"/>
  </dependentAssembly>

Removing all the duplicates solved the problem.

Roe

Answer 14

In my case the problem was related to "Referencing .NET standard library in classic asp.net projects" and these two issues

https://github.com/dotnet/standard/issues/873

https://github.com/App-vNext/Polly/issues/628

and downgrading to Polly v6 was enough to workaround it

Conflagrant

Answer 15

In my case this error occured when logging in SAP Business One 9.1 application. In Windows events I could find also another error event in addition to the one reported by the OP:

Nome dell'applicazione che ha generato l'errore: SAP Business One.exe, versione: 9.10.160.0, timestamp: 0x551ad316
Nome del modulo che ha generato l'errore: clr.dll, versione: 4.0.30319.34014, timestamp: 0x52e0b784
Codice eccezione: 0xc0000005
Offset errore 0x00029f55
ID processo che ha generato l'errore: 0x1d7c
Ora di avvio dell'applicazione che ha generato l'errore: 0x01d0e6f4fa626e78
Percorso dell'applicazione che ha generato l'errore: C:\Program Files (x86)\SAP\SAP Business One\SAP Business One.exe
Percorso del modulo che ha generato l'errore: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
ID segnalazione: 3fd8e0e7-52e8-11e5-827f-74d435a9d02c
Nome completo pacchetto che ha generato l'errore: 
ID applicazione relativo al pacchetto che ha generato l'errore: 

The machine run Windows 8.1, with .NET Framework 4.0 installed and without the 4.5 version. As it seemed from the internet that it could be also a bug in .NET 4, I tried installing .NET Framework 4.5.2 and I solved the issue.

Nilsson

Answer 16

Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.Reflection.TargetInvocationException

I have faced this Error, Application was working fine on some PCs and on some PCs giving the above error. I uninstall the Framework 4.5 and re-install this resolved my problem.

Cheer.

Ify

Answer 17

I never figured out why this was happening for me. It was consistently reproducible for one of my applications, but went away after simply rebooting.

I am running Windows 2004 Build 19582.1001 (Insider Preview) with .net-4.8 and I also would not be surprised if this were due to something like a hardware memory error. Also, my application does load some unmanaged code and initialize it, so I can’t prove that the crash didn’t come from that.

Triumvirate

Answer 18

I had same issue on Windows 2016 while installing SQL server 2017. Error was

"ScenarionEngine.exe Crashes With "Internal Error In The .NET Runtime"

First check the Microsoft .NET Framework version if it is 4.5 then upgrade is to latest version. Currently it is Microsoft .NET Framework 4.8. If this not work for you then try the next step.

Next Step: Please uninstall KB4486129, and then download the Microsoft .NET Framework 4.8. Same error has been resolved for me.

Chancy