What to do with "The version of SOS does not match the version of CLR you are debugging" in WinDbg?

Question

I'm having a problem with some of my apps. It's a wcf-based app running under IIS6 in Windows 2003 Server (x86):
In Event Log I get such an error from "W3SVC-WP" source (EventID=2262):

ISAPI 'C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'.

I'm trying figuring out what's going on. I've set up creating dump for Orphan Worker Process as described in this KB. When an deadlock occured a minidump is created.
Then I take this minidump to try to understand what's happened. Here's I'm stuck.

I run WinDbg x86, open my dump and then:

0:037> .loadby sos clr
0:037> .sympath SRV*c:\temp\symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*c:\temp\symbols*http://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: srv*c:\temp\symbols*http://msdl.microsoft.com/download/symbols
0:037> !clrstack
The version of SOS does not match the version of CLR you are debugging.  Please load the matching version of SOS for the version of CLR you are debugging.
CLR Version: 4.0.30319.1
SOS Version: 4.0.30319.235
CLRDLL: C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\mscordacwks.dll:4.0.30319.235 f:8 doesn't match desired version 4.0.30319.01 f:8
CLRDLL: Loaded DLL c:\temp\symbols\mscordacwks_x86_x86_4.0.30319.01.dll\4BA1D9EF66f000\mscordacwks_x86_x86_4.0.30319.01.dll
OS Thread Id: 0x690 (37)
Unable to walk the managed stack. The current thread is likely not a managed thread.
You can run !threads to get a list of managed threads in the process

What to do with this error - "The version of SOS does not match the version of CLR you are debugging" ?

The same error ("The version of SOS does not match the version of CLR you are debugging") I'm getting when I open the minidump in VS2010.

I've read this post - http://tech-thinker.com/Forums/tabid/62/forumid/12/postid/471/scope/posts/Default.aspx, and tried installing KB2518870. It doesn't help.

Formwork

Nice article about SOS/MSCORDACWKS compatibility - jonathan.dickinsons.co.za/blog/2010/08/windbg-stack-fix – Formwork

This helped me: blogs.msdn.com/b/dougste/archive/2009/02/18/… – Anastos

Answer 1

WinDbg will not be able to use the debugging adapter mscordacwks.dll unless it is the same version as the one from the original machine. You can get around this error by copying this DLL from the target machine which generated the dump to your Debugging Tools for Windows directory.

We debug .NET 2.0 applications with WinDbg. We would continually get this same error regarding mscordacwks_x86_x86_2.0.50727.3615.dll. I had to copy this file from the server onto my client and put it in the C:\Program Files\Debugging Tools for Windows (x86)\ folder. WinDbg stopped complaining after that.

If all else fails, you can try debugging with WinDbg on the same server from which you retrieved the crash dump.

Parturition

thanks. BTW it seems that .sympath set up supplied path globally. – Formwork

Answer 2

This is what worked for me:

Download the following DLLs:

• clr.dll
• mscordacwks.dll
• SOS.dll

from this folder on the machine that generated the dump:

C:\Windows\Microsoft.NET\Framework64\v4.0.30319

Run the following command. The path to SOS.DLL should be without quotes, unescaped path delimiters:

.load path to downloaded SOS.DLL

I think a new WinDbg session is required for this to work.

Lamarlamarck

I used .load psscor4.dll and followed your advice to grab clr sos and mscordacwks from target. This fixed it for me. – Rolfston

Where can I obtain the correct versions of those DLLs if I don't have access to the faulty machine? I can locate those DLLs from the correct Windows updates (e.g., this one), and have downloaded the update file (.msu). But when I unzip the .msu file, they're just a bunch of cabs file. I seemed to remember a website with all versions of CLR binaries to download. – Edile

make sure to check the bitness of the dump you're trying to load, the path above is for x64 clr, for an x86 dump take the dlls from C:\Windows\Microsoft.NET\Framework\v4.0.30319 – Substantiate

Answer 3

The core problem usually lies with a mismatching mscordacwks.dll version (mscorwks.dll itself should not be needed if a full dump was taken). In theory, it should be attainable from the symbol server - simply run .cordll -ve -u -l. For more information on mscordacwks.dll see Failed to load data access DLL, 0x80004005” – OR – What is mscordacwks.dll.

Unfortunately, some versions of mscordacwks.dll have not been indexed, meaning the above won't always work. In such cases you could try and get the correct version from the machine on which the dump was taken, as Yocahi and Thomas mentioned (e.g. from C:\Windows\Microsoft.NET\Framework64\v4.0.30319). Once you get it, issue the following command to load it: .cordll -u -ve -lp PathToFolderContainingMscorDAC. Of course, that machine may be inaccessible, or it may have been patched since the time the dump was taken.

Fortunately, there is a way to extract mscorwdacwks.dll from the actual update KB package (it resides in one of the cab files inside the self extracting executable - use a tool such as 7-Zip to extract it). There also exist repositories of .NET updates (courtesy of MS employee Doug Stewart), so you can browse them for the exact build number you require:

• Version history of the CLR 2.0
• Version history of the CLR 4.0

Once you have the correct mscordacwks.dll, the SOS.dll warning could be ignored in most cases, as the most recent SOS.dll version would work most of the time despite the warning. However, in some cases the correct SOS.dll version is needed as well (and as a bonus you get rid of the pesky warnings). Dunken links to a blog post that should be helpful in that regard (basically you need to place the symbol server in the _NT_SYMBOL_PATH environment variable and run !analyze –v without loading SOS.dll first - it will load the correct version itself). If that doesn't work, you could try extracting SOS.dll from one of the update packages as described above. This site may prove easier to use for that purpose, as it specifically indexes SOS.dll versions.

Finally, consider PsscorR2 (for .NET 2.0-3.5) and Psscor4 (for .NET 4.0). Psscor is a superset of SOS.dll which doesn't complain about mismatched versions, so long as you're using the appropriate major version. It should be noted that over time, it hasn't been maintained as well as SOS.dll, so the latter may include enhancements and bug fixes absent from the former. At the time of writing there was no Psscor version for .NET 4.5.

Monogenesis

Answer 4

The version of SOS does not match the version of CLR you are debugging.  Please load the matching version of SOS for the version of CLR you are debugging.
CLR Version: 4.0.30319.1
SOS Version: 4.0.30319.235

This means the target machine which made the dump is running on CLR version 4.0.30319.1.
Your system is running with version 4.0.30319.235.

This is because there was a security update to .Net 4.0 which changed the CLR and SOS files. And some computers may not have this update yet.

See: http://support.microsoft.com/kb/2572078

This may cause some of the lines in the stack to be a bit wrong... You can avoid the error by obtaining the SOS.dll and CLR.dll and mscordacwks.dll and mscorwks.dll of the original version and load those when you load the SOS.
The original files are usualy under : C:\Windows\Microsoft.NET\Framework\v4.0.30319
Depends on the framework version... and then copy them to a specific folder.
Load the correct files like this:

.load C:\CurrectFiles\sos

Note that it's just "sos" and not sos.dll.

Chalcedony

Answer 5

You can automatically load the right SOS.dll. Check out John Robbins' great blog post http://wintellect.com/blogs/jrobbins/automatically-load-the-right-sos-for-the-minidump

You also might to check with .chain what's already loaded. In some cases you have to unload (e.g. .unload sos) the wrongly loaded dlls first.

Winnick

The link is dead. – Joejoeann

Answer 6

In short, do the following:

1. Get the CLR version form the dump
2. Find and download appropriate Microsoft patch
3. Extract the sos.dll and mscordacwks.dll from the patch
4. Use it

---

Below is an example:

1. After loading a crash dump I get the version I need:

>lm vm clr

it gives me

File version:     4.0.30319.18051

2. I google for MS update that contains this version:

sos.dll 4.0.30319.18051

In this case google gives an MS KB page with a download link. I usually download x64 version, because it contains both x86 and x64 dlls, so I have Windows8-RT-KB2833958-x64.msu now.

Note: sometimes it's tricky to get the required patch, but not in this example.

3. Using FAR file manager I extract cabinet archive from this MSU:

Windows8-RT-KB2833958-x64.cab

Note: Sometimes there're several cabinets inside, so you need to check which one contains sos.dll.

Note: Sometimes patches are distributed as .EXE, so you first need to extract MSU or MSP files (I do it with FAR), and then extract cabinets from them.

4. Sometimes files from CABs can be extracted by FAR, but sometimes they have very different structure and I use Expand.exe from WinAIK. WinAIK is 1.7 Gb ISO, but you need only a small part. I use the following BAT file

mkdir Extracted
..\winaik_amd64\servicing\Expand.exe "%1" -F:sos.dll "Extracted"
..\winaik_amd64\servicing\Expand.exe "%1" -F:mscordacwks.dll "Extracted"

This command extracts all versions of specified dlls, each one inside its own dir. Sometimes there're 2 versions of both mscordacwks.dll and sos.dll. I believe this is because of GRD/LDR(QFE) staff. In our example there're 4.0.30319.18051 and 4.0.30319.19079. Check file properties with Windows Explorer.

5. Rename the files appropriately: mscordacwks.dll must be named as mscordacwks_%arch%_%arch%_%version%.dll and placed near the sos.dll

So mscordacwks.dll (4.0.30319.18051) goes to mscordacwks_AMD64_AMD64_4.0.30319.18051.dll

(x86 version rename to mscordacwks_x86_x86_4.0.30319.18051.dll)

sos.dll might stay as is intact, but I rename it to sos.4.0.30319.18051.dll

Do the same for 4.0.30319.19079 version (for possible future needs)

6. Copy these files to 'C:\SOS\' folder which contains a lot of sos.4.x.x.x.dll and mscordacwks_AMD64_AMD64_4.x.x.x.dll

7. Use it with

.load C:\SOS\sos.4.0.30319.18051.dll

Note: Sometimes for .Net 4.5 you need to add additional '0' to mscordacwks version mscordacwks_AMD64_AMD64_4.6.1055.00.dll instead of mscordacwks_AMD64_AMD64_4.6.1055.0.dll. I didn't dig deeper though, because could handle this within a small timeframe.

BTW, WinDbg will say if mscordacwks cannot be found and will specify the version (which will have double '0' at the end).

Aforementioned