Over the past few months my process for determining the cause for crashes has been the same: open the dump, load SOS, load SOSEX and run !pe. This has always had the same result:

0:019> !pe
PDB symbol for mscorwks.dll not loaded
Exception object: 029e87f4
Exception type: System.ArgumentNullException
Message: Value cannot be null.
InnerException:
StackTrace (generated):
    SP       IP       Function

StackTraceString:
HResult: 80004003
There are nested exceptions on this thread. Run with -nested for details

As has the output of !clrstack:

0:019> !clrstack
OS Thread Id: 0x5d0 (19)
ESP       EIP
0f31f340 7c80bef7 [HelperMethodFrame_1OBJ: 0f31f340]
0f31f398 1449167a
0f31f3c4 144915e6
0f31f3f4 10d81b48
0f31f4b4 793e25cf
0f31f4bc 79366b3d
0f31f4d4 793e2734
0f31f4e8 793e26ac
0f31f678 79e7c0e3 [GCFrame: 0f31f678]

This invariably leads to me closing WinDBG and start manually trying to determine what is going wrong without it. The reason? Microsoft needs to rethink documentation around WinDBG, SOS, and MSCORDACWKS. According to them all you need is the correct version of MSCORDACWKS; that will get you as far as the above.

Getting a Memory Dump From Your Customer

After using whichever method you suggest (e.g. ADPlus) they will also need to do this:

1. Navigate to %WINDIR%\Microsoft.NET\Framework\v2.0.50727
   1. Copy SOS.DLL
   2. Copy MSCORDACWKS.DLL
2. If 64bit Navigate to %WINDIR%\Microsoft.NET\Framework64\v2.0.50727
   1. Copy SOS.DLL
   2. Copy MSCORDACWKS.DLL
3. If they you are optionally or additionally using .Net 4.0 they will need to do steps 1 and 2 for \v4.0.30319.

You will need those files individually; and keep them along-side the memory dump.

Setting Up Your Environment

You are going to need a local symbols directory (we have a file share on a server). WinDBG is rather unforgiving when it comes to the MSCORDACWKS search (e.g. having MSCORDACWKS in the same directory as the memory dump doesn’t work); so you have to put their MSCORDACWKS in your symbol store.

Determining where to store MSCORDACWKS

FYI: I have been able to get this DLL from the Microsoft Symbol Servers; so you should probably try this first:

.sympath srv*http://msdl.microsoft.com/download/symbols;cache*D:\symbols
.reload

Otherwise, there is a way to determine this; but using WinDBG to figure out the path is more reliable:

0:019> !sym noisy
noisy mode - symbol prompts on
0:019> .sympath+ D:\symbols
DBGHELP: Symbol Search Path: d:\symbols
DBGHELP: Symbol Search Path: d:\symbols
Symbol search path is: D:\symbols
Expanded Symbol search path is: d:\symbols
0:019> !loadby sos mscorwks
0:019> .cordll -ve -u -l
CLR DLL status: No load attempts
0:019> !threads
PDB symbol for mscorwks.dll not loaded
...
SYMSRV:  d:\symbols\mscordacwks_x86_x86_2.0.50727.1873.dll\4A7D131D59b000\mscordacwks_x86_x86_2.0.50727.1873.dll not found
...

That SYMSRV line is your answer. Create the directory, rename the file and copy the file there.

Loading up SOS.DLL

First we should undo the noisy symbol prompts:

0:019> !sym quiet
quiet mode - symbol prompts on

We can then load up the correct version of SOS:

0:019> !load D:\Dump Files\2010-01-01\x86\v2.0\SOS.dll

And now we can get stack traces:

0:019> !clrstack
OS Thread Id: 0x5d0 (19)
ESP       EIP
0f31f340 7c80bef7 [HelperMethodFrame_1OBJ: 0f31f340] System.Threading.Monitor.Enter(System.Object)
0f31f398 1449167a SourceCode.**
0f31f3c4 144915e6 SourceCode.**
0f31f3f4 10d81b48 K2Worker.**
0f31f4b4 793e25cf System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(System.Object)
0f31f4bc 79366b3d System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0f31f4d4 793e2734 System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(System.Threading._ThreadPoolWaitCallback)
0f31f4e8 793e26ac System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)
0f31f678 79e7c0e3 [GCFrame: 0f31f678]

Good luck and happy debugging!

Update: According to Tess Ferrandez of WinDBG fame the new SOS (called PSSCOR2.DLL) used by the employees at Microsoft will also fix this. Read about it more over at her blog.