Intermittent errors while de-serializing object from XML

Question

I have a program that takes objects stored as XML in a database (basicly a message queue) and de-serializes them. Intermittently, I will get one of the following errors:

System.Runtime.InteropServices.ExternalException: Cannot execute a program. The command being executed was "C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\csc.exe" /noconfig /fullpaths @"C:\Documents and Settings\useraccount\Local Settings\Temp\lh21vp3m.cmdline".
   at System.CodeDom.Compiler.Executor.ExecWaitWithCaptureUnimpersonated(SafeUserTokenHandle userToken, String cmd, String currentDir, TempFileCollection tempFiles, String& outputName, String& errorName, String trueCmdLine)
   at System.CodeDom.Compiler.Executor.ExecWaitWithCapture(SafeUserTokenHandle userToken, String cmd, String currentDir, TempFileCollection tempFiles, String& outputName, String& errorName, String trueCmdLine)
   at Microsoft.CSharp.CSharpCodeGenerator.Compile(CompilerParameters options, String compilerDirectory, String compilerExe, String arguments, String& outputFile, Int32& nativeReturnValue, String trueArgs)
   at Microsoft.CSharp.CSharpCodeGenerator.FromFileBatch(CompilerParameters options, String[] fileNames)
   at Microsoft.CSharp.CSharpCodeGenerator.FromSourceBatch(CompilerParameters options, String[] sources)
   at Microsoft.CSharp.CSharpCodeGenerator.System.CodeDom.Compiler.ICodeCompiler.CompileAssemblyFromSourceBatch(CompilerParameters options, String[] sources)
   at System.CodeDom.Compiler.CodeDomProvider.CompileAssemblyFromSource(CompilerParameters options, String[] sources)
   at System.Xml.Serialization.Compiler.Compile(Assembly parent, String ns, XmlSerializerCompilerParameters xmlParameters, Evidence evidence)
   at System.Xml.Serialization.TempAssembly.GenerateAssembly(XmlMapping[] xmlMappings, Type[] types, String defaultNamespace, Evidence evidence, XmlSerializerCompilerParameters parameters, Assembly assembly, Hashtable assemblies)
   at System.Xml.Serialization.TempAssembly..ctor(XmlMapping[] xmlMappings, Type[] types, String defaultNamespace, String location, Evidence evidence)
   at System.Xml.Serialization.XmlSerializer.GenerateTempAssembly(XmlMapping xmlMapping, Type type, String defaultNamespace)
   at System.Xml.Serialization.XmlSerializer..ctor(Type type, String defaultNamespace)
   at System.Xml.Serialization.XmlSerializer..ctor(Type type)
   .....

Or I'll get this one:

System.InvalidOperationException: Unable to generate a temporary class (result=1).
error CS0016: Could not write to output file 'c:\Documents and Settings\useraccount\Local Settings\Temp\nciktsd7.dll' -- 'Could not execute CVTRES.EXE.'

   at System.Xml.Serialization.Compiler.Compile(Assembly parent, String ns, XmlSerializerCompilerParameters xmlParameters, Evidence evidence)
   at System.Xml.Serialization.TempAssembly.GenerateAssembly(XmlMapping[] xmlMappings, Type[] types, String defaultNamespace, Evidence evidence, XmlSerializerCompilerParameters parameters, Assembly assembly, Hashtable assemblies)
   at System.Xml.Serialization.TempAssembly..ctor(XmlMapping[] xmlMappings, Type[] types, String defaultNamespace, String location, Evidence evidence)
   at System.Xml.Serialization.XmlSerializer.GenerateTempAssembly(XmlMapping xmlMapping, Type type, String defaultNamespace)
   at System.Xml.Serialization.XmlSerializer..ctor(Type type, String defaultNamespace)
   at System.Xml.Serialization.XmlSerializer..ctor(Type type)
   ....

The program process thousands of messages a day successfully, but I only get these errors maybe 2 or 3 times a day. They don't appear to be correlated to any specific kind of message, just completely random.

Any idea what causes those errors and how to fix it?

ETA - Here's the code that is causing the errors, in case that helps:

public class MessageContextBuilder<T> where T : MessageContextBase 
{
    private static IDictionary<string, XmlSerializer> SerializerCache { get; set; }
    public ILog Logger { get; set; }


    public MessageContextBuilder() {
        if (SerializerCache == null) SerializerCache = new Dictionary<string, XmlSerializer>();
        Logger = LogContextManager.Context.GetLogger<MessageContextBuilder<T>>();
    }

    public T BuildContextFromMessage(IEmailQueueMessage msg) {
        XmlSerializer serializer = GetSerializer(typeof(T));
        XmlReader r = XmlReader.Create(new StringReader(msg.MessageDetails));
        if (serializer.CanDeserialize(r)) {
            T rval = (T)serializer.Deserialize(r);
            rval.EmailAddress = msg.EmailAddress;
            rval.LocaleID = msg.LocaleID;
            rval.StoreID = msg.StoreID;
            rval.MessageID = msg.UniqueKey;
            return rval;
        } else {
            throw new ArgumentException("Cannot deserialize XML in message details for message #" + msg.UniqueKey);
        }
    }

    public XmlSerializer GetSerializer(Type t) {
        if (!SerializerCache.ContainsKey(t.FullName)) {
            SerializerCache.Add(t.FullName, new XmlSerializer(t)); // Error occurs here, in XmlSerializer constructor, intermittently
        }
        return SerializerCache[t.FullName];
    }
}

Brazilin

Answer 1

You can pre-create serializers: http://msdn.microsoft.com/en-us/library/bk3w6240%28v=VS.100%29.aspx Just give it a try. The next canonical candidate for such problems is your virus scanner. Your tool is writing to disc while creating serializers. I have seen virus scanner producing all kind of strange errors in such situations.

Profant

+1, Some sites have noted the CVTRES.EXE error with virus scanner like tools running. – Interminable

+1 as well - I think you are on to something with the virus scanner idea. As for pre-creating the serializers, that's a good option as well, although it could get a bit complicated for the particular app I'm working with. – Brazilin

Disabling my virus scanner worked for me. XmlSerializer constructor never returned when AVG Internet Security was enabled. – Trellis

Answer 2

XmlSerializer is supposed to be thread safe.

Even if that's the case, you can notice the behavior you are getting is in both cases failing at: XmlSerializer..ctor(Type type)

Given that, it seriously look like a multi-threading limitation trying to create serializers.

I suggest to take this code you have:

public XmlSerializer GetSerializer(Type t) {
        if (!SerializerCache.ContainsKey(t.FullName)) {
            SerializerCache.Add(t.FullName, new XmlSerializer(t)); // Error occurs here, intermittently
        }
        return SerializerCache[t.FullName];
    }

And implement a lock on the Add. This way you are only creating 1 serializer at a time. The hit is small if you aren't processing tons of different types.

Note that you need the lock anyway, as the way it is you could get duplicate exceptions when 2 types try to be added at the same time.

static object serializerCacheLock = new object();
public XmlSerializer GetSerializer(Type t) {
        if (!SerializerCache.ContainsKey(t.FullName))
        lock(serializerCacheLock)
        if (!SerializerCache.ContainsKey(t.FullName)) {
            SerializerCache.Add(t.FullName, new XmlSerializer(t));
        }
        return SerializerCache[t.FullName];
    }

If the above still isn't enough, I'd try with a read/write lock on serializer constructor vs. serializers usage. Line of thought being that maybe the multi-threading issue is worth than 2 ctors running at the same time.

All above is a Huge guess, but if it were me I'd definitely confirm is not that.

Cervantez

A good point, but the application is not multi-threaded, so this should not be an issue in my case. – Brazilin

@Eric so it isn't a web service or something like that? Is a client side program going through a log of messages being processed, which is done 1 at a time? – Cervantez

that is correct. It's a command line app that runs as a scheduled task and processes records from a "queue" in a database table one at a time. – Brazilin

@Eric how about 2 scheduled activations happening at the same time? / shouldn't result in that problem, just wondering. Anyway, the solution in my answer wouldn't work in that scenario & I definitely expect something like that not to cause the issue. But it is a weird error, so I wouldn't rule out anything without checking. – Cervantez

Nope, that shouldn't occur either as the task is set not to run if for some reason it is already running. Regardless, that would be a separate process so it shouldn't interfere as you mentioned. – Brazilin

Answer 3

For the first error (cannot execute a program), you might be running into the same XmlSerializer bug that we ran into. It turns out XmlSerlializer throws that exception when Directory.CurrentDirectory is set to a folder that no longer exists.

Our specific situation is different than yours, but I'll give the details in case it helps shed light on what might be happening for you, or it helps anyone else. In our case, a small number of our customers would get that error after launching our WinForms application directly from the installer, i.e. they chose the "run now" option after installing or upgrading. (Unclear why it happened to some but not others). What we suspect is happening is that our installer (InstallAware) occasionally starts our application with the current directory set to a folder that no longer exists, or is about to be deleted. To test this theory, I wrote a test app which simulates launching from the installer:

    string dir = @"C:\Users\me\Documents\Temp\WillBeDeleted";
    Directory.CreateDirectory(dir);
    Directory.SetCurrentDirectory(dir);

    Process.Start(@"C:\Program Files (x86)\...\our.exe");

    Directory.SetCurrentDirectory(@"C:\");  // otherwise, won't be able to delete
    Directory.Delete(dir);

Sure enough, as soon as the launched application created a new instance of XmlSerializer, the exception would be thrown. I put in trace statements to show the result of GetCurrentDirectory(), and indeed it was set to the WillBeDeleted folder. The fix was to SetCurrentDirectory to a valid location during application initialization, before any serialization took place.

Protolithic

Answer 4

This is a sign that you are not caching your serialisers which is not good at all => it leads to memory leak and I suspect you will experience this.

Remember that .NET generates code and compiles them into assemblies every time you create a serialiser.

Always create your serialisers and then cache them.

Here is a sample:

public class SerialiserCache
{

    private static readonly SerialiserCache _current = new SerialiserCache();
    private Dictionary<Type, XmlSerializer> _cache = new Dictionary<Type, XmlSerializer>();

    private SerialiserCache()
    {

    }

    public static SerialiserCache Current
    {
        get { return _current; }
    }

    public XmlSerializer this[Type t]
    {
        get
        {
            LoadIfNecessary(t);
            return _cache[t];
        }
    }

    private void LoadIfNecessary(Type t)
    {

        // double if to prevent race conditions 
        if (!_cache.ContainsKey(t))
        {
            lock (_cache)
            {
                if (!_cache.ContainsKey(t))
                {
                    _cache[t] = new XmlSerializer(typeof(T));
                }
            }
        }
    }

}

Suavity

A good tip, and I'll look at that, but they way this program is set up, it probably doesn't matter. Basicly it runs as a job every few minutes, processes a few messages, then exits. So caching the serializers would be of minimal benefit. – Brazilin

It probably will if you are creating them many times during the same ruun. – Suavity

According to this question, the .NET framework caches the serializers internally (I'm not using any XmlAttributeOverrides). Is that not actually the case? #679176 – Brazilin

So I implemented serializer caching as you suggested, but I'm still getting these same errors. – Brazilin