XslCompiledTransform and custom XmlUrlResolver: "An entry with the same key already exists"

Question

Is there a way to debug XSLT documents that are loaded from a database by a custom XmlUrlResolver or does anyone know, what the errormessage below is about?

I have a XSLT stylesheet that imports a common xslt document:

<xsl:import href="db://common.hist.org"/>

The Scheme is handled by a custom XmlResolver that loads the XSLT document from a DB, but I get an error:

An entry with the same key already exists.

The common XSLT document referred to by the xsl:import contains some common XSLT templates, each with a unique name.

This error began to occur after having moved the XSLT documents from the local file system to the database. When using default import schemes pointing to local files and when loading the XSLT documents from the local filesystem, the error does not occur.

I also tried to turn on debugging when creating the instance of the XslCompiledTransform, but somehow it is not possible to "step into" the database-based XSLT.

_xslHtmlOutput = new XslCompiledTransform(XSLT_DEBUG);

---

Update: The following is basically the resolver code as requested, but the exception is not happening inside my code; thus I guess no obvious reason in this code below. (This same code is actually used to load the XSLT stylesheets that contain the imports, and when commenting out the imports everything works as expected.)

public class XmlDBResolver : XmlUrlResolver
{
    private IDictionary<string,string> GetUriComponents(String uri)
    {
        bool useXmlPre = false;
        uri = uri.Replace("db://", "");
        useXmlPre = uri.StartsWith("xml/");
        uri = uri.Replace("xml/", "");
        IDictionary<string, string> dict = new Dictionary<string, string>();
        string app = null, area = null, subArea = null;

        if (!String.IsNullOrWhiteSpace(uri))
        {
            string[] components = uri.Split('.');

            if (components == null)
                throw new Exception("Invalid Xslt URI");

            switch (components.Count())
            {
                case 3:
                    app = components[0];
                    break;
                case 4:
                    area = components[0];
                    app = components[1];
                    break;
                case 5:
                    subArea = components[0];
                    area = components[1];
                    app = components[2];
                    break;
                default:
                    throw new Exception("Invalid Xslt URI");
            }

            dict.Add("application", app);
            dict.Add("area", area);
            dict.Add("subArea", subArea);
            dict.Add("xmlPreTransform", String.Format("{0}", useXmlPre));
        }

        return dict;
    }

    public override System.Net.ICredentials Credentials
    {
        set { /* TODO: check if we need credentials */ }
    }

    public override object GetEntity(Uri absoluteUri, string role, Type ofObjectToReturn)
    {
        /*
         *  db://<app>.hist.org
         *  db://<area>.<app>.hist.org
         *  db://<subArea>.<area>.<app>.hist.org
         * 
         * */

        Tracing.TraceHelper.WriteLine(String.Format("GetEntity {0}", absoluteUri));

        XmlReader reader = null;

        switch (absoluteUri.Scheme)
        {
            case "db":
                string origString = absoluteUri.OriginalString;
                IDictionary<string, string> xsltDict = GetUriComponents(origString);

                if(String.IsNullOrWhiteSpace(xsltDict["area"]))
                {
                    reader = DatabaseServiceFactory.DatabaseService.GetApplicationXslt(xsltDict["application"]);
                }
                else if (!String.IsNullOrWhiteSpace(xsltDict["area"]) && String.IsNullOrWhiteSpace(xsltDict["subArea"]) && !Boolean.Parse(xsltDict["xmlPreTransform"]))
                {
                    reader = DatabaseServiceFactory.DatabaseService.GetAreaXslt(xsltDict["application"], xsltDict["area"]);
                }
                else if (!String.IsNullOrWhiteSpace(xsltDict["area"]) && !String.IsNullOrWhiteSpace(xsltDict["subArea"]))
                {
                    if(Boolean.Parse(xsltDict["xmlPreTransform"]))
                        reader = DatabaseServiceFactory.DatabaseService.GetSubareaXmlPreTransformXslt(xsltDict["application"], xsltDict["area"], xsltDict["subArea"]);
                    else
                        reader = DatabaseServiceFactory.DatabaseService.GetSubareaXslt(xsltDict["application"], xsltDict["area"], xsltDict["subArea"]);
                }
                return reader;

            default:
                return base.GetEntity(absoluteUri, role, ofObjectToReturn);
        }
    }

and for completeness the IDatabaseService interface (relevant parts):

public interface IDatabaseService
{
    ...
    XmlReader GetApplicationXslt(String applicationName);
    XmlReader GetAreaXslt(String applicationName, String areaName);
    XmlReader GetSubareaXslt(String applicationName, String areaName, String subAreaName);
    XmlReader GetSubareaXmlPreTransformXslt(String applicationName, String areaName, String subAreaName);
}

---

Update: I tried to isolate the problem by temporarily loading the stylesheets from a web server instead, which works. I learned that the SQL Server apparently stores only XML fragments without the XML declaration, in contrast to the stylesheets being stored on a webserver.

Update: The stacktrace of the Exception:

System.Xml.Xsl.XslLoadException: XSLT-Kompilierungsfehler. Fehler bei (9,1616). ---> System.ArgumentException: An entry with the same key already exists.. bei System.Collections.Specialized.ListDictionary.Add(Object key, Object value) bei System.Collections.Specialized.HybridDictionary.Add(Object key, Object value) bei System.Xml.Xsl.Xslt.XsltLoader.LoadStylesheet(XmlReader reader, Boolean include) bei System.Xml.Xsl.Xslt.XsltLoader.LoadStylesheet(Uri uri, Boolean include) bei System.Xml.Xsl.Xslt.XsltLoader.LoadStylesheet(XmlReader reader, Boolean include) --- Ende der inneren Ablaufverfolgung des Ausnahmestacks --- bei System.Xml.Xsl.Xslt.XsltLoader.LoadStylesheet(XmlReader reader, Boolean include) bei System.Xml.Xsl.Xslt.XsltLoader.Load(XmlReader reader) bei System.Xml.Xsl.Xslt.XsltLoader.Load(Compiler compiler, Object stylesheet, XmlResolver xmlResolver) bei System.Xml.Xsl.Xslt.Compiler.Compile(Object stylesheet, XmlResolver xmlResolver, QilExpression& qil) bei System.Xml.Xsl.XslCompiledTransform.LoadInternal(Object stylesheet, XsltSettings settings, XmlResolver stylesheetResolver) bei System.Xml.Xsl.XslCompiledTransform.Load(String stylesheetUri, XsltSettings settings, XmlResolver stylesheetResolver) bei (my namespace and class).GetXslTransform(Boolean preTransform) bei (my namespace and class).get_XslHtmlOutput() bei (my namespace and class).get_DisplayMarkup()

Andradite

i believe the problem is that i return a XmlReader from the DB-Service, that is shot if the underlying connection closes. I currently try to create a IXPathNavigable from the XmlReader inside the DB-Service and return a XPathDocument instead. – Andradite

the problem persists using a XPathDocument instead of XmlReader – Andradite

The error happens within the GetUriComponents(String uri) but you haven't shown the code of this method. To verify this, try filling manually the desired dictionary items -- and not calling this method. It is likely you won't get that error message in this case. – Cystotomy

@DimitreNovatchev I assure you, the error is not within that method. The only calls to IDictionary.Add are done manually with fixed (and distinct) keys. – Andradite

Please, face the facts: The error message you get is issued by the Add() method of the Dictionary class. And in our profession it is usual for things that are "impossible to happen" to actually happen. – Cystotomy

Probably you are right. I can provide the code, but i guess in this case, the "impossible to happen" would require the IL to execute a part of a block twice without having the actual instruction to do so, since the Dictionary is created, filled and returned in that function, without the use of any loops or jumps or whatever. Additionally it would require the IL to ignore scopes completely. ps: i added the code – Andradite

and: the error is caused by the Add() of IDictionary, but that IF is used so extensively throughout the whole runtime, you cannot tell where the place is. That is why one part of the problem is to locate the issue, when using XslCompiledTransforms and xsl:import instructions. I have tried to include the MS Symbol References but unfortunately VS is not stepping into anything. So i really cannot tell where precisely the Exception occurs. – Andradite

The MSDN documentation of GetEntity states that you should return a Stream object. However, you are returning an XmlReader. Possibly this results in the specified error somewhere in the .NET framework. – Solute

@Solute thanks for the hint. The example i found stated that the object returned can be either a Stream, XmlReader or XPathNavigable. I tried it with a XPathNavigable but no success. I'll try to return a stream explicitly. – Andradite

Can you provide a stack trace of the exception? – Unison

Answer 1

Short answer:

Your IDatabaseService interface methods return XmlReader objects. When you construct these, make sure to pass a baseUri to the constructor; e.g.:

public XmlReader GetApplicationXslt(string applicationName)
{
    …
    var baseUri = string.Format("db://{0}.hist.org", applicationName);
    return XmlReader.Create(input: …, 
                            settings: …,
                            baseUri: baseUri);  // <-- this one is important!
}

If you specify this parameter, everything just might work fine. See the last section of this answer to see why I am suggesting this.

---

Long answer, introduction: Possible error sources:

Let's first briefly think about what component(s) could cause the error:

"This error began to occur after having moved the XSLT documents from the local file system to the database. When using default import schemes pointing to local files and when loading the XSLT documents from the local filesystem, the error does not occur."

Putting stylesheets in the database means that you must have…

1. changed the import paths in the stylesheets (introduced db://… paths)
2. implemented and hooked up a custom XmlDbResolver for handling the db:// import scheme
3. implemented database access code in the form of IDatabaseService, which backs XmlDbResolver

If the stylesheets are unchanged except for the import paths, it would seem likely that the error is either in your XmlResolver class and/or in IDatabaseService implementation. Since you haven't shown the code for the latter, we cannot debug your code without some guessing.

I have created a mock project using your XmlDbResolver (a full description follows below). I could not reproduce the error, thus I suspect that your IDatabaseService implementation causes the error.

Update: I have been able to reproduce the error. See the OP's comment & the last section of this answer.

---

My attempt to reproduce your error:

I've created a console application project in Visual Studio 2010 (which you can retrieve by cloning this Gist using Git (git clone https://gist.github.com/fbbd5e7319bd6c281c50b4ebb1cee1f9.git) and then checking out the 2nd commit, git checkout d00629). I'll describe each of the solution's items in more detail below.

(Note that the Copy to output directory property of SqlServerDatabase.mdf, TestInput.xml, and of both .xslt project items should be set to Always.)

---

SqlServerDatabase.mdf:

This is a service-based database which I'll be attaching to a local instance of SQL Server Express 2008. (This is done via the connection string in App.config; see below.)

I've set up the following items inside this database:

This table contains two columns which are defined as follows:

The tables are initially empty. Test data will be added to the database at runtime (see Program.cs and CommonHistOrg.xslt below).

---

App.config:

This file contains a connection string entry for the above database.

<?xml version="1.0"?>
<configuration>
  <connectionStrings>
    <add name="SqlServerDatabase"
         connectionString="Data Source=.\SQLEXPRESS;
                           AttachDbFilename=|DataDirectory|\SqlServerDatabase.mdf;
                           Integrated Security=True;
                           User Instance=True"
         />
  </connectionStrings>
</configuration>

---

IDatabaseService.cs:

This file contains the definition for your IDatabaseService interface, which I'm not repeating here.

---

SqlServerDatabaseService.cs:

This contains a class that implements IDatabaseService. It reads/writes data to the above database:

using System;
using System.Collections.Generic;
using System.Configuration;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using System.IO;
using System.Xml;

class SqlServerDatabaseService : IDatabaseService
{
    // creates a connection based on connection string from App.config: 
    SqlConnection CreateConnection()
    {
        return new SqlConnection(connectionString: ConfigurationManager.ConnectionStrings["SqlServerDatabase"].ConnectionString);
    }

    // stores an XML document into the 'ApplicationDocuments' table: 
    public void StoreApplicationDocument(string applicationName, XmlReader document)
    {
        using (var connection = CreateConnection())
        {
            SqlCommand command = connection.CreateCommand();
            command.CommandText = "INSERT INTO ApplicationDocuments (ApplicationName, Document) VALUES (@applicationName, @document)";
            command.Parameters.Add(new SqlParameter("@applicationName", applicationName));
            command.Parameters.Add(new SqlParameter("@document", new SqlXml(document)));
                                                             //  ^^^^^^^^^^^^^^^^^^^^
            connection.Open();
            int numberOfRowsInserted = command.ExecuteNonQuery();
            connection.Close();
        }
    }

    // reads an XML document from the 'ApplicationDocuments' table:
    public XmlReader GetApplicationXslt(string applicationName)
    {
        using (var connection = CreateConnection())
        {
            SqlCommand command = connection.CreateCommand();
            command.CommandText = "SELECT Document FROM ApplicationDocuments WHERE ApplicationName = @applicationName";
            command.Parameters.Add(new SqlParameter("@applicationName", applicationName));

            connection.Open();
            var plainXml = (string)command.ExecuteScalar();
            connection.Close();

            if (plainXml != null)
            {
                return XmlReader.Create(new StringReader(plainXml));
            }
            else
            {
                throw new KeyNotFoundException(message: string.Format("Database does not contain a application document named '{0}'.", applicationName));
            }
        }
    }

    … // (all other methods throw a NotImplementedException)
}

---

XmlDbResolver.cs:

This contains the XmlDbResolver class, which is identical to your XmlDBResolver class except for two changes:

1. The public constructor accepts an IDatabaseService object. This is used instead of DatabaseServiceFactory.DatabaseService.

2. I've had to remove the call to Tracing.TraceHelper.WriteLine.

---

CommonHistOrg.xslt:

This is the db://common.hist.org stylesheet, which will be put into the database at runtime (see Program.cs below):

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:template match="Foo">
    <Bar/>
  </xsl:template>
</xsl:stylesheet>

---

TestStylesheet.xml:

This is a stylesheet which references the above db://common.hist.org stylesheet:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:import href="db://common.hist.org"/>
</xsl:stylesheet>

---

TestInput.xml:

This is the XML test input document that we're going to transform using the above TestStylesheet.xslt:

<?xml version="1.0" encoding="utf-8" ?>
<Foo/>

---

Program.cs:

This contains the test application code:

using System;
using System.Text;
using System.Xml;
using System.Xml.Xsl;

class Program
{
    static void Main(string[] args)
    {
        var databaseService = new SqlServerDatabaseService();

        // put CommonHistOrg.xslt into the 'ApplicationDocuments' database table:
        databaseService.StoreApplicationDocument(
            applicationName: "common",
            document:        XmlReader.Create("CommonHistOrg.xslt"));

        // load the XSLT stylesheet:
        var xslt = new XslCompiledTransform();
        xslt.Load(@"TestStylesheet.xslt", 
            settings: XsltSettings.Default, 
            stylesheetResolver: new XmlDbResolver(databaseService));

        // load the XML test input:
        var input = XmlReader.Create("TestInput.xml");

        // transform the test input and store the result in 'output':
        var output = new StringBuilder();
        xslt.Transform(input, XmlWriter.Create(output));

        // display the transformed output:
        Console.WriteLine(output.ToString());
        Console.ReadLine();
    }
}

Works like a charm on my machine: The output is an XML document with an empty root element <Bar/>, which is what the db://common.hist.org stylesheet outputs for the matched <Foo/> element from the test input.

---

Update: Error reproduction & fix:

1. Insert the following statement in the Main method:

   databaseService.StoreApplicationDocument(
       applicationName: "test",
       document:        XmlReader.Create("TestStylesheet.xslt"));
   

2. Instead of

   xslt.Load(@"TestStylesheet.xslt", …);
   

   do

   xslt.Load(@"db://test.hist.org", …);
   

   This triggers the error reported by the OP.

After some debugging, I have found out that the following does not cause this problem.

• The fact that the Document column in the database table has type XML. It fails with NTEXT, too.

• The fact that the <?xml … ?> header is missing from the documents that are returned from the DB. The error persists even when the XML header is manually added back before SqlServerDatabaseService returns control to the framework.

In fact, the error is triggered somewhere in the .NET Framework code. Which is why I decided to download and install the .NET Framework reference source. (I changed the solution to use version 3.5 of the framework for debugging purposes.) Installing this and restarting VS then allows you to see and step through the framework code during a debugging session.

Starting at the call to xslt.Load(…;) in our Main method, I stepped into the framework code and eventually came to a method LoadStylesheet inside XsltLoader.cs. There's a HybridDictionary called documentUrisInUse, which apparently stores base URIs of already-loaded stylesheets. So if we load more than one stylesheet with an empty or missing base URI, this method will try to add null to that dictionary twice; and this is what causes the error.

So once you assign a unique base URI to each stylesheet returned by your IDatabaseService, everything should work fine. You do this by passing a baseUri to the XmlReader constructor. See a code example at the very beginning of my answer. You can also retrieve an updated, working solution by downloading or cloning this Gist (git clone https://gist.github.com/fbbd5e7319bd6c281c50b4ebb1cee1f9.git).

Balkin

You can reproduce the error by also storing the TestStylesheet.xslt into the DB (ApplicationName=test) and then specify the URI db://test.hist.org as the first parameter for the Load method of the XslCompiledTransform. Is it possible that you cannot Load the complete document hierarchy using Resolvers? Is there a maximum depth of imports when using XmlResolvers in conjunction with SqlServer? I'll try to do the loading of the "TopLevel"-Documents seperately from DB and post the results. – Andradite

I can reproduce the error now. Sprinkling some Debug.WriteLines over XmlDbResolver.GetEntity shows that both stylesheets (db://test.hist.org specified in Program.cs, and db://common.hist.org in the db://test.hist.org document) have been fully loaded from the database. The error happens right afterward, somewhere inside the .NET Framework. I'm now downloading the .NET reference source code & symbols to see whether I can step through the framework code with the debugger to see what causes the error. – Balkin

@ovm: I think I've found a solution. See my updated answer ("short answer" at the very beginning; then a more detailed explanation at the end of my answer, which also includes a download link for a working solution). – Balkin

Thats it! Answer accepted and bounty rewarded! Now it makes sense that the error doesn't occur when loading the documents via http, since the baseUri can be derived from the original location, which is not possible with the custom URI-scheme. Thank you – Andradite

Thanks! This has been a very interesting challenge, and I've learnt a lot of new things along the way. – Balkin