AppFabric CreateRoutingClient Error

Question

We have a problem with AppFabric that is causing the below error to occur:

Exception type: ArgumentException 
Exception message: An item with the same key has already been added.
    at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
    at Microsoft.ApplicationServer.Caching.DataCacheFactory.CreateRoutingClient(String cacheName, NamedCacheConfiguration config)
    at Microsoft.ApplicationServer.Caching.DataCacheFactory.CreateNewCacheClient(DataCacheDeploymentMode mode, String cacheName, NamedCacheConfiguration config, IClientChannel channel)
    at Microsoft.ApplicationServer.Caching.DataCacheFactory.GetCache(String cacheName)
    at Microsoft.ApplicationServer.Caching.DataCacheFactory.GetDefaultCache()
    ... our code where we are attempting to retrieve a cache item from the default cache by its key

This error happens infrequently in our test environment (we have not found a scenario to reproduce the issue on demand), but seems to always happen in our production environment just after each deployment. Our deployments are automated and we have verified that the steps to deploy to our various environments are the same.

Our server setup for a given environment is as follows:

• Server1 - Hosts our .net website, and is our AppFabric 1.1 server
• Server2 - Hosts our .net website

These servers are load balanced. AppFabric local caching has been enabled on both applications. Our test and production servers are setup the same. We are aware that it would be better to have AppFabric on a dedicated server, but don't think that would cause/resolve this issue.

We are stumped by this issue as we have not found any mention of it elsewhere online and because the stack trace seems to indicate that it is a problem with AppFabric itself. The exception mentions inserting something, but all we are doing when this happens is trying to get the default cache from the DataCacheFactory so that we can retrieve an item from it. So, what does this error mean and how might we resolve it?

Update

Here is the code I am using to create the DataCacheFactory and pull data from the cache:

private static readonly Lazy<DataCacheFactory> _DATA_CACHE_FACTORY = new Lazy<DataCacheFactory>(() => new DataCacheFactory());
private static readonly Lazy<DataCache> _CACHE = new Lazy<DataCache>(() => _DATA_CACHE_FACTORY.Value.GetDefaultCache());

public object Get(string key)
{
    return _CACHE.Value.Get(key);
}

Iscariot

Can you post the code you use to get the default cache via the DataCacheFactory? Also examples of your code that uses the cache? – Iong

Thanks for looking into this. Sorry for the slow reply. I was waiting for an opportunity to deploy my latest code to see if it fixed the issue. I have updated the question to include my code. – Iscariot

Have you taken a look at the overload for Lazy<T> that passes an additional thread safety type parameter? msdn.microsoft.com/en-us/library/… Seems like ExecutionAndPublication may be the right option – Iong

Answer 1

I am 100% certain that the duplicate key error is generated by bad access into the private _myCache property of the DataCacheFactory. That property is a Hashtable. Repeated calls to Hashtable.Add("mykey","myvalue"); will generate this same expection that your'e seeing.

I ran multiple tests and calling GetCache("default") and GetDefaultCache() back to back doesn't produce the error. It is definitely something odd about how App Fabric is trying to populate that. Here is my code that has never generated this error. I wanted to post for reference incase you can see something that is obviously different from what your code is doing

if (cache == null)
{
    if(factory == null)
        factory = new DataCacheFactory();

    if(string.IsNullOrWhiteSpace(cacheName))
        cacheName = ConfigurationManager.AppSettings["APP_FABRIC_CACHE_NAME"];

    cache = factory.GetCache(cacheName);
    return cache;
}

In the above example, cache and factory are private static versions of their respective types, inside a static class called Cache.

Hope this helps

Iong

This would make sense for the error message, though it does seem odd that AppFabric is modifying the HashTable in a non-thread safe manner. My code is very similar to yours. The differences are thread safety and I do not have a facility to provide an alternate cache name. Given that I am still having the issue, are there any other factors that may produce this error? – Iscariot

none that I'm aware of. You may want to add some lock(mutexObject){//yourcodehere} blocks around the cache construction. It's kind of grasping at straws at this point, but that's one thing that may be worth trying. – Iong

Answer 2

We use a lock around our factory creation and have never had this issue. We are also using AppFab 1.1 for our cluster. I would recommend you do something like this:

lock (cacheLock)
{
    if(cacheFactory == null)
    {
        DataCacheFactoryConfiguration config = new DataCacheFactoryConfiguration(...); 
        cacheFactory = new DataCacheFactory(config);
    }
    return cacheFactory;
}

Troupe