First WCF connection made in new AppDomain is very slow

Question

I have a library that I use that uses WCF to call an http service to get settings. Normally the first call takes ~100 milliseconds and subsequent calls takes only a few milliseconds. But I have found that when I create a new AppDomain the first WCF call from that AppDomain takes over 2.5 seconds.

Does anyone have an explanation or fix for why the first creation of a WCF channel in a new AppDomain would take so long?

These are the benchmark results(When running without debugger attached in release in 64bit), notice how in the second set of numbers the first connections takes over 25x longer

Running in initial AppDomain
First Connection: 92.5018 ms
Second Connection: 2.6393 ms

Running in new AppDomain
First Connection: 2457.8653 ms
Second Connection: 4.2627 ms

This isn't a complete example but shows most of how I produced these numbers:

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Running in initial AppDomain");
        new DomainRunner().Run();

        Console.WriteLine();
        Console.WriteLine("Running in new thread and AppDomain");
        DomainRunner.RunInNewAppDomain("test");

        Console.ReadLine();
    }
}

class DomainRunner : MarshalByRefObject
{
    public static void RunInNewAppDomain(string runnerName)
    {
        var newAppDomain = AppDomain.CreateDomain(runnerName);
        var runnerProxy = (DomainRunner)newAppDomain.CreateInstanceAndUnwrap(typeof(DomainRunner).Assembly.FullName, typeof(DomainRunner).FullName);

        runnerProxy.Run();
    }

    public void Run()
    {
        AppServSettings.InitSettingLevel(SettingLevel.Production);
        var test = string.Empty;

        var sw = Stopwatch.StartNew();
        test += AppServSettings.ServiceBaseUrlBatch;
        Console.WriteLine("First Connection: {0}", sw.Elapsed.TotalMilliseconds);

        sw = Stopwatch.StartNew();
        test += AppServSettings.ServiceBaseUrlBatch;
        Console.WriteLine("Second Connection: {0}", sw.Elapsed.TotalMilliseconds);
    }
}

The call to AppServSettings.ServiceBaseUrlBatch is creating a channel to a service and calling a single method. I have used wireshark to watch the call and it only takes a milliseconds to get a response from the service. It creates the channel with the following code:

public static ISettingsChannel GetClient()
{
    EndpointAddress address = new EndpointAddress(SETTINGS_SERVICE_URL);

    BasicHttpBinding binding = new BasicHttpBinding
    {
        MaxReceivedMessageSize = 1024,
        OpenTimeout = TimeSpan.FromSeconds(2),
        SendTimeout = TimeSpan.FromSeconds(5),
        ReceiveTimeout = TimeSpan.FromSeconds(5),
        ReaderQuotas = { MaxStringContentLength = 1024},
        UseDefaultWebProxy = false,
    };

    cf = new ChannelFactory<ISettingsChannel>(binding, address);

    return cf.CreateChannel();
}

From profiling the app it shows that in the first case constructing the channel factory and creating the channel and calling the method takes less than 100 milliseconds

In the new AppDomain constructing the channel factory took 763 milliseconds, 521 milliseconds to create the channel, 1,098 milliseconds to call the method on the interface.

TestSettingsRepoInAppDomain.DomainRunner.Run() 2,660.00 TestSettingsRepoInAppDomain.AppServSettings.get_ServiceBaseUrlBatch() 2,543.47 Tps.Core.Settings.Retriever.GetSetting(string,!!0,!!0,!!0) 2,542.66 Tps.Core.Settings.Retriever.TryGetSetting(string,!!0&) 2,522.03 Tps.Core.Settings.ServiceModel.WcfHelper.GetClient() 1,371.21 Tps.Core.Settings.ServiceModel.IClientChannelExtensions.CallWithRetry(class System.ServiceModel.IClientChannel) 1,098.83

EDIT

After using perfmon with the .NET CLR Loading object I can see that when it loads the second AppDomain it is loading way more classes into memory than it does initially. The first flat line is a pause I put in after the first appdomain, there it has 218 classes loaded. The second AppDomain causes 1,944 total classes to be loaded.

I assume its the loading of all these classes that is taking up all of the time, so now the question is, what classes is it loading and why?

UPDATE

The answer turns out to be because of the fact that only one AppDomain is able to take advantage of the native image system dlls. So the slowness in the second appdomain was it having to rejit all of the System.* dlls used by wcf. The first appdomain could use the pre ngened native versions of those dlls, so it didn't have the same startup cost.

After investigating the LoaderOptimizationAttribute that Petar suggested, that indeed seemed to fix the issue, using either MultiDomain or MultiDomainHost results in the second AppDomain to take the same amount of time as the first time to access stuff over wcf

Here you can see the default option, note how in the second AppDomain none of the assemblies say Native, meaning they all had to be rejitted, which is what was taking all of the time

Here is after adding the LoaderOptimization(LoaderOptimization.MultiDomain) to Main. You can see that everything is loaded into the shared AppDomain

Here is after user LoaderOptimization(LoaderOptimization.MultiDomainHost) to main. You can see that all system dlls are shared, but my own dlls and any not in the GAC are loaded seperately into each AppDomain

So for the service that prompted this question using MultiDomainHost is the answer, because it has fast startup time and I can unload AppDomains to remove the dynamically built assemblies that the service uses

Lore

Answer 1

You can decorate your Main with LoaderOptimization attribute to tell the CLR loader how to load classes.

[LoaderOptimization(LoaderOptimization.MultiDomain)]
MultiDomain - Indicates that the application will probably have many domains that use the same code, and the loader must share maximal internal resources across application domains.

Sheri

Thanks, adding that does indeed reduce the time in the new AppDomain to be 35 ms for the first and 2ms for the second, but would this prevent unloading of any assemblies loaded into the second AppDomain? One reason the app uses AppDomains at all is because it's a long running service that needs to load custom code to perform an action, so I would still need to be able to unload some assemblies so that the size of the service doesn't grow unbounded. – Lore

This setting should not prevent unloading assemblies from second AppDomain. – Dentition

Which app is producing the screen shots? Sysinternals stuff? – Dentition

Yeah Process Explorer – Lore

Answer 2

Do you have an HTTP proxy defined in IE? (maybe an auto configure script). This can be a cause.

Otherwise I would guess it is the time that takes to load all the dlls. Try to deparate the proxy creation from the actull call to the service, to see what's taking the time.

Grenade

I previously checked the proxy settings and I dont think that's it. I also doubt its the loading dlls because if it was then why wouldn't the first wcf attempt see the same delays as from the new AppDomain? One thing I did notice was that while it was creating the new AppDomain the CPU usages was maxed for that core for the 2.5 seconds – Lore

Answer 3

I found the following article that talks about how only the first AppDomain can use native image dlls, so a child appdomain will always be forced to JIT lots of stuff that the initial AppDomain doesn't have to. This could lead to the performancce impact I am seeing, but would it be possible to somehow not get this performance penalty?

If there is a native image for the assembly, only the first AppDomain can use the native image. All other AppDomains will have to JIT-compile the code which can result in a significant CPU cost.

Lore