Slow SoapHttpClientProtocol constructor
Asked Answered
H

6

30

I'm doing some experiments with Microsoft Dynamics CRM. You interact with it through web services and I have added a Web Reference to my project. The web service interface is very rich, and the generated "Reference.cs" is some 90k loc.

I'm using the web reference in a console application. I often change something, recompile and run. Compilation is fast, but newing up the web service reference is very slow, taking some 15-20 seconds: CrmService service = new CrmService(); Profiling reveals that all time is spent in the SoapHttpClientProtocol constructor.

The culprit is apparently the fact that the XML serialization code (not included in the 90k loc mentioned above) is generated at run time, before being JIT'ed. This happens during the constructor call. The wait is rather frustrating when playing around and trying things out.

I've tried various combinations of sgen.exe, ngen and XGenPlus (which takes several hours and generates 500MB of additional code) but to no avail. I've considered implementing a Windows service that have few CrmService instances ready to dish out when needed but that seems excessive.

Any ideas?

Hying answered 5/10, 2008 at 14:52 Comment(3)
Re-tagged. Question has very little to do with the specifics of the author-specific CRM integration problem that he was using to discover the issue, and much more to do with the start-up performance of xml serialization.Photophobia
Agreed, although the extremely large web service API may be somewhat unique to mscrm and that tag could attract people who had solved the same problem with that platform.Hying
Same problem with SalesForce API WSDL mine is 41kloc's and climbingSwashbuckler
A
39

The following is ripped from this thread on the VMWare forums:

Hi folks,

We've found that sgen.exe does work. It'just that there is a couple of additional steps beyond pre-generating the serializer dll's that we missed in this thread. Here is the detailed instruction

PROBLEM

When using the VIM 2.0 SDK from .NET requires long time to instantiate the VimService class. (The VimService class is the proxy class generated by running 'wsdl.exe vim.wsdl vimService.wsdl')

In other words, the following line of code:

_service = new VimService();

Could take about 50 seconds to execute.

CAUSE

Apparently, the .NET XmlSerializer uses the System.Xml.Serialization.* attributes annotating the proxy classes to generate serialization code in run time. When the proxy classes are many and large, as is the code in VimService.cs, the generation of the serialization code can take a long time.

SOLUTION

This is a known problem with how the Microsoft .NET serializer works.

Here are some references that MSDN provides about solving this problem:

http://msdn2.microsoft.com/en-us/library/bk3w6240.aspx http://msdn2.microsoft.com/en-us/library/system.xml.serialization.xmlserializerassemblyattribute.aspx

Unfortunately, none of the above references describe the complete solution to the problem. Instead they focus on how to pre-generate the XML serialization code.

The complete fix involves the following steps:

  1. Create an assembly (a DLL) with the pre-generated XML serializer code

  2. Remove all references to System.Xml.Serialization.* attributes from the proxy code (i.e. from the VimService.cs file)

  3. Annotate the main proxy class with the XmlSerializerAssemblyAttribute to point it to where the XML serializer assembly is.

Skipping step 2 leads to only 20% improvement in the instantiation time for the VimService class. Skipping either step 1 or 3 leads to incorrect code. With all three steps 98% improvement is achieved.

Here are step-by-step instructions:

Before you begin, makes sure you are using .NET verison 2.0 tools. This solution will not work with version 1.1 of .NET because the sgen tool and the XmlSerializationAssemblyAttribute are only available in version 2.0 of .NET

  1. Generate the VimService.cs file from the WSDL, using wsdl.exe:

    wsdl.exe vim.wsdl vimService.wsdl

    This will output the VimService.cs file in the current directory

  2. Compile VimService.cs into a library

    csc /t:library /out:VimService.dll VimService.cs

  3. Use the sgen tool to pre-generate and compile the XML serializers:

    sgen /p VimService.dll

    This will output the VimService.XmlSerializers.dll in the current directory

  4. Go back to the VimService.cs file and remove all System.Xml.Serialization.* attributes. Because the code code is large, the best way to achieve that is by using some regular expression substitution tool. Be careful as you do this because not all attributes appear on a line by themselves. Some are in-lined as part of a method declaration.

    If you find this step difficult, here is a simplified way of doing it:

    Assuming you are writing C#, do a global replace on the following string:

    [System.Xml.Serialization.XmlIncludeAttribute

    and replace it with:

    // [System.Xml.Serialization.XmlIncludeAttribute

    This will get rid of the Xml.Serialization attributes that are the biggest culprits for the slowdown by commenting them out. If you are using some other .NET language, just modify the replaced string to be prefix-commented according to the syntax of that language. This simplified approach will get you most of the speedup that you can get. Removing the rest of the Xml.Serialization attributes only achieves an extra 0.2 sec speedup.

  5. Add the following attribute to the VimService class in VimService.cs:

    [System.Xml.Serialization.XmlSerializerAssemblyAttribute(AssemblyName = "VimService.XmlSerializers")]

    You should end up with something like this:

    // ... Some code here ... [System.Xml.Serialization.XmlSerializerAssemblyAttribute(AssemblyName = "VimService.XmlSerializers")] public partial class VimService : System.Web.Services.Protocols.SoapHttpClientProtocol { // ... More code here

  6. Regenerate VimSerice.dll library by

    csc /t:library /out:VimService.dll VimService.cs

  7. Now, from your application, you can add a reference to VimSerice.dll library.

  8. Run your application and verify that VimService object instanciation time is reduced.

ADDITIONAL NOTES

The sgen tool is a bit of a black box and its behavior varies depending on what you have in your Machine.config file. For example, by default it is supposed to ouptut optimized non-debug code, but that is not always the case. To get some visibility into the tool, use the /k flag in step 3, which will cause it to keep all its temporary generated files, including the source files and command line option files it generated.

Even after the above fix the time it takes to instantiate the VimService class for the first time is not instantaneous (1.5 sec). Based on empirical observation, it appears that the majority of the remaining time is due to processing the SoapDocumentMethodAttribute attributes. At this point it is unclear how this time can be reduced. The pre-generated XmlSerializer assembly does not account for the SOAP-related attributes, so these attributes need to remain in the code. The good news is that only the first instantiation of the VimService class for that app takes long. So if the extra 1.5 seconds are a problem, one could try to do a dummy instantiation of this class at the beginning of the application as a means to improve user experience of login time.

Aho answered 8/6, 2009 at 16:48 Comment(4)
I was having similar problems consuming a 3rd party webserver, it was taking 10-15 seconds to instantiate the proxy code wsdl.exe generated. The steps above cut it down to 2 seconds.Greengrocer
I ran in to the exact same issue this week, where it was taking a while to initially instantiate a web service client proxy. However, there is a major catch here. When you comment out the XmlIncludeAttribute attributes for types used by the web service, the client will no longer know how to serialize/deserialize these types. Therefore this option won't work when you need to pass or retrieve custom types to and from the client. The Microsoft Dynamics CRM service is a perfect example of where this won't work, as it contains a boatload of custom types.Dewan
Unconstructive but grateful comment: the proxy instantiation time fell down from 4mn to 4s. So thanks. A lot.Vapor
I used Visual Studio for regular expression search and replace. Here are the patterns I used. Search: (?<attr>\[System.Xml.Serialization.[^\]]*\]) Replace: /*${attr}*/Dicarlo
P
1

You might wish to look into the Sgen.exe tool that comes with .NET. There's also a handy little thing in Visual Studio's C# project properties "Build" page, at the very bottom, called "Build serialization assembly" that automatically runs Sgen for you.

Photophobia answered 5/10, 2008 at 15:3 Comment(3)
It doesn't help, constructor is still slow. I can confirm that the "*.XmlSerializers.dll" gets build. It's the same for both Debug and Release modes. Does one have to reference the XmlSerializers.dll assembly?Hying
Odd. It should just automatically load up. Is the .XmlSerializers.dll file in the same directory as the executable, and if so, are you changing the current working directory or anything?Photophobia
Correction, automatically using sgen as Alex descripes consistently brings down constructor time from about 15 sec to about 7 sec. It adds a similar delay to the actual compilation step (which makes sense) so for edit-compile-run it's not a big boon. The remaining 7 sec are due to JIT'ing I assume.Hying
R
1

I believe that this is not an SGEN issue. I have looked at the constructor code, and I see that it is doing a lot of reflection (based on the XmlIncludeAttribute on the class). It reflects on all of them, and can take a really long time.

Rim answered 18/11, 2008 at 4:51 Comment(0)
R
1

There is a pre-generated XmlSerializer assembly that comes with CRM. Check to see whether you have SdkTypeProxy.XmlSerializers.dll and SdkProxy.XmlSerializers.dll in the GAC.

If you don't then that means that when you create the CrmService, .net will generate the XmlSerializer assembly which can take some time. Hope this helps

Russian answered 17/12, 2008 at 22:7 Comment(0)
I
0

I came across this thread when trying to find out why my initial SoapHttpClientProtocol calls were taking so long.

I found that setting the Proxy to null/Empty stopped the Proxy AutoDetect from occurring - This was taking up to 7 seconds on the initial call:

this.Proxy = GlobalProxySelection.GetEmptyWebProxy();
Incumbency answered 7/12, 2012 at 1:13 Comment(0)
B
0

I have used above detailed answer as guide, and went a few steps forward, making a script to automate process. Script is made out of two files :

generateproxy.bat :

REM if your path for wsdl, csc or sgen is missing, please add it here (it varies from machine to machine)
set PATH=%PATH%;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools;C:\Program Files (x86)\MSBuild\14.0\Bin

wsdl http://localhost:57237/VIM_WS.asmx?wsdl REM create source code out of WSDL
PowerShell.exe -ExecutionPolicy Bypass -Command "& '%~dpn0.ps1'" REM proces source code (remove annotations, add other annotation, put class into namespace)
csc /t:library /out:references\VIM_Service.dll VIM_WS.cs REM compile source into dll
sgen /p references\VIM_Service.dll /force REM generate serializtion dll

generateproxy.ps1

(Get-Content VIM.cs) | 
    ForEach-Object { 
        $_ -replace "(?<attr>\[global::System.Xml.Serialization.[^\]]*\])", "/*${attr}*/" `
            -replace "public partial class VIM", "[System.Xml.Serialization.XmlSerializerAssemblyAttribute(AssemblyName = ""VIM_Service.XmlSerializers"")] `npublic partial class VIM" `
            -replace "using System;", "namespace Classes.WS_VIM {   `n`nusing System;"
    } |
Set-Content VIM.cs
Add-Content VIM.cs "`n}"

I have added those two files to client project, and in the pre-build event I have added lines

cd..\..
generateproxy

So, before every build, proxy classes are regenerated, and developer has (almost) no need to think about it. While building, WS must be up and running, and its URL must be in bat file. As a result of prebuild, two dll files will regenerate in client project's subfolder references. After first execution of scripts, you should add reference to new dll.

Bamako answered 8/9, 2016 at 21:8 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.