Transfer large amount of data in WCF service

Question

I have created a web service in WCF which returns more than 54000 data-rows with 10 data in each row. I have used the wsHttpBinding for communication. The service works well with less data (i.e. 2000 rows) but it bombs out when attempting to send large record set with 50000+ rows(~2MB). The exception message is like this

An error occurred while receiving the HTTP response to http://localhost:9002/MyService.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details.

Please don't tell me to use pagination on the client side - I know that it will solve the problem. But I need the whole chunk of data in the client-end.

My service configuration on server is as

<system.serviceModel>
  <bindings>
    <wsHttpBinding>
      <binding name="MyWsHttpBinding" />
    </wsHttpBinding>
  </bindings>
  <services>
    <service name="AdminService">
      <endpoint address="AdminSrv"
                binding="wsHttpBinding"
                contract="IAdminService"/>
      <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
      <host>
        <baseAddresses>
          <add baseAddress="/Bus/IRfotoWCF" />
        </baseAddresses>
      </host>
    </service>
  </services>
  <behaviors>
    <serviceBehaviors>
      <behavior>
        <!-- To avoid disclosing metadata information, 
                  set the value below to false and remove the metadata endpoint above before deployment -->
        <serviceMetadata httpGetEnabled="True"/>
        <!-- To receive exception details in faults for debugging purposes, 
                  set the value below to true.  Set to false before deployment 
                  to avoid disclosing exception information -->
        <serviceDebug includeExceptionDetailInFaults="True" />
      </behavior>
    </serviceBehaviors>
  </behaviors>
  <serviceHostingEnvironment multipleSiteBindingsEnabled="true"></serviceHostingEnvironment>
</system.serviceModel>

My client configuration is as

<system.serviceModel>
  <bindings>
    <basicHttpBinding>
      <binding name="BasicHttpBinding_IAdminService" closeTimeout="00:01:00"
               openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
               allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
               maxBufferSize="65536" maxBufferPoolSize="524288" maxReceivedMessageSize="65536"
               messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered" 
               useDefaultWebProxy="true">
        <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384" 
                      maxBytesPerRead="4096" maxNameTableCharCount="16384" />
        <security mode="None">
          <transport clientCredentialType="None" proxyCredentialType="None" realm="" />
          <message clientCredentialType="UserName" algorithmSuite="Default" />
        </security>
      </binding>
    </basicHttpBinding>
  </bindings>
  <client>
    <endpoint address="http://localhost/TestService/AdminService.svc" 
              binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_IAdminService"
              contract="IAdminService" name="BasicHttpBinding_IAdminService" />
  </client>
</system.serviceModel>

Would someone help me out with excact configuration both in the client and server side. Even if i need to change binding from wsHttpBinding to netTcpBinding - i have no problem doing that. Thanks in advance.

Microsecond

Answer 1

After a lot of investigation finnally i got the solution. Actually a number of things need to be changed.

The following changes needed to be done in Server-side.

First I had to set a maxRequestLength to a larger value in my httpRuntime element to run the request for longer period.

<system.web>    
<httpRuntime maxRequestLength="102400" />
</system.web>

Second i introduced netTcpBinding binnding with custom changes on maxBufferSize, maxBufferPoolSize, maxReceivedMessageSize with a large value of 2147483647.

<binding name="myNetTcpBinding" 
maxBufferPoolSize="2147483647" 
maxBufferSize="524288" 
maxReceivedMessageSize="2147483647">

Third add maxItemsInObjectGraph in both of the serviceBehaviors and endpointBehaviors like bellow (dont forget to mention the behaviour names in the service and endpoint node)

    <behaviors>
      <serviceBehaviors>        
        <behavior name="myNetTcpBehaviour">
          <serviceMetadata httpGetEnabled="true"/>
          <serviceDebug includeExceptionDetailInFaults="true"/>
          <dataContractSerializer maxItemsInObjectGraph="2147483647"/>
        </behavior>
      </serviceBehaviors>
      <endpointBehaviors>
        <behavior name="myNetTcpEndPointBehaviour">
          <dataContractSerializer maxItemsInObjectGraph="2147483647"/>
        </behavior>
      </endpointBehaviors>
    </behaviors>

Finally my server configuration looks like this

<system.web>    
    <httpRuntime maxRequestLength="102400" />
</system.web>


  <system.serviceModel>
    <bindings>
      <wsHttpBinding>
        <binding name="MyWsHttpBinding" />
      </wsHttpBinding>
      <netTcpBinding>
        <binding name="myNetTcpBinding"
                 closeTimeout="00:01:00"
                 openTimeout="00:01:00"
                 receiveTimeout="00:10:00"
                 sendTimeout="00:01:00"
                 transactionFlow="false"
                 transferMode="Buffered"
                 transactionProtocol="OleTransactions"
                 hostNameComparisonMode="StrongWildcard"
                 listenBacklog="10"
                 maxBufferPoolSize="2147483647"
                 maxBufferSize="524288"
                 maxConnections="10"
                 maxReceivedMessageSize="2147483647">
          <readerQuotas maxDepth="32"
                        maxStringContentLength="8192"
                        maxArrayLength="16384"
                        maxBytesPerRead="4096"
                        maxNameTableCharCount="16384" />
          <reliableSession ordered="true"
                           inactivityTimeout="00:10:00"
                           enabled="false" />
          <security mode="Transport">
            <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />
          </security>
        </binding>
      </netTcpBinding>
    </bindings>
    <services>
      <service name="AdminService" behaviorConfiguration="myNetTcpBehaviour">
        <endpoint address="AdminSrv" 
                  binding="netTcpBinding" 
                  bindingConfiguration="myNetTcpBinding"
                  contract="IAdminService"
                  behaviorConfiguration="myNetTcpEndPointBehaviour"/>

        <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
        <host>
          <baseAddresses>
            <add baseAddress="/Bus/IRfotoWCF" />
          </baseAddresses>
        </host>
      </service>
    <behaviors>
      <serviceBehaviors>        
        <behavior name="myNetTcpBehaviour">
          <serviceMetadata httpGetEnabled="true"/>
          <serviceDebug includeExceptionDetailInFaults="true"/>
          <dataContractSerializer maxItemsInObjectGraph="2147483647"/>
        </behavior>
      </serviceBehaviors>
      <endpointBehaviors>
        <behavior name="myNetTcpEndPointBehaviour">
          <dataContractSerializer maxItemsInObjectGraph="2147483647"/>
        </behavior>
      </endpointBehaviors>
    </behaviors>
    <serviceHostingEnvironment multipleSiteBindingsEnabled="true"></serviceHostingEnvironment>
  </system.serviceModel>

Now on the client-side configuratioin you need to change the maxBufferSize="2147483647" maxBufferPoolSize="524288" maxReceivedMessageSize="2147483647"

and also you need to add maxItemsInObjectGraph="2147483647" in endpoint behaviour configuration.

        <endpointBehaviors>
            <behavior name="myEndPointBehavior">
                <dataContractSerializer maxItemsInObjectGraph="2147483647" />
            </behavior>
        </endpointBehaviors>

Now i can transmit 30000 rows within 5.30 min where the query executed for 10 sec so the transmission time is 5.20 min - still a lot.

Feel free to comment and any suggestion for improvement.

Microsecond

I don't understand. Is it a really good solution, when one service waits for another service during 5.20 minutes. I think here is the big question about architecture, but i can't find a solution. – Gig

I continue testing speed with transfer big data. Using "Chunks" method i can sent 300 000(!) rows,saves it to database within 4.34min without any changes in my service's configurations. I'm just separated my data for chunks with 50 rows. – Gig

You should use the Datacontract Serializer instead the XML. This is so far a manual replace job inside the reference.cs. – Foible

Answer 2

If you look at the binding details they do not match entirely on the server and that of the client side. The attributes for maxBufferSize, maxBufferPoolSize, maxReceivedMessageSize are to be defined in the server side as well. And then you need to put the values according to the size you are looking at.

Mckale

I have tried these maxBufferSize, maxBufferPoolSize, maxReceivedMessageSize with values 2147483647 both in server and client side. Still the same exception. – Microsecond

I could not see the settings in the config file in your post. Are you sure that the binding configuration is set properly? – Mckale

Yap, I configured those in the binding configuration in my local both on server and client side. But now i have solved the problem-see the answer. Thanks anyway, plz feel free to comment on the answer. – Microsecond

Good that it helped you in reaching the solution. – Mckale

Answer 3

Instead of using for loop over WCF for bulky data , Use user defined table type ( if you are using SQL) . It will reduce the time from 6 min to 15-20 sec.

Huskey