WCF Streaming large data (500MB / 1GB) on a self hosted service

Question

I'm currently experiencing an issue trying to send large data using WCF self hosted service (no IIS). Transferring 500MB using streaming results with my service crashing with System.OutOfMemoryException. Is it possible at all to transfer such amount of data?

Here is my WCF configuration:

<system.serviceModel>
<services>
  <service  name="CIService" behaviorConfiguration="CIBehavior">        
    <host>
      <baseAddresses>
        <add baseAddress="net.tcp://localhost:6547/CIService/CIService.svc" />
      </baseAddresses>
    </host>
    <endpoint binding="netTcpBinding" 
        bindingConfiguration="netTcpBindingConfig" 
        behaviorConfiguration="CIBehavior.EndpointBehavior" 
        contract="CIService.ICreatable" />
    <endpoint address="mex" 
        binding="mexHttpBinding" 
        name="mexTcpBinding" 
        contract="IMetadataExchange" />
  </service>
</services>
<serviceHostingEnvironment multippleSiteBindingEnabled="True" />
<bindings>
  <netTcpBinding>
    <binding name="netTcpBindingConfig" closeTimeout="00:01:00" openTimeout="00:01:00" 
        receiveTimeout="01:00:00" sendTimeout="00:10:00" 
        hostNameComparisonMode="StrongWildcard" listenBacklog="10" maxConnections="10"
        maxBufferSize="2147483647" maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647" 
        transferMode="Streamed">
      <readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647"
        maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
    </binding>
  </netTcpBinding>
</bindings>

<behaviors>
  <serviceBehaviors>
    <behavior name="CIBehavior">
      <serviceMetadata httpGetEnabled="False" />
      <serviceDebug includeExceptionDetailInFaults="true" />
      <serviceThrottling maxConcurrentCalls="200"  maxConcurrentInstances="2147483647" maxConcurrentSessions="100" />
      <dataContractSerializer maxItemsInObjectGraph="2147483647"/>
    </behavior>
  </serviceBehaviors>
  <endpointBehavior>
    <behavior name="CIBehavior.EndpointBehavior">
      <dataContractSerializer maxItemsInObjectGraph="2147483647" />
    </behavior>
  </endpointBehavior>
</behaviors>
</system.serviceModel>

My client configuration:

<system.serviceModel>
<bindings>
  <netTcpBinding>
    <binding name="NetTcpBinding_ICreatable" 
             closeTimeout="00:01:00" openTimeout="00:01:00" 
             receiveTimeout="01:00:00" sendTimeout="00:10:00" 
             transactionFlow="false" 
             transferMode="Streamed" 
             transactionProtocol="OleTransactions" 
             hostNameComparisonMode="StrongWildcard" 
             listenBacklog="10" 
             maxBufferPoolSize="2147483647" 
             maxBufferSize="2147483647" 
             maxConnections="10"
             maxReceivedMessageSize ="2147483647">
      <readerQuotas
        maxDepth="2147483647" 
        maxStringContentLength="2147483647" 
        maxArrayLength="2147483647" 
        maxBytesPerRead="2147483647" 
        maxNameTableCharCount="2147483647" />
      <reliableSession ordered="true" inactivityTimeout="00:10:00" anabled="false" />
    </binding>
  </netTcpBinding>
</bindings>
<client>
  <endpoint name="NetTcpBinding_ICreatable" 
      address="net.tcp://localhost:6547/CIService/CIService.svc" 
      binding="netTcpBinding" 
      bindingConfiguration="NetTcpBinding_ICreatable" 
      behaviorConfiguration="CIBehavior.EndpointBehavior" 
      contract="ICreatable" />
</client>
<behaviors>
  <endpointBehavior>
    <behavior name="CIBehavior.EndpointBehavior">
      <dataContractSerializer maxItemsInObjectGraph="2147483647" />
    </behavior>
  </endpointBehavior>
</behaviors>
</system.serviceModel> 

Hypaesthesia

You will have to chunk the data into smaller pieces. – Portent

Transferring big-sized messages is definitely possible. see this orand.blogspot.fr/2007/12/wcf-message-size-tester.html – Fraunhofer

Answer 1

You dont need maxBufferSize or maxBufferPoolSize set so high these are possibly causing your out of memory exception. The defaults should be fine.

Check out Large Data and Streaming on MSDN, specifically the section Special Security Considerations for Large Data this piece of the text is important

The MaxBufferSize property is required to constrain the memory that WCF buffers. It is important to set this to a safe value (or keep it at the default value) when streaming. For example, suppose your service must receive files up to 4 GB in size and store them on the local disk. Suppose also that your memory is constrained in such a way that you can only buffer 64 KB of data at a time. Then you would set the MaxReceivedMessageSize to 4 GB and MaxBufferSize to 64 KB. Also, in your service implementation, you must ensure that you read only from the incoming stream in 64-KB chunks and do not read the next chunk before the previous one has been written to disk and discarded from memory.

I put together a very simple example of streaming data from a self hosted service to a console client. To keep the post short I only added the server code and part of the client.

The service contract

using System.IO;
using System.ServiceModel;

namespace Service
{
    [ServiceContract]
    public interface IStream
    {
        [OperationContract]
        Stream GetLargeObject();
    }
}

The service implementation

using System;
using System.IO;
using System.ServiceModel;

namespace Service
{
   [ServiceBehavior]
   public class StreamService : IStream
   {
       public Stream GetLargeObject()
       {
           // Add path to a big file, this one is 2.5 gb
           string filePath = Path.Combine(Environment.CurrentDirectory, "C:\\Temp\\BFBC2_PC_Client_R11_795745_Patch.exe");

        try
        {
            FileStream imageFile = File.OpenRead(filePath);
            return imageFile;
        }
        catch (IOException ex)
        {
            Console.WriteLine(String.Format("An exception was thrown while trying to open file {0}", filePath));
            Console.WriteLine("Exception is: ");
            Console.WriteLine(ex.ToString());
            throw;
        }
    }
 }
}

The service main

using System;
using System.ServiceModel;

namespace Service
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                using (var serviceHost = new ServiceHost(typeof(StreamService)))
                {
                    serviceHost.Open();

                    Console.WriteLine("Press Any Key to end");
                    Console.ReadKey();
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.ToString());
            }
        }
    }
}

The service app.config

<?xml version="1.0"?>
<configuration>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/>
  </startup>

  <system.serviceModel>
    <behaviors>
      <serviceBehaviors>
        <behavior name="StreamServiceBehavior">
          <serviceMetadata httpGetEnabled="True" />
        </behavior>
      </serviceBehaviors>
    </behaviors>
    <bindings>
      <netTcpBinding>
        <binding name="NewBinding0" transferMode="Streamed"/>
      </netTcpBinding>
    </bindings>
    <services>
      <service behaviorConfiguration="StreamServiceBehavior" name="Service.StreamService">
        <endpoint address="net.tcp://localhost:9000/streamserver" binding="netTcpBinding"
          bindingConfiguration="NewBinding0" bindingName="" contract="Service.IStream" />
        <endpoint address="mex" binding="mexHttpBinding"
          contract="IMetadataExchange" />
        <host>
          <baseAddresses>
            <add baseAddress="http://localhost:8080/StreamService" />
          </baseAddresses>
        </host>
      </service>
    </services>
  </system.serviceModel>
</configuration>

Launch the service, may need to run under admin account to open the socket. Create a client console application and add a service reference using the url http:// localhost:8080 / StreamService, using Service as the namespace for the generated client.

The client main

using System;
using System.IO;
using Client.Service;

namespace Client
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                using (StreamClient streamClient = new StreamClient())
                {
                    streamClient.Open();

                    using (FileStream fileStream = new FileStream("c:\\temp\\bigfile.exe",FileMode.Create))
                    {
                        streamClient.GetLargeObject().CopyTo(fileStream);    
                    }
                }

                Console.WriteLine("Press any key to end");
                Console.ReadKey();
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex);
            }
        }
    }
}

The generated client config file will need to be modified slightly, increase receiveTimeout and set maxReceivedMessageSize="4294967295"

<system.serviceModel>
    <bindings>
        <netTcpBinding>
            <binding name="NetTcpBinding_IStream" closeTimeout="00:01:00"
                openTimeout="00:01:00" receiveTimeout="00:30:00" sendTimeout="00:01:00"
                transactionFlow="false" transferMode="Streamed" transactionProtocol="OleTransactions"
                hostNameComparisonMode="StrongWildcard" listenBacklog="10"
                maxBufferPoolSize="524288" maxBufferSize="65536" maxConnections="10"
                maxReceivedMessageSize="4294967295">
                <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
                    maxBytesPerRead="4096" maxNameTableCharCount="16384" />
                <reliableSession ordered="true" inactivityTimeout="00:10:00"
                    enabled="false" />
                <security mode="Transport">
                    <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />
                    <message clientCredentialType="Windows" />
                </security>
            </binding>
        </netTcpBinding>
    </bindings>
    <client>
        <endpoint address="net.tcp://localhost:9000/streamserver" binding="netTcpBinding"
            bindingConfiguration="NetTcpBinding_IStream" contract="Service.IStream"
            name="NetTcpBinding_IStream">

        </endpoint>
    </client>
</system.serviceModel>

Launch the service then the client. It will stream a large file without issue.

Solfeggio

Great example, should be the solution! – Leakage