How do I upload large (> 25MB) files to a web service?

Question

I have a web service that takes a byte[] and saves it.

This works fine for "small" files, but once I hit a certain size the web service fails and returns "The request failed with HTTP status 404: Not Found."

From what I've seen this appears to be an IIS setting that limits the size of a file that can be posted (to prevent Denial of Service attacks). I've tried to increase that setting, but I am having trouble determining what setting and where/how one would set it. I am using IIS7 and the webservice is done in .net (asmx).

In the web.config of the web service I have added the following (which seemed to increase the size of file that can be accepted, but not all the way to this setting size)

  <system.web>
     <httpRuntime executionTimeout="999999" maxRequestLength="2097151" />
     ...
  </system.web>

Any suggestions on where (and how) to increase the size of file that the web service would be greatly appreciated.

Eusporangiate

Regarding WCF/FTP/WS-Attachment/DIME: Excellent suggestions and some interesting reading. These are probably the types of long term solutions that I will end up using. However if there was some short term solution for changing the upload size in IIS7 that would be excellent. Thanks. – Eusporangiate

Answer 1

In addition to the httpRuntime/maxRequestLength mentioned in the question, it looks like there is an additional item that can be added to the web service's web.config file to permit large file transfers.

  <system.webServer>
    <security>
      <requestFiltering>
        <requestLimits maxAllowedContentLength="2000000000" />
      </requestFiltering>
    </security>
  </system.webServer>

This appears to enable larger files to be uploaded via web services.

Eusporangiate

Just a note on this: an indication that you're missing this setting is when you get a 404 from the WCF service but there is no error message in WCF tracing (yes very helpful I know - thanks WCF). Glad you found this Chris, you've saved me wasting any more hours on this. – Lampoon

Yup, this "extra" setting wasted numerous hours of my day today. I didn't realise that, with IIS7, you need to quote the biggest filesize in TWO places in your web.config. But, yes, with this maxAllowedContentLength setting in place, I could finally upload 31Mb files. – Sannyasi

@AndyBritcliffe "Thanks WCF". It seems to me in this case the 404 originates from IIS itself, so WCF probably hasn't even heard of the event, and thus had little chance to log anything about it. Rephrase to "Thanks Microsoft" though, and the criticism still stands. :) – Takamatsu

Another vote of thanks, this has been doing my head in for hours. – Anatomize

Answer 2

You should keep in mind that web services aren't primarily designed as file transfer mechanisms. Any purpose-designed file transfer protocol will likely do a better job than a web service. For instance, such protocols are more likely to deal with error recovery, partial uploads, etc.

However, if you're going to use web services for this purpose in .NET, you should use WCF, if at all possible. Among other benefits, WCF handles streaming, and will therefore be a lot more efficient in terms of memory usage. I'm concerned that if you follow the two (accurate) suggestions above, your next result will be "out of memory or resources" exceptions, as the old ASMX technology tries to load your entire 25MB file into memory at once. In fact, it may have several copies in memory at the same time!

Toadstool

Answer 3

If I was stuck having to use web services and needed to support very large files I would look at implementing a system that allows you to upload files in pieces.

Eg.

• ticketId GetTicket(size)
• UploadData(ticketId, byte[] payload) (this can be called as many times as you want)
• FinalizeUpload(ticketId)

This would allow you to chunk up the big uploads, and not hold too much data in memory. The disadvantage is that you are still using a fairly inefficient transport mechanism.

Boggart

This should really be the accepted solution as it's less memory intensive and can even support parallelism. – Fillip

Answer 4

Just to add information to people googling this web.config:

C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\ISAPI

<location path="Copy.asmx"> <!-- Name of you asmx -->
    <system.webServer>
      <security>
        <requestFiltering>
          <requestLimits maxAllowedContentLength="104857600"/> <!-- 100 megs -->
        </requestFiltering>
      </security>
    </system.webServer>
  </location>

This solved our problem after troubleshooting this issue for quite som time.

Misbehavior

Answer 5

If you're set on using Web Services to move around files I would at least consider using WS-Attachment / DIME attachments. The primary problem with sending byte[]'s over web services is that they get put in the SOAP body which is gets encoded as a base 64 string. Encoding files like this grows the size of the file by as much as two thirds in the soap body (ie. a 6 MB file becomes a 9 MB file over the wire).

It's likely that your 25 MB upload is turning into HUGE soap envelopes.

I'd strongly suggest reading this. Which might get you into DIME.

Here's an excerpt.

Microsoft's WSE Toolkit allows large attachments to be sent along with a Web service method using the DIME and WS-Attachments standards. We'll examine these standards and why they are more efficient than sending large amounts of binary data in a Web service call through other common means.

Hope that helps!

Ellissa

Exactly what I was going to suggest. XML Denial of Service filters will buffer the whole SOAP message, and the buffer size is limited. Sending the base-64 encoded binary as an attachment avoids that. DIME will make it easier for Java, PHP, and other clients to use the service as well. – Babble

Sorry, but WSE IS OBSOLETE. DO NOT USE IT UNLESS YOU HAVE NO OTHER CHOICES AT ALL. Sorry to shout, but it seams that not everyone has heard this, or not all have taken the hint that WSE is not supported in Visual Studio 2008 or above. – Toadstool

John makes a great point, WSE IS deprecated and it's road map is pretty bleak compared to WCF. That being said if we're talking about updates to an existing web service that isn't about to be overhauled to WCF (or the .NET v3.0+ framework isn't viable) I'll stick to my guns and recommend DIME/WS-Attachment. – Ellissa

@Tyler: you may want to check my profile here and see if that makes things more clear. The correct word for WSE is not "deprecated", it's "obsolete". It's much closer to being actually unsupported than "deprecated" might suggest. Truly, the correct answer for WSE is "just don't go there". See social.msdn.microsoft.com/Forums/en-US/asmxandxml/thread/…. – Toadstool

This saved me after 2 hours of search, if your file limit is 10 mgb, you need to put way bigger limit on the web service. Thanks – Consonant

Answer 6

maxRequestLength is in KB, not bytes. This should give you a 30 MB limit within a 4-minute timeout window.

<httpRuntime executionTimeout="240" maxRequestLength="30000" />

Having numbers that are too high may be actually preventing your values from being applied. I think I ran into this a few years ago when I thought it was a byte limit (vague memory).

Beaty

I changed to these values, same result. The values I have were taken from this (somewhat related/somewhat unrelated) article: support.microsoft.com/kb/925083 - both value sets seem to have the same issue. – Eusporangiate

What is your ScriptTimeout set to on the receiving page? – Beaty

Answer 7

This doesn't specifically answer you question, but what I've done in the past is use WCF to transfer file names/paths/listings, but then use an FTP library to transfer the file via FTP.

Polyglot

Answer 8

this worked for me:

            <binding name="uploadFilesBasicHttpBinding" maxReceivedMessageSize="2147483647" maxBufferSize="2147483647" maxBufferPoolSize="2147483647" receiveTimeout="00:10:10" sendTimeout="00:10:00" openTimeout="00:10:00" closeTimeout="00:10:00">
                <readerQuotas maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxDepth="2147483647" maxNameTableCharCount="2147483647" maxStringContentLength="2147483647"/>
                <security mode="TransportWithMessageCredential">
                    <message clientCredentialType="UserName"/>
                </security>
            </binding>

Hoplite

bingo! learn.microsoft.com/en-us/dotnet/framework/wcf/feature-details/… – Honolulu