Attempting to upload to FTP: System.Net.WebException: System error

Question

I have an API that takes in XML and ultimately uploads files based on information in the XML. The uploads are on a schedule (also from XML), and I have tested everything surrounding it and know it works.

I am getting an error about 40% of the time on the first file that I attempt to upload in each time cycle (time cycle = 45 minutes for some files, 30 minutes for others).

Here is my code for the upload:

try {
    LoggerFTP.Log("Uploading file: " + filename, false);

    // Create the request.
    FtpWebRequest request = (FtpWebRequest)WebRequest.Create(appSettingsFTP.ftpUrl + @"/" + filename);
    request.Method = WebRequestMethods.Ftp.UploadFile;
    request.Timeout = 6000000; //set to 100 minutes
    //request.Timeout = -1; //set to infinite

    // Add the login credentials.
    request.Credentials = new NetworkCredential(appSettingsFTP.ftpLogin, appSettingsFTP.ftpPassword);

    // Grab the file contents.
    StreamReader sourceStream = new StreamReader(appSettingsFTP.uploadFileDirectory + filename);
    byte[] fileContents = Encoding.UTF8.GetBytes(sourceStream.ReadToEnd());
    sourceStream.Close();
    request.ContentLength = fileContents.Length;

    // Copy the file contents to the outgoing stream.
    Stream requestStream = request.GetRequestStream();
    requestStream.Write(fileContents, 0, fileContents.Length);
    requestStream.Close();

    FtpWebResponse response = (FtpWebResponse)request.GetResponse();
    //Logger.Log(filename.ToString() + " " + "Upload Complete, Status: " + response.StatusCode + " " + response.StatusDescription, false);
    //Took response.StatusDescription out because it appears to be creating extra line feeds.
    LoggerFTP.Log(filename.ToString() + " " + "Upload Complete, Status: " + response.StatusCode, false);
}
catch (Exception ex) {
    LoggerFTP.Log(ex.ToString(), false);
}

I have researched the issue and saw something online about it potentially being a speed thing. Like, there is a timeout. But I have my timeout set to 100 minutes for my FtpWebRequest, so it can't possibly be that? I don't know. This is also running as a service so it is hard to test this aspect of the code.

Here is the exception that is getting logged in my logger (e.ToString):

System.Net.WebException: System error. ---> System.Net.InternalException: System error.
   at System.Net.PooledStream.PrePush(Object expectedOwner)
   at System.Net.ConnectionPool.PutConnection(PooledStream pooledStream, Object owningObject, Int32 creationTimeout, Boolean canReuse)
   at System.Net.FtpWebRequest.FinishRequestStage(RequestStage stage)
   at System.Net.FtpWebRequest.SyncRequestCallback(Object obj)
   at System.Net.FtpWebRequest.RequestCallback(Object obj)
   at System.Net.CommandStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.IO.Stream.Dispose()
   at System.Net.ConnectionPool.Destroy(PooledStream pooledStream)
   at System.Net.ConnectionPool.PutConnection(PooledStream pooledStream, Object owningObject, Int32 creationTimeout, Boolean canReuse)
   at System.Net.FtpWebRequest.AttemptedRecovery(Exception e)
   at System.Net.FtpWebRequest.SubmitRequest(Boolean async)
   --- End of inner exception stack trace ---
   at System.Net.FtpWebRequest.GetRequestStream()
   at CPMainSpringAPIExportsSC.UploadFTP.FTPUploadMethod(String viewname, String filename)

Hancock

I'm having a similar error, did you ever find a solution? – Roxyroy

I actually don't know what I did. I look through my log file and realized that the error still happens occasionally, but more infrequently. Do you have it working some times or most of the time? The fact that it happens doesn't bother me because I upload another one after 15 minutes and it almost never fails twice in a row. – Hancock

I've got a service that fires an elapsed event every 30 seconds to check for new uploads. Typically it will go through, but occasionally I will get this error. The weird thing is, any time it fails, it will fail 3 times consecutively and go through on the 4th elapsed event. – Roxyroy

Yeah, mine does weird stuff, too, sometimes. But like I said, it happens too infrequently for me to go back and fix it. Already deployed and customer is happy =). Good luck with yours though, man. Wish I knew what to say to help. – Hancock

Do you want me to Bounty this for you? I can spare 50 rep and see if someone answers it. – Hancock

Nah, not at this point at least. I think I can get around it by manually forcing a retry when the exception is thrown. – Roxyroy

Please don't hesitate to let me know if you change your mind. I really don't mind at all. Good luck, man. – Hancock

I am encountering this issue quite frequently and it seems to depend on the comms infrastructure. We have a embedded device that may be connected via 3G or WiFi and we seldom see this on WiFi but quite often on 3G, and in some locations we get this all the time even though the internet works fine with other apps, like a browser. I would be happy to donate 100 bounty to find out the cause and figure out what to do about it. – Druid

Are you calling this code on multiple threads simultaneously? I used to get this error very frequently, but when I dedicated a single thread to image file sending, it went down to something like once every 5 million transfers. – Fescue

Answer 1

I am getting exactly the same stack trace in an SSIS package attempting to FTP over SSL. It works great without SSL, but as soon as I enable SSL, it blows up.

    System.Net.WebException: System error. --->
    System.Net.InternalException: System error.    at
    System.Net.PooledStream.PrePush(Object expectedOwner)    at
    System.Net.ConnectionPool.PutConnection(PooledStream pooledStream, Object owningObject, Int32 creationTimeout, Boolean canReuse)    at
    System.Net.FtpWebRequest.FinishRequestStage(RequestStage stage)    at
    System.Net.FtpWebRequest.SyncRequestCallback(Object obj)    at
    System.IO.Stream.Close()    at
    System.Net.ConnectionPool.Destroy(PooledStream pooledStream)    at
    System.Net.ConnectionPool.PutConnection(PooledStream pooledStream, Object owningObject, Int32 creationTimeout, Boolean canReuse)    at
    System.Net.FtpWebRequest.AttemptedRecovery(Exception e)    at
    System.Net.FtpWebRequest.SubmitRequest(Boolean async)
    --- End of inner exception stack trace ---    at
    System.Net.FtpWebRequest.CheckError()    at
    System.Net.FtpWebRequest.GetRequestStream()    at
    ST_0ff7348de65a468bb358ab0206e3721f.ScriptMain.Main() in c:\Users\Stephens\AppData\Local\Temp\Vsta\e664c8a71bb647ff9e9dc6ac32d7b615\ScriptMain.cs:line 155 at 
    System.Net.FtpWebRequest.CheckError()    at
    System.Net.FtpWebRequest.GetRequestStream()    at
    ST_0ff7348de65a468bb358ab0206e3721f.ScriptMain.Main() in c:\Users\Stephens\AppData\Local\Temp\Vsta\e664c8a71bb647ff9e9dc6ac32d7b615\ScriptMain.cs:line 155

Because the error is so generic, I decided to look at the .NET source to see if I can catch a clue about what is breaking. If you go here:

http://labs.developerfusion.co.uk/SourceViewer/browse.aspx?assembly=SSCLI&namespace=System.Net#{%22pageClientState%22%3A%22type-2844%2Ccsharp%22}

and skip down to line 281, you will see the definition for internal void PrePush(object expectedOwner) which is what is executing when the exception happens. Here is what it looks like:

    internal void PrePush(object expectedOwner)
    {
        lock (this) {
            //3 // The following tests are retail assertions of things we can't allow to happen.
            if (null == expectedOwner) {
                if (null != m_Owner && null != m_Owner.Target)
                    throw new InternalException();
                // new unpooled object has an owner
            }
            else {
                if (null == m_Owner || m_Owner.Target != expectedOwner)
                    throw new InternalException();
                // unpooled object has incorrect owner
            }

            m_PooledCount++;

            if (1 != m_PooledCount)
                throw new InternalException();
            // pushing object onto stack a second time
            if (null != m_Owner)
                m_Owner.Target = null;
        }
    }

Eventually I discovered that FtpWebRequest only supports explicit FTP (port 21) and not implicit FTP (port 990). This was definitively stated here:

Does .NET FtpWebRequest Support both Implicit (FTPS) and explicit (FTPES)?

Anyway, in my case, it was a firewall issue. Originally we configured for implicit FTP, which was ports 989, 990, 49152-65535 (per the vendor's tech staff). I checked with my network guy and we opened up ports 20, 21, 989, 990 and 40000-655535 for explicit and things worked like a champ afterwards.

However, in your case, you appear to have some connection pool excitement going on. There is a good post on this subject here:

How to improve the Performance of FtpWebRequest?

You might want to take a look at mucking around with your connection pool set up and see if you can make some progress. Hope this helps!

Regards,

Stuart

Cavalla

Answer 2

I know this is an old topic, but for anyone with the same problem, the code that is missing is:

            request.EnableSsl = false;

Hexylresorcinol