An existing connection was forcibly closed by the remote host

Question

I am working with a commercial application which is throwing a SocketException with the message,

An existing connection was forcibly closed by the remote host

This happens with a socket connection between client and server. The connection is alive and well, and heaps of data is being transferred, but it then becomes disconnected out of nowhere.

Has anybody seen this before? What could the causes be? I can kind of guess a few causes, but also is there any way to add more into this code to work out what the cause could be?

Any comments / ideas are welcome.

... The latest ...

I have some logging from some .NET tracing,

System.Net.Sockets Verbose: 0 : [8188] Socket#30180123::Send() DateTime=2010-04-07T20:49:48.6317500Z

System.Net.Sockets Error: 0 : [8188] Exception in the Socket#30180123::Send - An existing connection was forcibly closed by the remote host DateTime=2010-04-07T20:49:48.6317500Z 

System.Net.Sockets Verbose: 0 : [8188] Exiting Socket#30180123::Send() -> 0#0

Based on other parts of the logging I have seen the fact that it says 0#0 means a packet of 0 bytes length is being sent. But what does that really mean?

One of two possibilities is occurring, and I am not sure which,

1. The connection is being closed, but data is then being written to the socket, thus creating the exception above. The 0#0 simply means that nothing was sent because the socket was already closed.

2. The connection is still open, and a packet of zero bytes is being sent (i.e. the code has a bug) and the 0#0 means that a packet of zero bytes is trying to be sent.

What do you reckon? It might be inconclusive I guess, but perhaps someone else has seen this kind of thing?

Pteropod

Just an update. It seems that wireshark is not going to cut it in this case because of our network setup. But I am hopefully going to try this, blogs.msdn.com/dgorti/archive/2005/09/18/471003.aspx which is tracing using .NET which should produce some log files. I will keep you posted ... – Pteropod

comcast is also known to send "zero" packets spoofed with fake id to mess with p2p traffic--- – Downs

Answer 1

This generally means that the remote side closed the connection (usually by sending a TCP/IP RST packet). If you're working with a third-party application, the likely causes are:

• You are sending malformed data to the application (which could include sending an HTTPS request to an HTTP server)
• The network link between the client and server is going down for some reason
• You have triggered a bug in the third-party application that caused it to crash
• The third-party application has exhausted system resources

It's likely that the first case is what's happening.

You can fire up Wireshark to see exactly what is happening on the wire to narrow down the problem.

Without more specific information, it's unlikely that anyone here can really help you much.

Seesaw

Great. Thanks. The other thing about wireshark. It collects so much data, how would I be able to filter out something like this? If wireshark shows up something I might post it here later on ... – Pteropod

You should be able to filter the Wireshark dump by IP address and port number, at least. After that, it's probably most helpful to look at the end of the stream (where something went wrong) and work backwards until you can spot where things first get messed up. Depending on the complexity of the protocol involved, it can be really easy to almost impossible... – Seesaw

Is there a source for the bulleted items, or did gamer's answer below just copy your list? – Stratigraphy

Please note that "sending malformed data" could mean sending a https request to a http server and probably vice versa. – Variate

In my case, I was getting this exception only when calling an api when running app locally. No problems in dev, qa, or prod environments. The fix? Using http instead of https locally. We think it may be related to a load balancer. But we just updated our dev, qa, and prod web.configs with transforms for https. Problem solved. – Machicolation

@Machicolation can you explain more how did you resolve the issue and what do you mean by transforms for https? have you managed to get it worked with https? – Kondon

@Machicolation Very interesting. We are also experiencing the same problem. A simple internal HTTP GET request to a local HTTPs endpoint is failing due to the "An existing connection was forcibly closed by the remote host". I'm also curious to know about your transforms. – Estremadura

In my case is sent requests from http to https which didn´t work. Setting both on https or http solved my problem – Disproportionate

Answer 2

Using TLS 1.2 solved this error.
You can force your application using TLS 1.2 with this (make sure to execute it before calling your service):

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 

Another solution :
Enable strong cryptography in your local machine or server in order to use TLS1.2 because by default it is disabled so only TLS1.0 is used.
To enable strong cryptography , execute these commande in PowerShell with admin privileges :

Set-ItemProperty -Path 'HKLM:\SOFTWARE\Wow6432Node\Microsoft\.NetFramework\v4.0.30319' -Name 'SchUseStrongCrypto' -Value '1' -Type DWord
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\.NetFramework\v4.0.30319' -Name 'SchUseStrongCrypto' -Value '1' -Type DWord 

You need to reboot your computer for these changes to take effect.

Bove

You can add multiple values together to support multiple protocols. So to support everything from SSL3 to TLS1.2, set SecurityProtocol = (SecurityProtocolType)4080. – Blodgett

Dumb question: I have two servers. One of the servers has this issue, but the other one works fine. I'm trying to determine what's different between these servers. I don't even see the registry key "SchUseStrongCrypto" in either of them. What does it mean when the key doesn't exist? It's going to use a default TLS? – Estremadura

For those who are on Windows and don't have access to the original source code of the program that is causing the issue. You can run the following script, which generates other scripts to fix any potential TLS compatibility issue at the OS level (it's specific to DevOps, but will probably work for other cases too): github.com/microsoft/azure-devops-tls12/blob/main/… (note: I found that I actually had to clone the repository with git clone - simply copying the contents of the file gave me an encoding issue and thus I couldn't run the script). – Towers

Answer 3

This is not a bug in your code. It is coming from .Net's Socket implementation. If you use the overloaded implementation of EndReceive as below you will not get this exception.

    SocketError errorCode;
    int nBytesRec = socket.EndReceive(ar, out errorCode);
    if (errorCode != SocketError.Success)
    {
        nBytesRec = 0;
    }

Wive

This is coming from the operating system, and ultimately from the peer. Further explanation is required of the allegation that it is a software bug. – Snowdrift

+1. In my C# program, I was banging my head as to why EndReceive is throwing an exception when there is a graceful client termination. Didnt know this is by design. I feel its bad design to throw exception in normal code flows. Thank God for the overloaded method. – Troop

@pavanManjunath how to do this :( Could you help me?? – Embodiment

@MSaudi This is using an asynchronous call, be sure you understand your code will not stop while it is processing the data being returned. Example uses msdn.microsoft.com/en-us/library/bew39x2a(v=vs.110).aspx . Basically you have to have a StateObject class with public byte[] buffer = new byte[1024], public Socket socket; and call a function called Receive(Socket s), which does StateObject so = new StateObject(); so.socket = s; s.BeginReceive(so.buffer, 0, 256, 0, new AsyncCallback(ReceiveCallback), so); and in void ReceiveCallback(IAsyncResult ar) you call that code, above. – Daye

@Wive While it offers a solution, I never saw value in asynchronous methods where you need to know right away what is coming back after you do your Send before you can do anything else. – Daye

Actually I found that this method is not foolproof. It can also cause the OP's error: #17791112 – Daye

Answer 4

Had the same bug. Actually worked in case the traffic was sent using some proxy (fiddler in my case). Updated .NET framework from 4.5.2 to >=4.6 and now everything works fine. The actual request was:
new WebClient().DownloadData("URL");
The exception was:

SocketException: An existing connection was forcibly closed by the remote host

Desimone

This was one way to solve things for me. It actually was related to which version of TLS .NET used by default. 4.5.2 and lower used TLS 1.0, while 4.6 and greater are smarter about allowing 1.1 and 1.2. This was also provable by dropping the TLS requirement on the server to 1.0, which fixed the issue as well. – Merganser

Answer 5

Simple solution for this common annoying issue:

Just go to your ".context.cs" file (located under ".context.tt" which located under your "*.edmx" file).

Then, add this line to your constructor:

public DBEntities() 
        : base("name=DBEntities") 
    { 
        this.Configuration.ProxyCreationEnabled = false; // ADD THIS LINE!
    }

Errhine

Is this a joke answer? Entity Framework 6 proxies is totally unrelated to networking/socket which this question is about. – Mccurdy

Answer 6

I've got this exception because of circular reference in entity.In entity that look like

public class Catalog
{
    public int Id { get; set; }
    public int ParentId { get; set; }
    public Catalog Parent { get; set; }
    public ICollection<Catalog> ChildCatalogs { get; set; }
}

I added [IgnoreDataMemberAttribute] to the Parent property. And that solved the problem.

Salish

Answer 7

If Running In A .Net 4.5.2 Service

For me the issue was compounded because the call was running in a .Net 4.5.2 service. I followed @willmaz suggestion but got a new error.

In running the service with logging turned on, I viewed the handshaking with the target site would initiate ok (and send the bearer token) but on the following step to process the Post call, it would seem to drop the auth token and the site would reply with Unauthorized.

Solution

It turned out that the service pool credentials did not have rights to change TLS (?) and when I put in my local admin account into the pool, it all worked.

Spittoon

Answer 8

For anyone getting this exception while reading data from the stream, this may help. I was getting this exception when reading the HttpResponseMessage in a loop like this:

using (var remoteStream = await response.Content.ReadAsStreamAsync())
using (var content = File.Create(DownloadPath))
{
    var buffer = new byte[1024];
    int read;

    while ((read = await remoteStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
    {
        await content.WriteAsync(buffer, 0, read);
        await content.FlushAsync();
    }
}

After some time I found out the culprit was the buffer size, which was too small and didn't play well with my weak Azure instance. What helped was to change the code to:

using (Stream remoteStream = await response.Content.ReadAsStreamAsync())
using (FileStream content = File.Create(DownloadPath))
{
    await remoteStream.CopyToAsync(content);
}

CopyTo() method has a default buffer size of 81920. The bigger buffer sped up the process and the errors stopped immediately, most likely because the overall download speeds increased. But why would download speed matter in preventing this error?

It is possible that you get disconnected from the server because the download speeds drop below minimum threshold the server is configured to allow. For example, in case the application you are downloading the file from is hosted on IIS, it can be a problem with http.sys configuration:

"Http.sys is the http protocol stack that IIS uses to perform http communication with clients. It has a timer called MinBytesPerSecond that is responsible for killing a connection if its transfer rate drops below some kb/sec threshold. By default, that threshold is set to 240 kb/sec."

The issue is described in this old blogpost from TFS development team and concerns IIS specifically, but may point you in a right direction. It also mentions an old bug related to this http.sys attribute: link

In case you are using Azure app services and increasing the buffer size does not eliminate the problem, try to scale up your machine as well. You will be allocated more resources including connection bandwidth.

Zonate

Answer 9

I had the same issue and managed to resolve it eventually. In my case, the port that the client sends the request to did not have a SSL cert binding to it. So I fixed the issue by binding a SSL cert to the port on the server side. Once that was done, this exception went away.

Nashom

Answer 10

I got the same issue while using .NET Framework 4.5. However, when I update the .NET version to 4.7.2 connection issue was resolved. Maybe this is due to SecurityProtocol support issue.

Slide

Answer 11

For me, it was because the app server I was trying to send email from was not added to our company's SMTP server's allowed list. I just had to put in SMTP access request for that app server.

This is how it was added by the infrastructure team (I don't know how to do these steps myself but this is what they said they did):

1.  Log into active L.B.
2.  Select: Local Traffic > iRules > Data Group List
3.  Select the appropriate Data Group
4.  Enter the app server's IP address
5.  Select: Add
6.  Select: Update
7.  Sync config changes

Supplementary

Answer 12

We are using a SpringBoot service. Our restTemplate code looks like below:

@Bean
    public RestTemplate restTemplate(final RestTemplateBuilder builder) {

        return builder.requestFactory(() -> {
            final ConnectionPool okHttpConnectionPool =
                    new ConnectionPool(50, 30, TimeUnit.SECONDS);
            final OkHttpClient okHttpClient =
                    new OkHttpClient.Builder().connectionPool(okHttpConnectionPool)
                            // .connectTimeout(30, TimeUnit.SECONDS)
                            .retryOnConnectionFailure(false).build();

            return new OkHttp3ClientHttpRequestFactory(okHttpClient);
        }).build();
    }

All our call were failing after the ReadTimeout set for the restTemplate. We increased the time, and our issue was resolved.

Speechless

Answer 13

Yet another possibility for this error to occur is if you tried to connect to a third-party server with invalid credentials too many times and a system like Fail2ban is blocking your IP address.

Mccurdy

Answer 14

I was trying to connect to the MQTT broker using the GO client, broker address was given as address + port, or tcp://address:port

Example: ❌

mqtt://test.mosquitto.org

which indicates that you wish to establish an unencrypted connection.

To request MQTT over TLS use one of ssl, tls, mqtts, mqtt+ssl or tcps.

Example: ✅

mqtts://test.mosquitto.org

Continuant

Answer 15

In my case, enable the IIS server & then restart and check again.

Cruel

Answer 16

I was getting this error sometimes too. Increasing the server memory solved my problem. please make sure you have enough memory on your server

Inconvertible

Answer 17

Sometimes this could be IIS issue. If you have created a new self signed certificate than: Open IIS manager, make sure you have updated the new ssl certificate name in each of the bindings in IIS. i.e (right click -> edit bindings -> https[edit->add new ssl])

Diastema

Answer 18

Although I do not recommend it, another quick and easy way to solve this is to remove your dependency injection implementation of this mail service and instantiate it each time you call it.

Kissner

Answer 19

I had this exception when I returned a complex type that contained DateTime properties. My server uses .Net 4.8.1 and my client uses .Net 6.0

For now I changed the datatype to string in order to make it work. I don't know how to change the default serialization from microsoft.

Another solution would be to not to return the complex type but a serialized json string, where you can chose the format.

Concavity

Answer 20

solved by adding :

   using (WebClient client = new WebClient())
        {
        ServicePointManager.Expect100Continue = true;
        ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;

        client.Encoding = Encoding.UTF8;
            string xmlData = client.DownloadString(apiUrl);}

Untaught

Answer 21

This error occurred in my application with the CIP-protocol whenever I didn't Send or received data in less than 10s.

This was caused by the use of the forward open method. You can avoid this by working with an other method, or to install an update rate of less the 10s that maintain your forward-open-connection.

Wynnie