python: [Errno 10054] An existing connection was forcibly closed by the remote host
Asked Answered
A

7

78

I am writing python to crawl Twitter space using Twitter-py. I have set the crawler to sleep for a while (2 seconds) between each request to api.twitter.com. However, after some times of running (around 1), when the Twitter's rate limit not exceeded yet, I got this error.

[Errno 10054] An existing connection was forcibly closed by the remote host.

What are possible causes of this problem and how to solve this?

I have searched through and found that the Twitter server itself may force to close the connection due to many requests.

Thank you very much in advance.

Academic answered 11/1, 2012 at 5:54 Comment(1)
Did you solve it ?Derwon
E
30

This can be caused by the two sides of the connection disagreeing over whether the connection timed out or not during a keepalive. (Your code tries to reused the connection just as the server is closing it because it has been idle for too long.) You should basically just retry the operation over a new connection. (I'm surprised your library doesn't do this automatically.)

Ennis answered 11/1, 2012 at 5:58 Comment(3)
I have the same problem. Using bottle library, and sending with httplib. I can't really send retry, because the original call was already executed on server. The connection was closed when I tried to read response data. This happens not all the time, usually when I just spam server with requests. Do you know any parameters I can tweak to make the communication stable?Anelace
@RomanHwang You either need a way to check on the previous operation without repeating it or you need to make your operations idempotent.Ennis
Thanks for the hint. I also found out the reason of why I get the error so often. It's because of implementation of default development server of bottle. It's single-threaded and is not suited to handle too many requests at a time.Anelace
B
14

there are many causes such as

  • The network link between server and client may be temporarily going down.
  • running out of system resources.
  • sending malformed data.

To examine the problem in detail, you can use Wireshark.

or you can just re-request or re-connect again.

Birdseed answered 11/1, 2012 at 20:44 Comment(0)
G
14

I know this is a very old question but it may be that you need to set the request headers. This solved it for me.

For example 'user-agent', 'accept' etc. here is an example with user-agent:

url = 'your-url-here'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36'}
r = requests.get(url, headers=headers)
Gladsome answered 22/7, 2020 at 16:3 Comment(2)
Can you add just some details?Tipi
Extra details: Imagine writing a crawler to poll twitter, and since the crawler isn't a browser it won't have the user-agent by default. So the website is saying please trick us into thinking you're using a real browser with established user-agent settings, like Mozilla, AppleWebKit, Chrome, etc browser.Cutcheon
Q
4

I got the same error ([WinError 10054] An existing connection was forcibly closed by the remote host) with websocket-client after setting ping_interval = 2 in websocket.run_forever(). (I had multiple threads connecting to the same host.)

Setting ping_interval = 10 and ping_timeout = 9 solved the issue. May be you need to reduce the amount of requests and stop making host busy otherwise it will forcibly disconnect you.

Quickly answered 20/1, 2022 at 12:26 Comment(0)
A
2

For me this problem arised while trying to connect to the SAP Hana database. When I got this error,

OperationalError: Lost connection to HANA server (ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

I tried to run the code for connection(mentioned below), which created that error, again and it worked.


    import pyhdb
    connection = pyhdb.connect(host="example.com",port=30015,user="user",password="secret")
    cursor = connection.cursor()
    cursor.execute("SELECT 'Hello Python World' FROM DUMMY")
    cursor.fetchone()
    connection.close()

It was because the server refused to connect. It might require you to wait for a while and try again. Try closing the Hana Studio by logging off and then logging in again. Keep running the code for a number of times.

Askwith answered 11/9, 2017 at 13:54 Comment(1)
A separate question, please. Any chance you know where Windows10 stores connection strings? I thought it was in C:\Users\User-Name\AppData\Roaming\Microsoft\MicrosoftSQL_Server\\110\Tools\Shell\RegServer.xml (This is for SQL Server, of course)Diagnose
I
2

I fixed it with a while try loop, waiting for the response to set the variable in order to exit the loop.

When the connection has an exception, it waits five seconds, and continues looking for the response from the connection.

My code before fix, with the failed response HTTPSConnectionPool(host='etc.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x000001E9955A2050>, 'Connection to example.net timed out. (connect timeout=None)'))

 

from __future__ import print_function
import sys
import requests


def condition_questions(**kwargs):
    proxies = {'https': 'example.com', 'http': 'example.com:3128'}
    print(kwargs, file=sys.stdout)
    headers = {'etc':'etc',}
    body = f'''<etc>
                </etc>'''

    try:
        response_xml = requests.post('https://example.com', data=body, headers=headers, proxies=proxies)
    except Exception as ex:
        print("exception", ex, file=sys.stdout)
        log.exception(ex)
    finally:
        print("response_xml", response_xml, file=sys.stdout)
        return response_xml

After fix, with successful response response_xml <Response [200]>:


import time
...

response_xml = ''
    while response_xml == '':
        try:
            response_xml = requests.post('https://example.com', data=body, headers=headers, proxies=proxies)
            break
        except Exception as ex:
            print("exception", ex, file=sys.stdout)
            log.exception(ex)
            time.sleep(5)
            continue
        finally:
            print("response_xml", response_xml, file=sys.stdout)
            return response_xml

based on Jatin's answer here --"Just do this,

import time

page = ''
while page == '':
    try:
        page = requests.get(url)
        break
    except:
        print("Connection refused by the server..")
        print("Let me sleep for 5 seconds")
        print("ZZzzzz...")
        time.sleep(5)
        print("Was a nice sleep, now let me continue...")
        continue

You're welcome :)"

Induplicate answered 1/12, 2022 at 20:33 Comment(0)
D
0

I got the same error and adding proxies to my request solved it. For example,

proxies = {
  "http": "http://proxy.company.com:8080",
  "https": "http://proxy.company.com:8080"
}
headers = {
  "Accept": "application/json"
}

response = requests.get(
  url=url, 
  auth=HTTPBasicAuth(username=username, password=password),
  headers=headers,
  proxies=proxies,
  timeout=10
)
Dubuffet answered 28/1 at 19:34 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.