Python http.client.Incomplete Read(0 bytes read) error

Question

I have seen this error on the forum and read through the responses yet I still don't understand what it is or how to address it. I'm scraping data from the internet from 16k links, my script scrapes similar information from each link and writes it to a .csv some of the date gets written before this error.

Traceback (most recent call last):
 File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 541, in _get_chunk_left
   chunk_left = self._read_next_chunk_size()
 File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 508, in _read_next_chunk_size
   return int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 558, in _readall_chunked
   chunk_left = self._get_chunk_left()
File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 543, in _get_chunk_left
   raise IncompleteRead(b'')
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "MoviesToDb.py", line 91, in <module>
html = r.read()
File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 455, in read
   return self._readall_chunked()
File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 565, in _readall_chunked
   raise IncompleteRead(b''.join(value))
http.client.IncompleteRead: IncompleteRead(17891 bytes read)

I would like to know:
1) What does this error mean?
2) How do I prevent it?

Heidyheifer

Answer 1

try to import :

from http.client import IncompleteRead

and add this in your script :

except IncompleteRead:
    # Oh well, reconnect and keep trucking
        continue

Blowhard

How do I incorporate this with docker on airflow? right now I am running into this issue when the docker layers are downloading. – Oenone

Answer 2

requests.exceptions.ChunkedEncodingError: (‘Connection broken: IncompleteRead(0 bytes read)’, IncompleteRead(0 bytes read)).

It is because the server of http protocal is 1.0 version，while python use 1.1 version. The solution is to assign the protocal version of client, like this

Python3 Version please add:

> import http.client
> http.client.HTTPConnection._http_vsn = 10
> http.client.HTTPConnection._http_vsn_str = 'HTTP/1.0'

Python2 Version please add:

> import http.client
> http.client.HTTPConnection._http_vsn = 10
> http.client.HTTPConnection._http_vsn_str = 'HTTP/1.0'

See the reference How to deal with "http.client.IncompleteRead: IncompleteRead(0 bytes read)" problem

Skewer