Scrapy Splash on Ubuntu server: got an unexpected keyword argument 'encoding'

Question

The Scrapy Splash I am using is working just fine on my local machine, but it returns this error when I use it on my Ubuntu server. Why is that? Is it caused by low memory?

  File "/usr/local/lib64/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 53, in process_response
    spider=spider)
  File "/usr/local/lib/python2.7/site-packages/scrapy_splash/middleware.py", line 387, in process_response
    response = self._change_response_class(request, response)
  File "/usr/local/lib/python2.7/site-packages/scrapy_splash/middleware.py", line 402, in _change_response_class
    response = response.replace(cls=respcls, request=request)
  File "/usr/local/lib/python2.7/site-packages/scrapy/http/response/text.py", line 50, in replace
    return Response.replace(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/scrapy/http/response/__init__.py", line 79, in replace
    return cls(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/scrapy_splash/response.py", line 33, in __init__
    super(_SplashResponseMixin, self).__init__(url, *args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'encoding'

UPDATE

It only happens when using localhost as SPLASH_URL

Methuselah

Answer 1

I solve it by using the exact URL instead:

SPLASH_URL = 'http://therealip:8050'

Any localhost solution doesn't work. I think it's a bug in Scrapy Splash.

UPDATE

It turns out, the error is also gone if I turn off Crawlera. But then it reproduces another error. It's best to not use localhost in the configuration.

Methuselah

Answer 2

TypeError: init() got an unexpected keyword argument 'encoding'

It seems that you passing somewhere to class init (instance creating probably) an extra parameter 'encoding'. Try to investigate that.

Laylalayman