fake_useragent module not connecting properly - IndexError: list index out of range

Question

I tried to use fake_useragent module with this block

from fake_useragent import UserAgent

ua = UserAgent()
print(ua.random)

But when the execution reached this line ua = UserAgent(), it throws this error

Traceback (most recent call last):
  File "/home/hadi/Desktop/excel/gatewayform.py", line 191, in <module>
    gate = GateWay()
  File "/home/hadi/Desktop/excel/gatewayform.py", line 23, in __init__
    ua = UserAgent()
  File "/usr/local/lib/python3.9/dist-packages/fake_useragent/fake.py", line 69, in __init__
    self.load()
  File "/usr/local/lib/python3.9/dist-packages/fake_useragent/fake.py", line 75, in load
    self.data = load_cached(
  File "/usr/local/lib/python3.9/dist-packages/fake_useragent/utils.py", line 250, in load_cached
    update(path, use_cache_server=use_cache_server, verify_ssl=verify_ssl)
  File "/usr/local/lib/python3.9/dist-packages/fake_useragent/utils.py", line 245, in update
    write(path, load(use_cache_server=use_cache_server, verify_ssl=verify_ssl))
  File "/usr/local/lib/python3.9/dist-packages/fake_useragent/utils.py", line 178, in load
    raise exc
  File "/usr/local/lib/python3.9/dist-packages/fake_useragent/utils.py", line 154, in load
    for item in get_browsers(verify_ssl=verify_ssl):
  File "/usr/local/lib/python3.9/dist-packages/fake_useragent/utils.py", line 99, in get_browsers
    html = html.split('<table class="w3-table-all notranslate">')[1]
IndexError: list index out of range

I use linux and I have installed the module using this command pip3 install fake_useragent --upgrade.

Is there any solution for this issue? if not, is there a better module to use?

Nitro

The error comes from this html = html.split('<table class="w3-table-all notranslate">')[1] and not the fake_useragent. – Prolong

Yes, and this line of code comes from utils.py which is in the fake_useragent module – Nitro

Answer 1

There is a solution for this, from Github pull request #110. Basically, all you need to do is change one character in one line of the fake_useragent/utils.py source code.

To do this on your system, open /usr/local/lib/python3.9/dist-packages/fake_useragent/utils.py^† in your favorite text editor using admin privileges. Go to line 99, and change the w3

    html = html.split('<table class="w3-table-all notranslate">')[1]
#                                    ^^ change this

to ws:

    html = html.split('<table class="ws-table-all notranslate">')[1]
#                                    ^^ to this

Save the file (with admin permissions), restart your Python session, and your code should work just fine.

---

† To find the fake_useragent directory in which utils.py resides, run the following code:

import fake_useragent
print(fake_useragent.__file__)

For example, on my Windows laptop, this printed

'C:\\Users\\mattdmo\\AppData\\Roaming\\Python\\Python310\\site-packages\\fake_useragent\\__init__.py'

so the folder to open is C:\Users\mattdmo\AppData\Roaming\Python\Python310\site-packages\fake_useragent.

Chapple

That's worked just fine! Well first, I tried to remove [1], but didn't work as well. Changing w3 to ws is worked. Thanks a lot! – Nitro

How can I change this when building the package through pip3 in docker on my server? – Misfit

@Misfit since the owner of the original repo seems to not be responding to the PR, the best course of action is to fork the repo, apply the fix, update the version string, then point pip3 to the new repo. – Chapple

Using VS Code in Windows, I selected Go=>Go to file=>Searched for Utils.py=Opened XXXX\Lib\site-packages\fake_useragent\utils.py. Then I edited the line mentioned here and it worked. – Reside

Answer 2

I tried UserAgent(use_cache_server=False, verify_ssl=False) but didn't work.

Upgrading the version to 0.1.13 worked for me.

pip3 install fake-useragent==0.1.13

I tried pip install fake-useragent -U but it somehow didn't upgrade the package correctly.

From the package owner: https://github.com/fake-useragent/fake-useragent/pull/136#issuecomment-1302431518

Benzedrine