Selenium AttributeError: 'WebDriver' object has no attribute 'find_element_by_css_selector'

Question

I am following this build of a scraper for LinkedIn job data.

Here is my code:

from selenium import webdriver
import time
import pandas as pd

url = 'https://www.linkedin.com/jobs/search?keywords=&location=San%20Francisco%2C%20California%2C%20United%20States&locationId=&geoId=102277331&f_TPR=&distance=100&position=1&pageNum=0'

wd = webdriver.Chrome(executable_path=r'/Users/voi/chromedriver')
wd.get(url)

no_of_jobs = int(wd.driver.find_element_by_css_selector('h1>span').get_attribute('innerText'))

I have seen this, and attempted the solution, but received a similar error, except with regards to the WebDriver object not having a driver attribute.

Here is the full error message:

cd /Users/voi ; /usr/bin/env /usr/local/bin/python3 /Users/voi/.vscode/extensions/ms-python.python-2
022.8.1/pythonFiles/lib/python/debugpy/launcher 59402 -- /Users/voi/jobscrape.py 
/Users/voi/jobscrape.py:7: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
  wd = webdriver.Chrome(executable_path=r'/Users/voi/chromedriver')
Traceback (most recent call last):
  File "/Users/voi/jobscrape.py", line 10, in <module>
    no_of_jobs = int(wd.find_element_by_css_selector('h1>span').get_attribute('innerText'))
AttributeError: 'WebDriver' object has no attribute 'find_element_by_css_selector'

Reg

Answer 1

Okay, I answered my own question. The individual methods find_element_by_* have been replaced by find_element, e.g.

no_of_jobs = int(wd.find_element(By.CSS_SELECTOR, 'h1>span'))

More info is here

Reg

You are also getting the DeprecationWarning to resolve this you can see this [https://mcmap.net/q/219571/-deprecationwarning-with-selenium-geckodriver] , hope this will help you. – Sanious

Answer 2

Selenium recently removed the 16 deprecated find_element(s)_by_x functions in favor of a general find_element and find_elements function that take the "by" part as their first argument.

To update your code, you can use your IDE's find-and-replace-all feature to replace these 16 search terms:

.find_element_by_class_name(
.find_element(By.CLASS_NAME, 

.find_element_by_css_selector(
.find_element(By.CSS_SELECTOR, 

.find_element_by_id(
.find_element(By.ID, 

.find_element_by_link_text(
.find_element(By.LINK_TEXT, 

.find_element_by_name(
.find_element(By.NAME, 

.find_element_by_partial_link_text(
.find_element(By.PARTIAL_LINK_TEXT, 

.find_element_by_tag_name(
.find_element(By.TAG_NAME, 

.find_element_by_xpath(
.find_element(By.XPATH, 

.find_elements_by_class_name(
.find_elements(By.CLASS_NAME, 

.find_elements_by_css_selector(
.find_elements(By.CSS_SELECTOR, 

.find_elements_by_id(
.find_elements(By.ID, 

.find_elements_by_link_text(
.find_elements(By.LINK_TEXT, 

.find_elements_by_name(
.find_elements(By.NAME, 

.find_elements_by_partial_link_text(
.find_elements(By.PARTIAL_LINK_TEXT, 

.find_elements_by_tag_name(
.find_elements(By.TAG_NAME, 

.find_elements_by_xpath(
.find_elements(By.XPATH, 

You'll also need to import By if you haven't already done so:

from selenium.webdriver.common.by import By

Eringo

Answer 3

Selenium just removed that method in version 4.3.0. See the CHANGES: https://github.com/SeleniumHQ/selenium/blob/a4995e2c096239b42c373f26498a6c9bb4f2b3e7/py/CHANGES

Selenium 4.3.0
* Deprecated find_element_by_* and find_elements_by_* are now removed (#10712)
* Deprecated Opera support has been removed (#10630)
* Fully upgraded from python 2x to 3.7 syntax and features (#10647)
* Added a devtools version fallback mechanism to look for an older version when mismatch occurs (#10749)
* Better support for co-operative multi inheritance by utilising super() throughout
* Improved type hints throughout

You now need to use:

driver.find_element("css selector", SELECTOR)

In your example, you would use:

no_of_jobs = int(wd.find_element("css selector", "h1 > span").get_attribute("innerText"))

For improved reliability, you should consider using WebDriverWait in combination with visibility_of_element_located.

Here's how that might look:

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

# ...

element = WebDriverWait(wd, 10).until(
    EC.visibility_of_element_located(("css selector", "h1 > span"))
)
no_of_jobs = int(element.get_attribute("innerText"))

Reseau

Answer 4

Starting from Selenium version 4.0, the methods find_element_by_* and find_elements_by_* are no longer recommended to be used and have been replaced by the methods find_element() and find_elements().

These new methods allow you to locate elements on a web page by specifying a search strategy, such as by CSS selector, ID, name, etc.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

url = 'https://www.linkedin.com/jobs/search?keywords=&location=San%20Francisco%2C%20California%2C%20United%20States&locationId=&geoId=102277331&f_TPR=&distance=100&position=1&pageNum=0'

wd = webdriver.Chrome(executable_path=r'/Users/voi/chromedriver')
wd.get(url)
element = WebDriverWait(wd, 20).until(
    EC.visibility_of_element_located((By.CSS_SELECTOR, "h1>span"))
)

Please checkout details here

Waspish

Answer 5

To add on to the answer from @m.lekk ( https://mcmap.net/q/218686/-selenium-attributeerror-39-webdriver-39-object-has-no-attribute-39-find_element_by_css_selector-39 ), I also tried to use dir() to get all the attributes from the object and find the text attribute that contains the information that I need.

Sthenic

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. – Diapedesis

This does not provide an answer to the question. Once you have sufficient reputation you will be able to comment on any post; instead, provide answers that don't require clarification from the asker. - From Review – Simard