option.add_argument("--headless=new") doesn't work for Chrome 122 when scrap Kayak

Question

Chrome 122.0.6261.95

Chrome driver 122.0.6261.94

Python 3.8.3

If I comment out option.add_argument("--headless=new"), it will print(len(elements)) will print 2. Otherwise, can not print anything. Thanks a lot

import time
import subprocess
import selenium
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

from bs4 import BeautifulSoup
import pandas as pd
#from playsound import playsound
import datetime
import threading
service = Service(executable_path='chromedriver.exe')


option = webdriver.ChromeOptions()
option.add_argument("--headless=new")
option.add_argument('--ignore-certificate-errors')
option.add_argument("--no-sandbox")
option.add_argument('disable-notifications')
driver = webdriver.Chrome(service=service,options=option)

def search(dep,arr,date):
    print(f'''Input: Date:{date},Departure: {dep} - Arrival: {arr}''')
    base_url = f'https://www.kayak.com/flights/SFO-PEK/2024-03-06?sort=price_a&fs=stops=0'
    
    print("before webdriver.ChromeOptions()")
   
    my_url = base_url
    driver.get(my_url)
    print(my_url)
    time.sleep(10) # set the time to wait till web fully loaded
    
    # Find all elements with class "nrc6"
    elements = driver.find_elements(By.CLASS_NAME, "nrc6")
    
    # Print the number of elements found
    print(len(elements))

Punke

Answer 1

Please update question with error encountered so that we can figure it out.

It might be due to incompatibility with your chromedriver and browser version.

I normally used WebDriverManager so that I don't need to keep track of what is latest and not.

Feel free to try using this plugin:

<dependency>
    <groupId>io.github.bonigarcia</groupId>
    <artifactId>webdrivermanager</artifactId>
    <version>5.6.4</version>
</dependency>

Your code will be a little different since you'll need to declare WebDriverManager first. https://pypi.org/project/webdriver-manager

Janae

thx. actually, it doesn't have error, just print zero at this step "print(len(elements))" I will try WDM you mentioned. btw, not sure if headless is not working, if this code can run on linux once I deployed on cloud. – Punke