I am trying to host a webscraping function on aws lambda and am running into webdriver errors for selenium. Could someone show me how you go about adding the chromedriver.exe file and how do you get the pathing to work in AWS Lambda function. This is the portion of my function that has to do with selenium,
from selenium import webdriver
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Select
from selenium.webdriver.chrome.service import Service
import pandas as pd
import mysql.connector
from sqlalchemy import create_engine
url = 'https://covid19criticalcare.com/pharmacies/'
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.maximize_window()
driver.get(url)
wait = WebDriverWait(driver, 5)
I tried creating a lambda layer with the chromedriver.exe file
I followed this guide (https://dev.to/awscommunity-asean/creating-an-api-that-runs-selenium-via-aws-lambda-3ck3) but I couldn't add the headless chromium because of the file size pushing me over my function limit (my pandas and numpy dependence layers have taken up most of my space)
I tried driver = webdriver.Chrome(with a path variable) and tried different pathing but wasn't sure what the beginning of the path would be since its on a lambda function.