Login/Register
Can't run Playwright on AWS Lambda with Docker & xvfb
Asked Answered
amazon-web-servicesdockeraws-lambdaplaywrightxvfb
B

1

9

I've been trying to run Playwright on AWS Lambda with xvfb for days. I was able to build a few variants of Docker images that worked locally, but when pushed to AWS always encountered errors.

Dockerfile:

FROM mcr.microsoft.com/playwright/python:v1.24.0-focal 
COPY index.py /
COPY start.sh /
RUN apt install xvfb
RUN chmod 755 /start.sh
CMD ["/bin/bash", "./start.sh"]

start.sh

#!/bin/bash
xvfb-run --auto-servernum --server-num 1 --server-args "-screen 0, 1920x1080x24" python index.py

index.py

from playwright.sync_api import sync_playwright

def lambda_handler(event, context):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)
        page = browser.new_page()
        page.set_default_timeout(0)
        page.goto("http://example.com")
        title = page.title()
        browser.close()
        return title

Looks like for some permission-related reason the browser can't be started - any ideas what might be causing it & how to solve it?

2022-08-30T09:50:28.053+02:00   Traceback (most recent call last):
2022-08-30T09:50:28.053+02:00   File "index.py", line 8, in lambda_handler
2022-08-30T09:50:28.053+02:00   page = browser.new_page()
2022-08-30T09:50:28.053+02:00   File "/usr/local/lib/python3.8/dist-packages/playwright/sync_api/_generated.py", line 11188, in new_page
2022-08-30T09:50:28.053+02:00   self._sync(
2022-08-30T09:50:28.053+02:00   File "/usr/local/lib/python3.8/dist-packages/playwright/_impl/_sync_base.py", line 89, in _sync
2022-08-30T09:50:28.053+02:00   return task.result()
2022-08-30T09:50:28.053+02:00   File "/usr/local/lib/python3.8/dist-packages/playwright/_impl/_browser.py", line 174, in new_page
2022-08-30T09:50:28.053+02:00   page = await context.new_page()
2022-08-30T09:50:28.053+02:00   File "/usr/local/lib/python3.8/dist-packages/playwright/_impl/_browser_context.py", line 230, in new_page
2022-08-30T09:50:28.053+02:00   return from_channel(await self._channel.send("newPage"))
2022-08-30T09:50:28.053+02:00   File "/usr/local/lib/python3.8/dist-packages/playwright/_impl/_connection.py", line 43, in send
2022-08-30T09:50:28.053+02:00   return await self._connection.wrap_api_call(
2022-08-30T09:50:28.053+02:00   File "/usr/local/lib/python3.8/dist-packages/playwright/_impl/_connection.py", line 369, in wrap_api_call
2022-08-30T09:50:28.053+02:00   return await cb()
2022-08-30T09:50:28.053+02:00   File "/usr/local/lib/python3.8/dist-packages/playwright/_impl/_connection.py", line 78, in inner_send
2022-08-30T09:50:28.053+02:00   result = next(iter(done)).result()
2022-08-30T09:50:28.053+02:00   playwright._impl._api_types.Error: Browser closed.
2022-08-30T09:50:28.053+02:00   ==================== Browser output: ====================
2022-08-30T09:50:28.053+02:00    <launching> /ms-playwright/chromium-1015/chrome-linux/chrome --disable-field-trial-config --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --no-sandbox --user-data-dir=/tmp/playwright_chromiumdev_profile-gHIdYn --remote-debugging-pipe --no-startup-window
<launching> /ms-playwright/chromium-1015/chrome-linux/chrome --disable-field-trial-config --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --no-sandbox --user-data-dir=/tmp/playwright_chromiumdev_profile-gHIdYn --remote-debugging-pipe --no-startup-window
2022-08-30T09:50:28.053+02:00   <launched> pid=129
2022-08-30T09:50:28.053+02:00
[pid=129][err] [0830/075021.572233:WARNING:crashpad_client_linux.cc(362)] prctl: Operation not permitted (1)
[pid=129][err] [0830/075021.572233:WARNING:crashpad_client_linux.cc(362)] prctl: Operation not permitted (1)
2022-08-30T09:50:28.053+02:00   [pid=129][err] prctl(PR_SET_NO_NEW_PRIVS) failed
2022-08-30T09:50:28.053+02:00   [pid=129][err] prctl(PR_SET_NO_NEW_PRIVS) failed
2022-08-30T09:50:28.053+02:00
[pid=129][err] [129:147:0830/075023.537911:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
[pid=129][err] [129:147:0830/075023.537911:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:147:0830/075023.538017:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:146:0830/075023.774216:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:146:0830/075023.774254:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
2022-08-30T09:50:28.053+02:00   [pid=129][err] prctl(PR_SET_NO_NEW_PRIVS) failed
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:146:0830/075024.183157:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:146:0830/075024.183204:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:129:0830/075024.449390:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:129:0830/075024.453692:ERROR:network_service_instance_impl.cc(461)] Network service crashed, restarting service.
2022-08-30T09:50:28.053+02:00   [pid=129][err] prctl(PR_SET_NO_NEW_PRIVS) failed
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:129:0830/075024.510964:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:161:0830/075024.511033:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:161:0830/075024.511061:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:161:0830/075024.511109:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:161:0830/075024.512161:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:161:0830/075024.512820:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:129:0830/075024.570748:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:129:0830/075024.632383:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:129:0830/075024.671409:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:129:0830/075024.674327:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:146:0830/075024.854785:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:146:0830/075024.854822:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:129:0830/075025.701411:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:129:0830/075025.712194:ERROR:network_service_instance_impl.cc(461)] Network service crashed, restarting service.
2022-08-30T09:50:28.053+02:00   [pid=129][err] prctl(PR_SET_NO_NEW_PRIVS) failed
2022-08-30T09:50:28.053+02:00   [pid=129][err] [129:129:0830/075025.793276:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00   [pid=129][err] [0830/075025.850673:ERROR:scoped_ptrace_attach.cc(27)] ptrace: Operation not permitted (1)
2022-08-30T09:50:28.053+02:00   [pid=129][err] Received signal 11 SEGV_MAPERR 000000000000
2022-08-30T09:50:28.053+02:00   [pid=129][err] #0 0x55fa8ab9efb9 base::debug::CollectStackTrace()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #1 0x55fa8ab11853 base::debug::StackTrace::StackTrace()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #2 0x55fa8ab9eac1 base::debug::(anonymous namespace)::StackDumpSignalHandler()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #3 0x7f93d4dc4420 (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so+0x1441f)
2022-08-30T09:50:28.053+02:00   [pid=129][err] #4 0x55fa88ae4589 content::WebContentsImpl::SetDeviceEmulationSize()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #5 0x55fa8864714b content::protocol::EmulationHandler::SetDeviceMetricsOverride()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #6 0x55fa886476e8 content::protocol::EmulationHandler::SetDeviceMetricsOverride()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #7 0x55fa884a52af content::protocol::Emulation::DomainDispatcherImpl::setDeviceMetricsOverride()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #8 0x55fa89b0c0c3 v8_crdtp::UberDispatcher::DispatchResult::Run()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #9 0x55fa886286fd content::DevToolsSession::HandleCommandInternal()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #10 0x55fa8862866b content::DevToolsSession::HandleCommand()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #11 0x55fa8862ac6e base::internal::Invoker<>::RunOnce()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #12 0x55fa8d741a18 ChromeDevToolsSession::HandleCommand()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #13 0x55fa8d740048 ChromeDevToolsManagerDelegate::HandleCommand()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #14 0x55fa8862858e content::DevToolsSession::DispatchProtocolMessageInternal()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #15 0x55fa8862823f content::DevToolsSession::DispatchProtocolMessage()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #16 0x55fa88624162 base::internal::FunctorTraits<>::Invoke<>()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #17 0x55fa8ab5b010 base::TaskAnnotator::RunTaskImpl()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #18 0x55fa8ab6ebfc base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #19 0x55fa8ab6e7ad base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #20 0x55fa8ab6f172 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #21 0x55fa8ab1834b base::(anonymous namespace)::WorkSourceDispatch()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #22 0x7f93d4c7917d g_main_context_dispatch
2022-08-30T09:50:28.053+02:00   [pid=129][err] #23 0x7f93d4c79400 (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.6+0x523ff)
2022-08-30T09:50:28.053+02:00   [pid=129][err] #24 0x7f93d4c794a3 g_main_context_iteration
2022-08-30T09:50:28.053+02:00   [pid=129][err] #25 0x55fa8ab18177 base::MessagePumpGlib::Run()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #26 0x55fa8ab6f3e3 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #27 0x55fa8ab37aed base::RunLoop::Run()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #28 0x55fa8859dcad content::BrowserMainLoop::RunMainMessageLoop()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #29 0x55fa8859f4f2 content::BrowserMainRunnerImpl::Run()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #30 0x55fa8859b7dc content::BrowserMain()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #31 0x55fa8a67ab88 content::RunBrowserProcessMain()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #32 0x55fa8a67c0da content::ContentMainRunnerImpl::RunBrowser()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #33 0x55fa8a67bc70 content::ContentMainRunnerImpl::Run()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #34 0x55fa8a678e1e content::RunContentProcess()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #35 0x55fa8a6797de content::ContentMain()
2022-08-30T09:50:28.053+02:00   [pid=129][err] #36 0x55fa86f8f227 ChromeMain
2022-08-30T09:50:28.053+02:00   [pid=129][err] #37 0x7f93d3eef083 __libc_start_main
2022-08-30T09:50:28.053+02:00   [pid=129][err] #38 0x55fa86f8f02a _start
2022-08-30T09:50:28.053+02:00   [pid=129][err] r8: aaaaaaaaaaaaaaaa r9: 00007fff828b2460 r10: 0000000000000003 r11: 00000000000002d0
2022-08-30T09:50:28.053+02:00   [pid=129][err] r12: 00007fff828b2730 r13: 00007fff828b2460 r14: 0000000000000000 r15: 00000000000002d0
2022-08-30T09:50:28.053+02:00   [pid=129][err] di: 00003ac400a8e070 si: 00007fff828b2460 bp: 00007fff828b2280 bx: 00003ac40108cc00
2022-08-30T09:50:28.053+02:00   [pid=129][err] dx: aaaaaaaaaaaaaaaa ax: 0000000000000000 cx: 000055fa90552dd8 sp: 00007fff828b2230
2022-08-30T09:50:28.053+02:00   [pid=129][err] ip: 000055fa88ae4589 efl: 0000000000010206 cgf: 002b000000000033 erf: 0000000000000004
2022-08-30T09:50:28.053+02:00   [pid=129][err] trp: 000000000000000e msk: 0000000000000000 cr2: 0000000000000000
2022-08-30T09:50:28.053+02:00   [pid=129][err] [end of stack trace]
Baker answered 30/8, 2022 at 8:16 Comment(0)
C
4

Regarding your lambda, I have made some progress. Firstly, I tested your lambda locally with AWS SAM commands sam build && sam local invoke [fn-name], and later deployed the stack to AWS with sam deploy --guided. To do this, I created a SAM template for your lambda code, I used the Playwright image as base and added the AWS Lambda runtime interface in the 2nd stage of the multi-stage build.

When I ran the lambda on AWS, I noticed that I needed to add the --single-process launch argument if xvfb-run was used in the entrypoint script. After some testing, I was able to run the lambda on AWS with:

browser = p.chromium.launch(headless=True, args=["--single-process"])

Unfortunately, even with xvfb-run in the entrypoint script, I was unable to switch to non-headless mode. However, I believe you want to use xvfb-run to view the browser window in the virtual frame buffer. In a previous project, I successfully used xvfb with x11vnc to establish a VNC connection to a Playwright docker container running with headless=False.

It's worth noting that connecting to the frame buffer running on Lambda may not be possible as this will be incoming traffic, and therefore you may need to run Playwright in headless mode anyway. Additionally, Lambda only runs for a maximum of 15 minutes.

Considering these limitations, I suggest using a service like Fargate to run your Docker image with xvfb and x11vnc. This will provide more control and flexibility for your project. Additionally, I think it will be easier to allow incoming VNC over SSH traffic.

Corposant answered 26/4, 2023 at 20:6 Comment(2)
If I understand correctly, @LordWilmore is saying that starting a bounty does not excuse you from this decorum (LordWilmore, correct me if I'm wrong). Although I agree that partial answers should generally not be posted as answers, I think that this answer is definitely at least somewhat useful and not worth deletion. Others may disagree with me, but I do not support deleting this answer. – Naphtha
I had a related issue that was solved by adding --single-process when launching the browser. – Aseity

© 2022 - 2024 — McMap. All rights reserved.