record Linux Wayland/DRM screen using ffmpeg's kmsgrab device with superimposed webcam

Question

Setup is Linux, ffmpeg using kabylake iGPU.

I am capturing a running sway instance using the kmsgrab device, which requires the use of a hardware backend to coherently process the image on my hardware. Only VA API fits this bill. I want to overlay the webcam in the bottom right corner during encoding. However, attempts at manipulating the filter graph to accomplish this have been unsuccessful. This is ultimately for Twitch/Tube stream.

Right now, I am actually capturing the webcam to an sdl window, and simply recording the screen using separate instances of ffmpeg. This doesn't actually solve my problem since the window is easily disguised by workspace switching or other windows.

This is the workaround:

#!/usr/bin/env zsh

# record webcam and open it in sdl window
ffmpeg -v quiet -hide_banner \
  -re -video_size 640X480 -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -i /dev/video0 \
  -vf 'format=nv12,hwupload' -c:v hevc_vaapi -f hevc - \
  | ffmpeg -v quiet -i - -f sdl2 - &

# wait for webcam window to open
until swaymsg -t get_tree | grep 'pipe:' &>/dev/null; do
  sleep 0.5
done

# position webcam in the bottom right corner of screen using sway
swaymsg floating enable
swaymsg resize set width 320 height 240
swaymsg move position 1580 795
swaymsg focus tiling

#screencast
ffmpeg -format bgra -framerate 60 -f kmsgrab -thread_queue_size 1024 -i - \
  -f alsa -ac 2 -thread_queue_size 1024 -i hw:0 \
  -vf 'hwmap=derive_device=vaapi,scale_vaapi=w=1920:h=1080:format=nv12' \
  -c:v h264_vaapi -g 120 -b:v 3M -maxrate 3M -pix_fmt vaapi_vld -c:a aac -ab 96k -threads $(nproc) \
  output.mkv

kill %1

Euplastic

As a further workaround you could fix the window displaying the webcam contents to stay on top all the time through the window manager – Enthymeme

Thanks, that was the first thing I tried. Unfortunately sway appears to have no such functionality from what I could find in the man page. I'm a long time user of xmonad, but I recently switched to sway since qutebrowser is wayland native. – Euplastic

Answer 1

in momento, this may be impossible.

It looks like a limitation in ffmpeg's vaapi backend. I noticed that Intel's QuickSync backend has an overlay_qsv filter, yet there is no equivalent overlay_vaapi.

In theory, there may be a mechanism to download the hardware surface to a software buffer using the hwdownload filter once it's been encoded in hevc_vaapi so as to make use of the software overlay filter to do the superimposition in the last phase of encoding. I will leave this open for now to see if someone can help work toward this end or knows anything more on the topic.

note on QuickSync:

After building with QuickSync support and attempting to use it, it appears to lack the capability to derive a surface from the kmsgrab device resulting in a function not implemented error from ffmpeg.

Euplastic

FYI, I also have the same issue with x11grab. Also, I tried to trick it by doing vaapi decode + vaapi scaling + qsv encode but no hope. Somehow it fails to parse hwmap filter – Calabash

Hello @Calabash can you share your FFmpeg command here? I can take a look and work on it. – Cellaret

There's an overlay filter in Vulkan in FFmpeg, let me see what can be whipped up to enable such a command to work. – Cellaret