Overlay animated images with transparency over a static background image using ffmpeg?

Question

I'm looking to create a video using a set of png images that have transparency merged with a static background.

After doing a lot of digging I seems like it's definitely possible by using the filters library.

My initial video making without including the background is:

ffmpeg -y -qscale 1 -r 1 -b 9600 -loop -i bg.png -i frame_%d.png -s hd720 testvid.mp4

Using -vf I can apply the background as overlay:

ffmpeg -y -qscale 1 -r 1 -b 9600 -i frame_%d.png -vf "movie=bg.png [wm];[in][wm] overlay=0:0 [out]" -s hd720 testvid.mp4

However the problem is it's overlaying the background over the input. According libacfilter I can split the input and play with it's content. I'm wondering if I can somehow change the overlay order?

Any help greatly appreciated!

UPDATE 1:
I'm trying to make the following filter work but I'm getting the movie without the background:

ffmpeg -y -qscale 1 -r 1 -b 9600 -i frame_%d.png -vf "movie=bg.png [bg]; [in] split [T1], fifo, [bg] overlay=0:0, [T2] overlay=0:0 [out]; [T1] fifo [T2]" -s hd720 testvid.mp4

UPDATE 2:
Got video making using -vf option. Just piped the input slit it applied image over it and overlayed the two split feeds! Probably not the most efficient way... but it worked!

ffmpeg -y -r 1 -b 9600 -i frame_%d.png -vf "movie=bg.png, scale=1280:720:0:0 [bg]; [in] format=rgb32, split [T1], fifo, [bg] overlay=0:0, [T2] overlay=0:0 [out]; [T1] fifo [T2]" -s hd720 testvid.mp4

Adjure

I have a a bad solution but works. After every image, put your baground image in between. This will get you the way you want the final output – Beanery

Interesting approach... The question is how do you do that? :) – Adjure

suppose you have n frame_%d.pngs, after every frame, add one bg wih after every frame... frame1.png and frame2.pn are there, then frame2.png is baground_ – Beanery

I'm not sure if this can be done without having multiple copies of a background with the proper alternating sequence. My requirements are to use the sequence of images and one background file. – Adjure

Answer 1

The overlay order is controlled by the order of the inputs, from the ffmpeg docs

[...] takes two inputs and one output, the first input is the "main" video on which the second input is overlayed.

You second command thus becomes:

ffmpeg -y -loop 1 -qscale 1 -r 1 -b 9600 -i frame_%d.png -vf "movie=bg.png [wm];[wm][in] overlay=0:0" -s hd720 testvid.mp4

With the latest versions of ffmpeg the new -filter_complex command makes the same process even simpler:

ffmpeg -loop 1 -i bg.png -i frame_%d.png -filter_complex overlay -shortest testvid.mp4

---

A complete working example:

The source of our transparent input images (apologies for dancing):

Exploded to frames with ImageMagick:

convert dancingbanana.gif -define png:color-type=6 over.png

(Setting png:color-type=6 (RGB-Matte) is crucial because ffmpeg doesn't handle indexed transparency correctly.) Inputs are named over-0.png, over-1.png, over-2.png, etc.

Our background image (scaled to banana):

Using ffmpeg version N-40511-g66337bf (a git build from yesterday), we do:

ffmpeg -loop 1 -i bg.png -r 5 -i over-%d.png -filter_complex overlay -shortest out.avi

-loop loops the background image input so that we don't just have one frame, crucial!
-r slows down the dancing banana a bit, optional.
-filter_complex is a very recently added ffmpeg feature making handling of multiple inputs easier.
-shortest ends encoding when the shortest input ends, which is necessary as looping the background means that that input will never end.

Using a slightly less cutting-edge build, ffmpeg version 0.10.2.git-d3d5e84:

ffmpeg -loop 1 -r 5 -i back.png -vf 'movie=over-%d.png [over], [in][over] overlay' -frames:v 8 out.avi

movie doesn't allow rate setting, so we slow down the background instead which gives the same effect. Because the overlaid movie isn't a proper input, we can't use -shortest and instead explicitly set the number of frames to output to how many overlaid input frames we have.

The final result (output as a gif for embedding):

Trembles

Awesome! Yeah your first suggestion I have tried but I get error "overlay area (...) not within area (...). So I tried scaling the bg.png but than I get "AV Filter told us it has a frame available but failed to output one" In regards to your second solution, what version of ffmpeg has those options? I tried 10.3 but didn't seem to have it. – Adjure

I got that 'failed to output' message in some of my testing too, seems like an ffmpeg bug. The version with -filter_complex is in the recent git builds. I don't know how long it's been present or which point releases it's in, but it's definitely fairly recent. Anything that makes the video filters easier to use is definitely a welcome addition. – Trembles

Ahaha.. For a sec I thought spam! Thanks for your post and details. It does work with the latest release. I found a solution of doing this another way too. Take a look at update 2. However it seems like your method might be less work intensive? I'm going to be using this in mobile device. – Adjure

Avoiding all the splitting and piping is probably less work, but there's no substitute for testing multiple options rather than just guessing at which works better. – Trembles

@Trembles this is a fantastic answer, however what if you wanted to take a .mov file as is and overlay it vs exploded images? – Locomotive

'ffmpeg -loop 1 -i bg.png -r 5 -i over-%d.png -filter_complex overlay -shortest out.avi', this keeps on looping over the images over-%.png and never stops. It seems like -shortest flag is not working as it supposed to. Can you suggest a fix ? – Shigella

Same here. I'm not able to make ffmpeg stop encoding. Seems like -shortest is never trigered. – Hakan

Why apologies? +1 for the dancing! – Entente

Answer 2

for references in the future as of 17/02/2015, the command-line is :

ffmpeg -loop 1 -i images/background.png -i images/video_overlay%04d.png -filter_complex overlay=shortest=1 testvid.mp4

thanks for llogan who took the time to reply here : https://trac.ffmpeg.org/ticket/4315#comment:1

Hakan