Compositing 2 videos on top of each other with alpha

Question

AVFoundation allows you to "compose" 2 assets (2 videos) as 2 "tracks", just like in Final Cut Pro, for example.

The theory says I can have 2 videos on top of each other, with alpha, and see both.

Either I'm doing something wrong, or there's a bug somewhere, because the following test code, although a bit messy, clearly states I should see 2 videos, and I only see one, as seen here: http://lockerz.com/s/172403384 -- the "blue" square is IMG_1388.m4v

For whatever reason, IMG_1383.MOV is never shown.

NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithBool:YES], AVURLAssetPreferPreciseDurationAndTimingKey, nil];
AVMutableComposition *composition = [AVMutableComposition composition];
CMTimeRange timeRange = CMTimeRangeMake(kCMTimeZero, CMTimeMakeWithSeconds(4, 1));
AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition];

// Track B
NSURL *urlVideo2 = [NSURL URLWithString:@"file://localhost/Users/me/Movies/Temp/IMG_1388.m4v"];
AVAsset *video2 = [AVURLAsset URLAssetWithURL:urlVideo2 options:options];
AVMutableCompositionTrack *videoTrack2 = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:0];
NSArray *videoAssetTracks2 = [video2 tracksWithMediaType:AVMediaTypeVideo];
AVAssetTrack *videoAssetTrack2 = ([videoAssetTracks2 count] > 0 ? [videoAssetTracks2 objectAtIndex:0] : nil);
[videoTrack2 insertTimeRange:timeRange ofTrack:videoAssetTrack2 atTime:kCMTimeZero error:&error];

AVMutableVideoCompositionLayerInstruction *to = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoAssetTrack2];
[to setOpacity:.5 atTime:kCMTimeZero];
[to setTransform:CGAffineTransformScale(videoAssetTrack2.preferredTransform, .5, .5) atTime:kCMTimeZero];

// Track A
NSURL *urlVideo = [NSURL URLWithString:@"file://localhost/Users/me/Movies/Temp/IMG_1383.MOV"];
AVURLAsset *video = [AVURLAsset URLAssetWithURL:urlVideo options:options];
AVMutableCompositionTrack *videoTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:1];
NSArray *videoAssetTracks = [video tracksWithMediaType:AVMediaTypeVideo];
AVAssetTrack *videoAssetTrack = ([videoAssetTracks count] > 0 ? [videoAssetTracks objectAtIndex:0] : nil);
[videoTrack insertTimeRange:timeRange ofTrack:videoAssetTrack atTime:kCMTimeZero error:nil];

AVMutableVideoCompositionLayerInstruction *from = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoAssetTrack];
[from setOpacity:.5 atTime:kCMTimeZero];

// Video Compostion
AVMutableVideoCompositionInstruction *transition = [AVMutableVideoCompositionInstruction videoCompositionInstruction];
transition.backgroundColor = [[UIColor clearColor] CGColor];
transition.timeRange = timeRange;
transition.layerInstructions = [NSArray arrayWithObjects:to, from, nil];
videoComposition.instructions = [NSArray arrayWithObjects:transition,  nil];
videoComposition.frameDuration = CMTimeMake(1, 30);
videoComposition.renderSize = CGSizeMake(480, 360);


// Export
NSURL *outputURL = [NSURL URLWithString:@"file://localhost/Users/me/Movies/Temp/export.MOV"];
AVAssetExportSession *exportSession = [[AVAssetExportSession alloc] initWithAsset:[[composition copy] autorelease] presetName:AVAssetExportPresetHighestQuality];
[exportSession setOutputFileType:@"com.apple.quicktime-movie"];
exportSession.outputURL = outputURL;
exportSession.videoComposition = videoComposition;
[exportSession exportAsynchronouslyWithCompletionHandler:nil];

// Player
AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:composition];
playerItem.videoComposition = videoComposition;
AVPlayer *player = [AVPlayer playerWithPlayerItem:playerItem];
AVPlayerLayer *playerLayer = [AVPlayerLayer playerLayerWithPlayer:player];

Are you seeing something wrong?

The "goal" of this code is to "record" the camera input (video 1) and the opengl output (video 2). I also tried to "compose" them "directly" with Buffers and all that, but I was as well unsuccessful :( Turns out AVFoundation is way less trivial than I thought.

Dumah

Hello @StuFF mc i want to merge multiple video with transition i have use your code from gist.github.com/stuffmc/1572592/… But i can't get output file in document directory and i have use djromero answer but also it will not work Please Help me any suggestion. – Anstice

Answer 1

It looks good, except this part:

AVMutableVideoCompositionLayerInstruction *from = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoAssetTrack];
AVMutableVideoCompositionLayerInstruction *to = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoAssetTrack2];

You need to use videoTrack and videoTrack2 to build the layer instructions, i.e., the tracks added to composition, instead of the original assets videoAssetTrack and videoAssetTrack2.

Also, adding a transformation to rotate the video it's a bit trickier (like anything in AVFoundation beyond the basics). I've just commented out the line to make it play the 2 videos.

This is your code with the modifications:

NSError *error = nil;
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithBool:YES], AVURLAssetPreferPreciseDurationAndTimingKey, nil];
AVMutableComposition *composition = [AVMutableComposition composition];
CMTimeRange timeRange = CMTimeRangeMake(kCMTimeZero, CMTimeMakeWithSeconds(4, 1));
AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition];

// Track B
NSURL *urlVideo2 = [[NSBundle mainBundle] URLForResource:@"b" withExtension:@"mov"];        
AVAsset *video2 = [AVURLAsset URLAssetWithURL:urlVideo2 options:options];
AVMutableCompositionTrack *videoTrack2 = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:0];
NSArray *videoAssetTracks2 = [video2 tracksWithMediaType:AVMediaTypeVideo];
AVAssetTrack *videoAssetTrack2 = ([videoAssetTracks2 count] > 0 ? [videoAssetTracks2 objectAtIndex:0] : nil);
[videoTrack2 insertTimeRange:timeRange ofTrack:videoAssetTrack2 atTime:kCMTimeZero error:&error];

AVMutableVideoCompositionLayerInstruction *to = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoTrack2];
[to setOpacity:.5 atTime:kCMTimeZero];
//[to setTransform:CGAffineTransformScale(videoAssetTrack2.preferredTransform, .5, .5) atTime:kCMTimeZero];

// Track A
NSURL *urlVideo = [[NSBundle mainBundle] URLForResource:@"a" withExtension:@"mov"];        
AVURLAsset *video = [AVURLAsset URLAssetWithURL:urlVideo options:options];
AVMutableCompositionTrack *videoTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:1];
NSArray *videoAssetTracks = [video tracksWithMediaType:AVMediaTypeVideo];
AVAssetTrack *videoAssetTrack = ([videoAssetTracks count] > 0 ? [videoAssetTracks objectAtIndex:0] : nil);
[videoTrack insertTimeRange:timeRange ofTrack:videoAssetTrack atTime:kCMTimeZero error:nil];

AVMutableVideoCompositionLayerInstruction *from = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoTrack];
[from setOpacity:.5 atTime:kCMTimeZero];

// Video Compostion
AVMutableVideoCompositionInstruction *transition = [AVMutableVideoCompositionInstruction videoCompositionInstruction];
transition.backgroundColor = [[UIColor clearColor] CGColor];
transition.timeRange = timeRange;
transition.layerInstructions = [NSArray arrayWithObjects:to, from, nil];
videoComposition.instructions = [NSArray arrayWithObjects:transition,  nil];
videoComposition.frameDuration = CMTimeMake(1, 30);
videoComposition.renderSize = composition.naturalSize; // CGSizeMake(480, 360);

Asur

I can't make it work yet (your code), but... It seems weird to me that I would have to add "videoTrack" which is an AVMutableCompositionTrack* where the documentation/header clearly states -[videoCompositionLayerInstructionWithAssetTrack:(AVAssetTrack *)track (so the parameter shall be an AVAssetTrack* my videoAssetTrack is. Thanks for your help though! At the end, I finally "composed" both videos "frame by frame" as stated on devforums.apple.com/message/601194#601194 – Dumah

AVMutableCompositionTrack is a subclass of AVAssetTrack. You need instructions for tracks in your composition (videoTrack and videoTrack2), otherwise it has no effect on the result. – Asur

true. Running your code with my :) videos gives me -[AVPlayerItem setVideoComposition:] video composition must have a positive renderSize though. Which code do you use to "show" the video on an AVPlayerLayer? – Dumah

The player code is the standard one, similar to yours. Did you copied last line: videoComposition.renderSize = composition.naturalSize; – Asur

Were you able to produce a video with two video on top of each other ? Let us know what changes you made to achieve that. – Electron

I know its old question but I am stuck on, How to manage both video's layer exact same on each other. – Mahlstick

Answer 2

I think you've got it wrong.

A video file may have multiple stream of data. For example, if it's a video with sound the file will have 2 streams, the Audio stream and the video stream. Another example is an audio surround video file which may include 5 or more audio stream and 1 video stream.

As with audio, most video file container format (mov, mp4, etc...) support multiple streams of video in 1 file but in fact this doesn't mean that the streams will have any relation to each other, they are just stored on the same file container. If you will open such file with QuickTime for example, you will get as many windows as video streams on such file.

Anyhow, the video streams will not get 'mix' this way. What you're trying to achieve is related to signal processing of the video stream, and I really recommend you reading more about it.

If you don't really need to 'mix' the video data together to a file, you might want to displaying both video files on each other using MPMediaPlayers. Keep in mind that dealing with video data is usually a CPU intensive problem which you might (sometime) wont be able to solve using now days iOS devices.

Exchange

he is clearly trying to mix two videos on top of one another and add opacity to one, so you can see the underlying video. If you are suggesting the composition only have one video track, how do you add the two tracks to the composer to mix? – Bucksaw