Memory leak from looping SKVideoNode (only on actual device)

Question

I have a memory leak that I cannot diagnose. I have tried multiple approaches to creating a seamlessly looped video - Besides AVPlayerLooper, all of the approaches I've encountered and tried involve creating an observer to watch AVPlayerItemDidPlayToEndTimeNotification and then either seeking to the beginning of the video (in the case of AVPlayer) or inserting the video to be looped into the video queue (in the case of AVQueuePlayer). Both seem to have similar performance, but both also have a consistent memory keep related to the seekToTime method (in the case of AVPlayer) and the insertItem method (in the case of AVQueuePlayer). My end goal is to create a subclass of SKVideoNode that loops by default. Below is my code for the subclass:

#import "SDLoopingVideoNode.h"
#import <AVFoundation/AVFoundation.h>


@interface SDLoopingVideoNode()
@property AVQueuePlayer *avQueuePlayer;
@property AVPlayerLooper *playerLooper;
@end

@implementation SDLoopingVideoNode

-(instancetype)initWithPathToResource:(NSString *)path withFiletype:(NSString *)filetype
{
if(self == [super init])
{
    NSString *resourcePath = [[NSBundle mainBundle] pathForResource:path ofType:filetype];
    NSURL *videoURL = [NSURL fileURLWithPath:resourcePath];
    AVAsset *videoAsset = [AVAsset assetWithURL:videoURL];
    AVPlayerItem * videoItem  = [AVPlayerItem playerItemWithAsset:videoAsset];


    self.avQueuePlayer = [[AVQueuePlayer alloc] initWithItems:@[videoItem]];

    NSNotificationCenter *noteCenter = [NSNotificationCenter defaultCenter];
    [noteCenter addObserverForName:AVPlayerItemDidPlayToEndTimeNotification
                            object:nil
                             queue:nil
                        usingBlock:^(NSNotification *note) {
                            AVPlayerItem *video = [[AVPlayerItem alloc] initWithURL:videoURL];
                            [self.avQueuePlayer insertItem:video afterItem:nil];
                            NSLog(@"Video changed");
                        }];

    self = (SDLoopingVideoNode*)[[SKVideoNode alloc] initWithAVPlayer: self.avQueuePlayer];
    return self;
}
return nil;
}


@end

And here is how the subclass is initialized in didMoveToView:

SDLoopingVideoNode *videoNode = [[SDLoopingVideoNode alloc]initWithPathToResource:@"147406" withFiletype:@"mp4"];
[videoNode setSize:CGSizeMake(self.size.width, self.size.height)];
[videoNode setAnchorPoint:CGPointMake(0.5, 0.5)];
[videoNode setPosition:CGPointMake(0, 0)];
[self addChild:videoNode];
[videoNode play];

Cribbing

Another strange thing I've noticed - this memory leak only shows itself on a device - when ran on the simulator there is no memory leak. – Cribbing

Answer 1

Short answer is, you will not be able to get that working with AVPlayer. Believe me, I have tried. Instead, it is possible to do seamless looping by using the H264 hardware to decode and then re-encode each video frame as a keyframe, github link here. I have also built a seamless looping layer that supports a full alpha channel. Performance even for full screen 1x1 video on and iPad or iPad pro is great. Also, no memory leaks with this code.

Methanol

Thanks for the reply! Is that your project? I downloaded and viewed that project earlier today - the looping is great, but do you know a way to embed that looped video into a SKVideoNode? The desired effect I am going for is a video background in Sprite Kit. – Cribbing

The looping on the code above works well, but initializing an SKVideoNode with the AVQueuePlayer and adding a video to the queue causes a memory link on every video loop. – Cribbing

No, you cannot embed that in a SKVideoNode directly. But, you can decode and then render a video frame to a pixel buffer and then set that pixel buffer as the texture for a normal SKSpriteNode node. For minimal memory use, one could use SKMutableTexture and wait for the async callback to copy over the pixel buffer contents. – Methanol

Excellent, thank you! This is an area I don't have much experience with, do you know of any good references to start? – Cribbing

The only docs are the Apple ones, but they are quite limited. If you want to see a working example of a fast FPS looping done with a texture upload approach (it is not exactly the same implementation, but close) see the linked example SpriteKitFireAnimation. #22471407 – Methanol

I ran that SpriteKitFireAnimation demo on an iPhone 6 running iOS 10 and I'm getting less than 3 FPS. – Cribbing

Are you running it in DEBUG mode? I have not actually upgraded my devices to iOS 10 yet, but I would start with making sure optimizations were turned on. Anyway, the demo does not directly matter to your problem as the implementation is not the same as decoding video and sending the decoded frame using SKMutableTexture. I was just providing an example of something like that approach in SpriteKit that you could actually run. You would still need to build your own implementation. – Methanol

I just tested that SpriteKitFireAnimation demo on iOS 10 and it is working at 60 FPS in Optimized mode. You have to wait for the textures to start loading and there is something weird going on where the FPS display in the lower right seems to not update until you rotate the screen. – Methanol