Preserving Side Data information for 360 video transcoding using FFmpeg

Question

I am trying to transcode a 360 Video using ffmpeg (to play on Gear VR, for now), mainly to reduce the bitrate (this is a requirement). But the output file seems to remove the "Side Data" that has the 360 video metadata.

Input file ffprobe result (note the Side Data section):

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input.mp4':
Metadata:
major_brand     : mp42
minor_version   : 0
compatible_brands: mp42mp41
creation_time   : 2016-10-28T10:41:42.000000Z
Duration: 00:02:09.56, start: 0.000000, bitrate: 20116 kb/s
Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709), 2560x1280, 19797 kb/s, 50 fps, 50 tbr, 50k tbn, 100 tbc (default)
Metadata:
  creation_time   : 2016-10-28T10:41:42.000000Z
  handler_name    : Alias Data Handler
  encoder         : AVC Coding
Side data:
  spherical: equirectangular (0.000000/0.000000/0.000000) 
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 317 kb/s (default)
Metadata:
  creation_time   : 2016-10-28T10:41:42.000000Z
  handler_name    : Alias Data Handler

Commands tried to transcode (values of bitrate and preset are just for testing):

ffmpeg -i <input_file_name> -vf scale=2560x1280,setdar=16:9 -r 30 -c:v libx264 -b:v 500k -pix_fmt yuv420p -preset ultrafast -c:a aac -b:a 192K -map_metadata 0:s:v:0 -write_id3v2 1 -y test.mp4
ffmpeg -i <input_file_name> -vf scale=2560x1280,setdar=16:9 -r 30 -c:v libx264 -b:v 500k -pix_fmt yuv420p -preset ultrafast -c:a aac -b:a 192K -map_metadata 0:s -y test.mp4
ffmpeg -i <input_file_name> -vf scale=2560x1280,setdar=16:9 -r 30 -c:v libx264 -b:v 500k -pix_fmt yuv420p -preset ultrafast -c:a aac -b:a 192K -map_metadata 0 -y test.mp4
ffmpeg -i <input_file_name> -vf scale=2560x1280,setdar=16:9 -r 30 -c:v libx264 -b:v 500k -pix_fmt yuv420p -preset ultrafast -c:a aac -b:a 192K -y test.mp4

In each case the ffprobe test.mp4 seems to show the missing "Side Data" section:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4':
Metadata:
major_brand     : isom
minor_version   : 512
compatible_brands: isomiso2avc1mp41
creation_time   : 2016-10-28T10:41:42.000000Z
encoder         : Lavf57.71.100
Duration: 00:02:09.57, start: 0.000000, bitrate: 708 kb/s
Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 2560x1280 [SAR 8:9 DAR 16:9], 507 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
Metadata:
  creation_time   : 2016-10-28T10:41:42.000000Z
  handler_name    : VideoHandler
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 193 kb/s (default)
Metadata:
  creation_time   : 2016-10-28T10:41:42.000000Z
  handler_name    : SoundHandler

Any help or advice to get ffmpeg to preserve the Side Data would be appreciated

Fernandez

Does -fflags keepside work as an input or output flag? – Busch

@Mulvya thanks, I tried it now as both input and output flag but the ffprobe of output is still same (without Side Data section). ffmpeg -fflags keepside -i ..... and ffmpeg -i input.mp4 -vf ..... -fflags keepside -y test.mp4 – Fernandez

@jeffcook2150 Do you have a short sample input file that you can share? – Arnoldoarnon

@LordNeckbeard see drive.google.com/file/d/1OTd5D0inwN-ir_IlMKo-Fnf23C_ux8BZ/… for a sample of a video that contains equirectangular side data. This is generated from raw Theta V file by RICOH Theta V Movie Converter: theta360.com/en/support/download/movieconverter . – Kappa

@LordNeckbeard please see drive.google.com/drive/folders/… for a folder of related samples and ffprobe outputs. – Kappa

@LordNeckbeard, was that sufficient/useful? – Kappa

@Kappa yes, see answer, it works with the present git version so make sure you're up to date. – Ecphonesis

Answer 1

In recent ffmpeg versions the spherical packet side-data is supported but to write it in MP4 you have to set the standard compliance mode to unofficial or experimental:

ffmpeg -i R0010309_er.MP4 -c copy -strict unofficial spherical.mp4

ffprobe spherical.mp4 2>&1 | grep -A1 Side
Side data:
  spherical: equirectangular (0.000000/0.000000/0.000000) 

---

The relevant code in libavformat/movenc.c function mov_write_video_tag:

    if (track->mode == MODE_MP4 && mov->fc->strict_std_compliance <= FF_COMPLIANCE_UNOFFICIAL) {
        AVStereo3D* stereo_3d = (AVStereo3D*) av_stream_get_side_data(track->st, AV_PKT_DATA_STEREO3D, NULL);
        AVSphericalMapping* spherical_mapping = (AVSphericalMapping*)av_stream_get_side_data(track->st, AV_PKT_DATA_SPHERICAL, NULL);

        if (stereo_3d)
            mov_write_st3d_tag(pb, stereo_3d);
        if (spherical_mapping)
            mov_write_sv3d_tag(mov->fc, pb, spherical_mapping);
    }

Ecphonesis

Thanks! I tried this with a git build too but I was missing the -strict unofficial. Looks like it's been supported since ffmpeg 3.3. – Kappa

@Kappa You're welcome. Please note that if you ever remux to another format (like Matroska) you don't need the compliance option. – Ecphonesis

I had the very same issue and -strict unofficial did the trick. On my side, I was actually trying to rotate the video 180° and thought that it was the issue of my video not being "360". But no, it was just the side data not being copied over :) Full command to rotate 180° a 360 video is: ffmpeg -i input.mp4 -strict unofficial -metadata:s:v rotate="-180" -codec copy output.mp4 – Ecstatics

What if our tracks are MOV and not MP4 ? ;) I can't get the Side metadata :( – Paramount