C++ FFmpeg distorted sound when converting audio

Question

I'm using the FFmpeg library to generate MP4 files containing audio from various files, such as MP3, WAV, OGG, but I'm having some troubles (I'm also putting video in there, but for simplicity's sake I'm omitting that for this question, since I've got that working). My current code opens an audio file, decodes the content and converts it into the MP4 container and finally writes it into the destination file as interleaved frames.

It works perfectly for most MP3 files, but when inputting WAV or OGG, the audio in the resulting MP4 is slightly distorted and often plays at the wrong speed (up to many times faster or slower).

I've looked at countless of examples of using the converting functions (swr_convert), but I can't seem to get rid of the noise in the exported audio.

Here's how I add an audio stream to the MP4 (outContext is the AVFormatContext for the output file):

audioCodec = avcodec_find_encoder(outContext->oformat->audio_codec);
if (!audioCodec)
    die("Could not find audio encoder!");


// Start stream
audioStream = avformat_new_stream(outContext, audioCodec);
if (!audioStream)
    die("Could not allocate audio stream!");

audioCodecContext = audioStream->codec;
audioStream->id = 1;


// Setup
audioCodecContext->sample_fmt = AV_SAMPLE_FMT_S16;
audioCodecContext->bit_rate = 128000;
audioCodecContext->sample_rate = 44100;
audioCodecContext->channels = 2;
audioCodecContext->channel_layout = AV_CH_LAYOUT_STEREO;


// Open the codec
if (avcodec_open2(audioCodecContext, audioCodec, NULL) < 0)
    die("Could not open audio codec");

And to open a sound file from MP3/WAV/OGG (from the filename variable)...

// Create contex
formatContext = avformat_alloc_context();
if (avformat_open_input(&formatContext, filename, NULL, NULL)<0)
    die("Could not open file");


// Find info
if (avformat_find_stream_info(formatContext, 0)<0)
    die("Could not find file info");

av_dump_format(formatContext, 0, filename, false);


// Find audio stream
streamId = av_find_best_stream(formatContext, AVMEDIA_TYPE_AUDIO, -1, -1, NULL, 0);
if (streamId < 0)
    die("Could not find Audio Stream");

codecContext = formatContext->streams[streamId]->codec;


// Find decoder
codec = avcodec_find_decoder(codecContext->codec_id);
if (codec == NULL)
    die("cannot find codec!");


// Open codec
if (avcodec_open2(codecContext, codec, 0)<0)
    die("Codec cannot be found");


// Set up resample context
swrContext = swr_alloc();
if (!swrContext)
    die("Failed to alloc swr context");

av_opt_set_int(swrContext, "in_channel_count", codecContext->channels, 0);
av_opt_set_int(swrContext, "in_channel_layout", codecContext->channel_layout, 0);
av_opt_set_int(swrContext, "in_sample_rate", codecContext->sample_rate, 0);
av_opt_set_sample_fmt(swrContext, "in_sample_fmt", codecContext->sample_fmt, 0);

av_opt_set_int(swrContext, "out_channel_count", audioCodecContext->channels, 0);
av_opt_set_int(swrContext, "out_channel_layout", audioCodecContext->channel_layout, 0);
av_opt_set_int(swrContext, "out_sample_rate", audioCodecContext->sample_rate, 0);
av_opt_set_sample_fmt(swrContext, "out_sample_fmt", audioCodecContext->sample_fmt, 0);

if (swr_init(swrContext))
    die("Failed to init swr context");

Finally, to decode+convert+encode...

// Allocate and init re-usable frames
audioFrameDecoded = av_frame_alloc();
if (!audioFrameDecoded)
        die("Could not allocate audio frame");

audioFrameDecoded->format = fileCodecContext->sample_fmt;
audioFrameDecoded->channel_layout = fileCodecContext->channel_layout;
audioFrameDecoded->channels = fileCodecContext->channels;
audioFrameDecoded->sample_rate = fileCodecContext->sample_rate;

audioFrameConverted = av_frame_alloc();
if (!audioFrameConverted)
        die("Could not allocate audio frame");

audioFrameConverted->nb_samples = audioCodecContext->frame_size;
audioFrameConverted->format = audioCodecContext->sample_fmt;
audioFrameConverted->channel_layout = audioCodecContext->channel_layout;
audioFrameConverted->channels = audioCodecContext->channels;
audioFrameConverted->sample_rate = audioCodecContext->sample_rate;

AVPacket inPacket;
av_init_packet(&inPacket);
inPacket.data = NULL;
inPacket.size = 0;

int frameFinished = 0;

while (av_read_frame(formatContext, &inPacket) >= 0) {

        if (inPacket.stream_index == streamId) {

                int len = avcodec_decode_audio4(fileCodecContext, audioFrameDecoded, &frameFinished, &inPacket);

                if (frameFinished) {

                        // Convert

                        uint8_t *convertedData=NULL;

                        if (av_samples_alloc(&convertedData,
                                             NULL,
                                             audioCodecContext->channels,
                                             audioFrameConverted->nb_samples,
                                             audioCodecContext->sample_fmt, 0) < 0)
                                die("Could not allocate samples");

                        int outSamples = swr_convert(swrContext,
                                                     &convertedData,
                                                     audioFrameConverted->nb_samples,
                                                     (const uint8_t **)audioFrameDecoded->data,
                                                     audioFrameDecoded->nb_samples);
                        if (outSamples < 0)
                                die("Could not convert");

                        size_t buffer_size = av_samples_get_buffer_size(NULL,
                                                                        audioCodecContext->channels,
                                                                        audioFrameConverted->nb_samples,
                                                                        audioCodecContext->sample_fmt,
                                                                        0);
                        if (buffer_size < 0)
                                die("Invalid buffer size");

                        if (avcodec_fill_audio_frame(audioFrameConverted,
                                                     audioCodecContext->channels,
                                                     audioCodecContext->sample_fmt,
                                                     convertedData,
                                                     buffer_size,
                                                     0) < 0)
                                die("Could not fill frame");

                        AVPacket outPacket;
                        av_init_packet(&outPacket);
                        outPacket.data = NULL;
                        outPacket.size = 0;

                        if (avcodec_encode_audio2(audioCodecContext, &outPacket, audioFrameConverted, &frameFinished) < 0)
                                die("Error encoding audio frame");

                        if (frameFinished) {
                                outPacket.stream_index = audioStream->index;

                                if (av_interleaved_write_frame(outContext, &outPacket) != 0)
                                        die("Error while writing audio frame");

                                av_free_packet(&outPacket);
                        }
                }
        }
}

av_frame_free(&audioFrameConverted);
av_frame_free(&audioFrameDecoded);
av_free_packet(&inPacket);

I have also tried setting appropriate pts values for outgoing frames, but that doesn't seem to affect the sound quality at all.

I'm also unsure how/if I should be allocating the converted data, can av_samples_alloc be used for this? What about avcodec_fill_audio_frame? Am I on the right track?

Any input is appreciated (I can also send the exported MP4s if necessary, if you want to hear the distortion).

Easterly

Note that many of the methods used in this code are now deprecated – Mich

Answer 1

if (avcodec_encode_audio2(audioCodecContext, &outPacket, audioFrameConverted, &frameFinished) < 0)
                die("Error encoding audio frame");

You seem to be assuming that the encoder will eat all submitted samples - it doesn't. It also doesn't cache them internally. It will eat a specific number of samples (AVCodecContext.frame_size), and the rest should be resubmitted in the next call to avcodec_encode_audio2().

[edit]

ok, so your edited code is better, but not there yet. You're still assuming the decoder will output at least frame_size samples for each call to avcodec_decode_audioN() (after resampling), which may not be the case. If that happens (and it does, for ogg), your avcodec_encode_audioN() call will encode an incomplete input buffer (because you say it's got frame_size samples, but it doesn't). Likewise, your code also doesn't deal with cases where the decoder outputs a number significantly bigger than frame_size (like 10*frame_size) expected by the encoder, in which case you'll get overruns - basically your 1:1 decode/encode mapping is the main source of your problem.

As a solution, consider the swrContext a FIFO, where you input all decoder samples, and loop over it until it's got less than frame_size samples left. I'll leave it up to you to learn how to deal with end-of-stream, because you'll need to flush cached samples out of the decoder (by calling avcodec_decode_audioN() with AVPacket where .data = NULL and .size = 0), flush the swrContext (by calling swr_context() until it returns 0) as well as flush the encoder (by feeding it NULL AVFrames until it returns AVPacket with .size = 0). Right now you'll probably get an output file where the end is slightly truncated. That shouldn't be hard to figure out.

This code works for me for m4a/ogg/mp3 to m4a/aac conversion:

#include "libswresample/swresample.h"
#include "libavcodec/avcodec.h"
#include "libavformat/avformat.h"
#include "libavutil/opt.h"

#include <stdio.h>
#include <stdlib.h>

static void die(char *str) {
    fprintf(stderr, "%s\n", str);
    exit(1);
}

static AVStream *add_audio_stream(AVFormatContext *oc, enum AVCodecID codec_id)
{
    AVCodecContext *c;
    AVCodec *encoder = avcodec_find_encoder(codec_id);
    AVStream *st = avformat_new_stream(oc, encoder);

    if (!st) die("av_new_stream");

    c = st->codec;
    c->codec_id = codec_id;
    c->codec_type = AVMEDIA_TYPE_AUDIO;

    /* put sample parameters */
    c->bit_rate = 64000;
    c->sample_rate = 44100;
    c->channels = 2;
    c->sample_fmt = encoder->sample_fmts[0];
    c->channel_layout = AV_CH_LAYOUT_STEREO;

    // some formats want stream headers to be separate
    if(oc->oformat->flags & AVFMT_GLOBALHEADER)
        c->flags |= CODEC_FLAG_GLOBAL_HEADER;

    return st;
}

static void open_audio(AVFormatContext *oc, AVStream *st)
{
    AVCodecContext *c = st->codec;
    AVCodec *codec;

    /* find the audio encoder */
    codec = avcodec_find_encoder(c->codec_id);
    if (!codec) die("avcodec_find_encoder");

    /* open it */
    AVDictionary *dict = NULL;
    av_dict_set(&dict, "strict", "+experimental", 0);
    int res = avcodec_open2(c, codec, &dict);
    if (res < 0) die("avcodec_open");
}

int main(int argc, char *argv[]) {
    av_register_all();

    if (argc != 3) {
        fprintf(stderr, "%s <in> <out>\n", argv[0]);
        exit(1);
    }

    // Allocate and init re-usable frames
    AVCodecContext *fileCodecContext, *audioCodecContext;
    AVFormatContext *formatContext, *outContext;
    AVStream *audioStream;
    SwrContext *swrContext;
    int streamId;

    // input file
    const char *file = argv[1];
    int res = avformat_open_input(&formatContext, file, NULL, NULL);
    if (res != 0) die("avformat_open_input");
    res = avformat_find_stream_info(formatContext, NULL);
    if (res < 0) die("avformat_find_stream_info");
    AVCodec *codec;
    res = av_find_best_stream(formatContext, AVMEDIA_TYPE_AUDIO, -1, -1, &codec, 0);
    if (res < 0) die("av_find_best_stream");
    streamId = res;
    fileCodecContext = avcodec_alloc_context3(codec);
    avcodec_copy_context(fileCodecContext, formatContext->streams[streamId]->codec);
    res = avcodec_open2(fileCodecContext, codec, NULL);
    if (res < 0) die("avcodec_open2");

    // output file
    const char *outfile = argv[2];
    AVOutputFormat *fmt = fmt = av_guess_format(NULL, outfile, NULL);
    if (!fmt) die("av_guess_format");
    outContext = avformat_alloc_context();
    outContext->oformat = fmt;
    audioStream = add_audio_stream(outContext, fmt->audio_codec);
    open_audio(outContext, audioStream);
    res = avio_open2(&outContext->pb, outfile, AVIO_FLAG_WRITE, NULL, NULL);
    if (res < 0) die("url_fopen");
    avformat_write_header(outContext, NULL);
    audioCodecContext = audioStream->codec;

    // resampling
    swrContext = swr_alloc();
    av_opt_set_channel_layout(swrContext, "in_channel_layout",  fileCodecContext->channel_layout, 0);
    av_opt_set_channel_layout(swrContext, "out_channel_layout", audioCodecContext->channel_layout, 0);
    av_opt_set_int(swrContext, "in_sample_rate", fileCodecContext->sample_rate, 0);
    av_opt_set_int(swrContext, "out_sample_rate", audioCodecContext->sample_rate, 0);
    av_opt_set_sample_fmt(swrContext, "in_sample_fmt", fileCodecContext->sample_fmt, 0);
    av_opt_set_sample_fmt(swrContext, "out_sample_fmt", audioCodecContext->sample_fmt, 0);
    res = swr_init(swrContext);
    if (res < 0) die("swr_init");

    AVFrame *audioFrameDecoded = av_frame_alloc();
    if (!audioFrameDecoded)
        die("Could not allocate audio frame");

    audioFrameDecoded->format = fileCodecContext->sample_fmt;
    audioFrameDecoded->channel_layout = fileCodecContext->channel_layout;
    audioFrameDecoded->channels = fileCodecContext->channels;
    audioFrameDecoded->sample_rate = fileCodecContext->sample_rate;

    AVFrame *audioFrameConverted = av_frame_alloc();
    if (!audioFrameConverted) die("Could not allocate audio frame");

    audioFrameConverted->nb_samples = audioCodecContext->frame_size;
    audioFrameConverted->format = audioCodecContext->sample_fmt;
    audioFrameConverted->channel_layout = audioCodecContext->channel_layout;
    audioFrameConverted->channels = audioCodecContext->channels;
    audioFrameConverted->sample_rate = audioCodecContext->sample_rate;

    AVPacket inPacket;
    av_init_packet(&inPacket);
    inPacket.data = NULL;
    inPacket.size = 0;

    int frameFinished = 0;

    while (av_read_frame(formatContext, &inPacket) >= 0) {
        if (inPacket.stream_index == streamId) {
            int len = avcodec_decode_audio4(fileCodecContext, audioFrameDecoded, &frameFinished, &inPacket);

            if (frameFinished) {

                // Convert

                uint8_t *convertedData=NULL;

                if (av_samples_alloc(&convertedData,
                             NULL,
                             audioCodecContext->channels,
                             audioFrameConverted->nb_samples,
                             audioCodecContext->sample_fmt, 0) < 0)
                    die("Could not allocate samples");

                int outSamples = swr_convert(swrContext, NULL, 0,
                             //&convertedData,
                             //audioFrameConverted->nb_samples,
                             (const uint8_t **)audioFrameDecoded->data,
                             audioFrameDecoded->nb_samples);
                if (outSamples < 0) die("Could not convert");

                for (;;) {
                     outSamples = swr_get_out_samples(swrContext, 0);
                     if (outSamples < audioCodecContext->frame_size * audioCodecContext->channels) break; // see comments, thanks to @dajuric for fixing this

                     outSamples = swr_convert(swrContext,
                                              &convertedData,
                                              audioFrameConverted->nb_samples, NULL, 0);

                     size_t buffer_size = av_samples_get_buffer_size(NULL,
                                    audioCodecContext->channels,
                                    audioFrameConverted->nb_samples,
                                    audioCodecContext->sample_fmt,
                                    0);
                    if (buffer_size < 0) die("Invalid buffer size");

                    if (avcodec_fill_audio_frame(audioFrameConverted,
                             audioCodecContext->channels,
                             audioCodecContext->sample_fmt,
                             convertedData,
                             buffer_size,
                             0) < 0)
                        die("Could not fill frame");

                    AVPacket outPacket;
                    av_init_packet(&outPacket);
                    outPacket.data = NULL;
                    outPacket.size = 0;

                    if (avcodec_encode_audio2(audioCodecContext, &outPacket, audioFrameConverted, &frameFinished) < 0)
                        die("Error encoding audio frame");

                    if (frameFinished) {
                        outPacket.stream_index = audioStream->index;

                        if (av_interleaved_write_frame(outContext, &outPacket) != 0)
                            die("Error while writing audio frame");

                        av_free_packet(&outPacket);
                    }
                }
            }
        }
    }

    swr_close(swrContext);
    swr_free(&swrContext);
    av_frame_free(&audioFrameConverted);
    av_frame_free(&audioFrameDecoded);
    av_free_packet(&inPacket);
    av_write_trailer(outContext);
    avio_close(outContext->pb);
    avcodec_close(fileCodecContext);
    avcodec_free_context(&fileCodecContext);
    avformat_close_input(&formatContext);

    return 0;
}

Fleecy

But I'm setting the frame size to the amount of samples, as you can see in my code. Shouldn't it eat up everything then? Or do I not understand something? Can you show how I can get it working? – Easterly

You cannot set frame_size, it's informationally provided to you by the codec. Consider it read-only. To be sure, check the return value of avcodec_encode_audioN() to see how many samples were really encoded. You'll notice it doesn't match the input sample count. – Fleecy

It returns 0 for me, which according to the documentation means success. Should I be checking with outPacket.size? None of the examples I've found do that, they simply check with frameFinished, which always gives 1 in my case. – Easterly

oh, yes, you're right, it returns 0. But you can't assign frame_size, and should assume that it consumed frame_size samples. – Fleecy

I've changed the code (no longer changing frame_size), but I'm still having the same issues: pastebin.com/ENzeaHpU Got any more hints? – Easterly

See my edit, it now includes a working example for ogg/m4a/mp3 conversion based on your code with some fixes. – Fleecy

Works great, thanks. I'll look into that flushing, enjoy your rep (in 15 hours) :) – Easterly

Just a quick question: If I want several sound files in the mp4, is it wise to use one audio stream for each (since the pts needs to be increasing)? – Easterly

I'm not sure what you mean, are you trying to concatenate two (different) songs in the same file? Or have two audio tracks (like different languages) of the same content? In the first case, I think you're just trying to concatenate the two songs, and one stream should be fine. In the second, you obviously need two streams. – Fleecy

An example in my case is a background song and some sound effects playing over it. The sound effects then need to play at the same time as the song. I'll be using two streams then (my other idea was to write a bit of song, a bit of sound effect, a bit of song... etc. on the same stream). – Easterly

oh, I see. You want a mixer. You basically blend the two sound sample arrays together with a particular weight. There's a ffmpeg filter for audio mixing, but it's easy to write your own. I wouldn't write two streams for this, since most players won't be able to mix two tracks together, unless you're going to use a custom player. – Fleecy

Well, I've successfully generated an array of AvPacket for each sound file, I guess that's what you mean with "sound sample array"? Or should I be storing them in a different fashion? – Easterly

Let us continue this discussion in chat. – Fleecy

@RonaldS.Bultje Thanks for the sample. I had the same problem. The code is working great, but the output audio contains some distortions (constant background tapping every ~0.5s). Do you know what it maybe the cause ? Thanks. – Eightieth

I fixed that using 'if (outSamples < audioCodecContext->frame_size * audioCodecContext->channels)' instead 'if (outSamples < fileCodecContext->frame_size)'. – Eightieth

Answer 2

I wanted to include a couple things I found when I was working with the above code. I had one file get stuck in an infinite loop. The reason is the file had a sample rate of 48000 and the code changes it to a 44100. This caused it to always have extra outSamples. swr_convert & would not grab them. So I ended up changing add_audio_stream to match the input streams sample rate.

        c->sample_rate = fileCodecContext->sample_rate;

Also I had to produce wav files as my output. And it had a framesize of 0. so I just chose a number after a few tests I went with 32. I noticed if I went too big (ex 128) I would get audio glitches.

 if (audioFrameConverted->nb_samples <= 0) audioFrameConverted->nb_samples = 32; //wav files have a 0 

Changed the if statement that breaks out of the loop to check nb_samples if frame_size is 0.

                            if ((outSamples < audioCodecContext->frame_size * audioCodecContext->channels) || audioCodecContext->frame_size==0 && (outSamples < audioFrameConverted->nb_samples * audioCodecContext->channels)) break; // see comments, thanks to @dajuric for fixing this

There was also a glitch when I was testing outputting to ogg files where the timestamp data was missing so the file wouldn't play correctly in vlc. There were a few lines I added that helped with that.

        out_audioStream->time_base = in_audioStream->time_base; // entered before avio_open.
                        outPacket.dts = audioFrameDecoded->pkt_dts;//rest after avcodec_encode_audio2
                        outPacket.pts = audioFrameDecoded->pkt_pts;
                        av_packet_rescale_ts(&outPacket, in_audioStream->time_base, out_audioStream->time_base);

Variables might be a little different I converted the code to c#. Thought this might help someone.

Discriminant

Answer 3

Actually swr_convert won't work for that, try to use swr_convert_frame instead.

Jopa