What does it mean "invalid NAL unit size" for h.264 decoder?

Question

I want to transmux a .mkv file to .mp4 using Libav but when I try to decode the video h.264 stream there is a malfunction in my code

Invalid NAL unit size 21274662>141

Error splitting the input into NAL units

The stream seems to contain AVCC extradata with Annex B formatted data which is invalid. no frame!

Could not send paket for decoding ("error invalid data when processing input")

A relevant section of code is available below.

while(!(ret = av_read_frame(ifmt_ctx, &input_packet))&&(ret>=0)){

         if ((ret = avcodec_send_packet(avctx, &input_packet)) < 0) {
            fprintf(stderr, "Could not send packet for decoding (error '%s')\n",get_error_text(ret));
            return ret;
        }

        ret = avcodec_receive_frame(avctx, iframe);
        if (ret == AVERROR(EAGAIN)) {
            goto read_another_frame;
            /* If the end of the input file is reached, stop decoding. */
        } else if (ret == AVERROR_EOF) {
            break;
        } else if (ret < 0) {
            fprintf(stderr, "Could not decode frame (error '%s')\n",get_error_text(ret));
            break;
        }
        // Default case: encode data
         else {

        }

I use mainly the new API (send / receive packet /frame) and the confusion exists because it seems like h.264 needs a special implementation. I'm looking forward to any idea from where I should start the debugging.

Glazunov

Answer 1

It means the ES format is not compatible with the container. Read this: Possible Locations for Sequence/Picture Parameter Set(s) for H.264 Stream

Bookstore

Is there any possibility to use only the Libav h.264 decoder to decode the data? – Glazunov

As opposed to what? If you just want to fill an AVPacket with an Access Unit and call decode, then yes. If you want to bypass that abstraction, still yes, but much more difficult. Either way, you must unwrap the ES from the container first. – Bookstore

Thank you! With ES data do you mean avctx->extradata_size (in my case 35) and avctx->extradata (in my case 0x16125e0 "\001M@\037\377")? Sorry for the dumb question... I couldn't find an appropriate documentation – Glazunov

No, I mean the raw elementary stream (decontainerized) – Bookstore

I read that documentation. But I still dont know how to avoid the error in the question above. – Quickie

Answer 2

I think this is because you're not checking if the packet is from a video stream. In other words your code sends all packets from all streams to the h.264 codec.

In this case, a way to tackle the problem would be to add a simple condition that skips non video steam packets as such:

if (input_packet->stream_index != video_stream->index) continue;

Assuming video_stream is the first video stream encoutered in the format context stream array ifmt_ctx->streams.

Sinfonia