ffmpeg.c what are pts and dts ? what does this code block do in ffmpeg.c?

Question

• In simple terms what are pts and dts values?
• Why are they important while transcoding [decode-encode] videos ?

What does this code bit do in ffmpeg.c , what is its purpose?

01562    ist->next_pts = ist->pts = picture.best_effort_timestamp;
01563    if (ist->st->codec->time_base.num != 0) {
01564        int ticks= ist->st->parser ? ist->st->parser->repeat_pict+1 : ist->st->codec->ticks_per_frame;
01565        ist->next_pts += ((int64_t)AV_TIME_BASE *
01566                         ist->st->codec->time_base.num * ticks) /
01567                         ist->st->codec->time_base.den;
01568    }

Audsley

Answer 1

Those are the decoding time stamp (DTS) and presentation time stamp (PTS). You can find an explanation here inside a tutorial.

So let's say we had a movie, and the frames were displayed like: I B B P. Now, we need to know the information in P before we can display either B frame. Because of this, the frames might be stored like this: I P B B. This is why we have a decoding timestamp and a presentation timestamp on each frame. The decoding timestamp tells us when we need to decode something, and the presentation time stamp tells us when we need to display something. So, in this case, our stream might look like this:

   PTS: 1 4 2 3
   DTS: 1 2 3 4
Stream: I P B B

Generally the PTS and DTS will only differ when the stream we are playing has B frames in it.

Indian

Can you please epand on what a B frame is, and what a P frame is? – Logy

@Logy that is actually explained fairly well here en.wikipedia.org/wiki/Video_compression_picture_types – Indian

I still don't understand. The stream has to be displayed as I B B P, But since P ought to come before B, we store it as I P B B. Hence we decode it in the order of 1 2 3 4, that makes sense. But shouldn't we present it in the order of 1 3 4 2? Why is it 1 4 2 3? When we have consecutive B frames, do these B frames take advantage of the I and P frames alone, or do they use the B frames next to each other too? – Logy

PTS of 1 4 2 3 makes it sound like we're displaying the frames in the order of I B(2) P B(1). Correct me if I'm wrong... – Logy

Quick follow up: As a rule, is the decoding timestamp always in ascending order? It appears so. – Logy

No, you're displaying in IBBP order. But because the first B relies on the I before it and the B after it, and subsequently the second B frame relies on the final P frame, to be able to display the first B frame you'll already need the information of the P frame. And that's why the stream is IPBB. – Indian

I'm sorry, I'm lost. Let's say the frames need to be displayed in 1 B(1) B(2) P. B(1) depends on 1 and B(2). I assume B(2) depends only on P. Therefore we should store it as I P B(2) B(1) right? Decoding is in this order, but then presentation should be I B(1) B(2) P, that is, 1 4 3 2, right? – Logy

B frames are bidirectional and depend on information before and after it. So B2 depends on both B1 and P. As the text states "we need to know the information in P before we can display either B frame". And therefore the final order you need the data in is IPBB. At least that is how I understand it. More in depth information is probably best requested from someone who is more up to date than I am. – Indian

Well if B1 depends on I and B2, and B2 depends on B1 and P, it sounds like a deadlock to me. Anyway, thanks. – Logy

In case @Logy or anyone else is still wondering, the frames are stored in dependency order, so for an AVC stream sent as I-P1-B1-B2-..., B2 may rely on B1 but not the other way around, regardless of their display order. – Annetteannex

@Annetteannex In that case, DTS is guaranteed to appear in ascending order, right? – Handy

@Handy Should; ffmpeg complains when it doesn't, eg “Non-monotonous DTS in output stream” (in this case they used -c copy). – Annetteannex

Note for confused readers like me: I, P, and B are not arbitrary frame names they stand for Intra-coded frame, Predicted frame, Bi-directional predicted frame, respectively. – Serilda

Answer 2

I found it clearer to find information in the answer page rather than a link.

The presentation time (PTS) is the correct one. The frames can be stored out-of-order in the file, and the data may need to be read or written out-of-order to be reconstructed. The DTS time stores this other codec-related ordering.

The pasted code snippet retrieves the PTS of a frame and calculates its duration so as to compute the PTS of the next one.

Pilar

Answer 3

B frames are predicted from I and P frames. B frames usually have more errors compared to I and P and hence are not recommended for prediction, though they might be closer in time. There are algorithms in which B is used for prediction but it is from a past B frame and not future B frames.

So in a sequence of I P B1 B2, Decode order is I P B1 B2 and Display order is I B1 B2 P. P is predicted from I, B1 from both I and P, B2 again from I and P.

Spline