Why isn't RDTSC a serializing instruction?

Question

The Intel manuals for the RDTSC instruction warn that out of order execution can change when RDTSC is actually executed, so they recommend inserting a CPUID instruction in front of it because CPUID will serialize the instruction stream (CPUID is never executed out of order). My question is simple: if they had the ability to make instructions serializing, why didn't they make RDTSC serializing? The entire point of it appears to be to get cycle accurate timings. Is there a situation under which you would not want to precede it with a serializing instruction?

Newer Intel CPUs have a separate RDTSCP instruction that is serializing. Intel opted to introduce a separate instruction rather than change the behavior of RDTSC, which suggests to me that there has to be some situation where a potentially out of order timing is what you want. What is it?

Farnesol

Note the question/assertion posed at the end: "..there has to be some situation where a potentially out of order timing is what you want. What is it?" – Hagride

RDTSCP isn't serializing the way CPUID is. It's only a one-way barrier for instructions, and doesn't stop later instructions from executing before it (and other earlier instructions). – Noonday

IMHO, although it is commonly recognized that rdtsc is not a serializing instruction, they serialize themselves. For back-to-back rdtscs, the latter never carries out before the former, and this guarantees any time gap we measured by rdtsc is always positive. – Stomy

@Stomy Is this documented somewhere? Since rdtsc doesn't consume any registers I'd expect that the normal dependency-chain ordering enforcement in the CPU would not apply. I've noticed this is a problem on the compiler level -- compilers will reorder rdtsc even w/ memory barriers b/c there's no dependency chain or memory access. – Farnesol

Actually come to think of it I kinda remember reading somewhere that consensus was this doesn't happen b/c there is no advantage to doing it. But it'd be better if there was a source. – Farnesol

Answer 1

If you are trying to use rdtsc to see if a branch mispredicts, the non-serializing version is what you want.

//math here
rdtsc
branch if zero to done
//do some work that always takes 1 cycle
done: rdtsc

If the branch is predicted correctly, the delta will be small (maybe even negative?). If the branch is mispredicted, the delta will be large.

With the serializing version, the branch condition will be resolved because the first rdtsc waits for the math to finish.

Wormeaten

Very interesting. You mean, assuming the branch isn't taken (since then the second rdtsc wouldn't run since we'd jump somewhere), and we want to check if it not being taken is predicted correctly, the second rdtsc will execute at the same time as the branch check (since the prediction is so the processor can pipeline), otherwise it won't be and the time will be larger. This assumes the CPU never speculatively executes both possibilities, but that was certainly true at the time (and maybe still is?). – Farnesol

I changed up the example to make the second rdtsc always execute. – Wormeaten

I don't think this is correct because rdtsc was not really designed to determine whether a branch was predicted correctly. Although the technique you described you may work, but that's not by design. The purpose of rdtsc is to provide a low-overhead, high-resolution method for measuring time of a region of code. – Substantialize

Answer 2

The time stamp counter was introduced on the Pentium microarchitecture. Out-of-order execution didn't show up until the Pentium Pro. Intel could have made rdtsc serializing (architecturally or internally), but it seems that they decided to keep it non-serializing, which is OK for general-purpose time measurements, and leave it up to the programmer to add serializing instructions if necessary. This is good for reducing the overhead of the measurement.

That's actually confirmed in the document you provide, with the following comment about Pentium and Pentium/MMX (in 4.2, slightly paraphrased):

All of the rules and code samples described in section 4.1 (Pentium Pro and Pentium II) also apply to the Pentium and Pentium/MMX. The only difference is, the CPUID instruction is not necessary for serialization.

And, from Wikipedia:

The Time Stamp Counter is a 64-bit register present on all x86 processors since the Pentium.

: : :

Starting with the Pentium Pro, Intel processors have supported out-of-order execution, where instructions are not necessarily performed in the order they appear in the executable. This can cause RDTSC to be executed later than expected, producing a misleading cycle count.

---

One of the two uses of RDTSCP is to give you the processor ID in addition to the time stamp information (it's right there in the name Read Time-Stamp Counter *AND* Processor ID), which is useful on systems with unsynced TSCs across cores or sockets (See: How to get the CPU cycle count in x86_64 from C++?). The additional serialization properties of rdtscp makes it more convenient at the end of the region of interest (See: Is there any difference in between (rdtsc + lfence + rdtsc) and (rdtsc + rdtscp) in measuring execution time?).

Lechner

I disagree that the document confirms it. Prior to out of order execution, there was no concept of a serializing instruction since instructions were always serial. So when they introduced out of order execution if they had made RTDSC a serializing instruction there wouldn't have been any observable change in its behavior from earlier processors. – Farnesol

@Joseph, I think you misunderstand what I'm saying it confirms. I'm not stating that what they did was correct, just that the timelines for timestamp counters and OOO execution were confirmed by that document. In fact, I believe what they did was wrong because they regressed the behaviour of RDTSC - it worked on the earlier processor and not on the latter one. I suspect someone didn't take into account OOOE until it was too late but that's just supposition on my part. – Lechner

Ah, yes, I agree then, but my goal is to figure out whether it's an error on their part or something deliberate :) – Farnesol

Intel? Make a mistake? Not a chance. As sure as 4195835 divided by 3145727 equals 1.333739068902037589, they're infallible. Foof, I'm stunned that you would think this possible :-) – Lechner

Answer 3

why didn't they make RDTSC serializing? The entire point of it appears to be to get cycle accurate timings

Well, most of the time it's to get high-resolution timestamps. At least some of the time, these timestamps are used for performance metrics. Making the intruction serializing would likely require a pipeline flush, which can be very expensive for CPU-bound applications.

Intel opted to introduce a separate instruction rather than change the behavior of RDTSC, which suggests to me that there has to be some situation where a potentially out of order timing is what you want.

Changing the behavior is almost always undesirable. Intel's customers would be disappointed to find out that RDTSC does something different on newer parts.

Thorax

Actually, they'd be used to that. The behaviour changed when switching from Pentium to Pentium Pro - it stopped giving useful results without serialising :-) But you're dead right about it being undesirable. – Lechner

Making the instruction serializing would require a pipeline flush, but it seems that it's also necessary for your high resolution timestamps to be usable, thus my confusion. The purpose of getting the timestamps is to compare them or get the difference between them -- if you allow the instruction to be pipelined then you're not always measuring the same thing, right? – Farnesol

@JosephGarvin: In a pipelined CPU, the time required to execute a piece of code often isn't a clearly-defined number. Flushing the cache before taking measurements will give cause the measurements to yield a well-defined number, but that number will have less relationship to real-world performance than would a number measured without the cache flushing. – Hetti

@JosephGarvin and Brian: A serializing rdtsc would not affect the resolution (it would still count at the TSC frequency), but it would increase the overhead of the measurement, which could be significant in some cases compared to the time of the region. – Substantialize

Answer 4

As paxdiably explains, RDTSC predates the concept of "serializing" instructions because it was implemented on an in-order CPU. Adding that behavior later would change the memory access behavior of code using it, and thus be incompatible for some purposes.

Instead, more recent CPUs have a related RDTSCP instruction that is defined as serializing (actually stronger: it promises to wait until all instructions issued before it have completed, not just that memory accesses have been done), for exactly this reason. Use that if you are running on modern CPUs.

Benioff

"Adding that behavior later would change the memory access behavior of code using it, and thus be incompatible for some purposes." Except that I don't think it would. If they had had an out of order CPU before with rdtsc, then yes, making it serializing in later CPUs would be a behavior change. But when they introduced out of order execution, there couldn't be any older programs that depended on rdtsc being serializing because serializing as a concept only exists when you have out of order execution. So my thinking right now is that it was an oversight by Intel. – Farnesol

rdtscp isn't serializing the way CPUID is. It's only a one-way barrier for instructions, and doesn't stop later instructions from passing it and other earlier instructions. – Noonday

"The RDTSCP instruction is not a serializing instruction, but it does wait until all previous instructions have executed and all previous loads are globally visible. But it does not wait for previous stores to be globally visible, and subsequent instructions may begin execution before the read operation is performed." – Murderous

A "serializing instruction" in x86 terminology means it drains the ROB and the store buffer, and not letting any later instructions execute ahead of it. Like cpuid. rdtscp is much weaker than this, only draining the ROB but not the store buffer. It like lfence; rdtsc, not lfence;rdtsc;lfence which you sometimes actually want. You normally wouldn't want it for you to wait for the store buffer to drain; you can wait for that with mfence. – Noonday

Update: Serializing also means discarding older instructions fetched, in case of cross-modifying code. Is there a cheaper serializing instruction than cpuid? . even mfence ; lfence wouldn't be safe for that. Anyway, as I said, rdtscp doesn't wait for older stores to commit from the store buffer to L1d cache the way mfence; rdtscp would. – Noonday