Can branch prediction cause illegal instruction?

Question

In the following pseudo-code:

if (rdtscp supported by hardware) {
    Invoke "rdtscp" instruction
} else {
    Invoke "rdtsc" instruction
}

Let's say the CPU does not support the rdtscp instruction and so we fallback to the else statement.

If CPU mispredicts the branch, is it possible for the instruction pipeline to try to execute rdtscp and throw an Illgal Instruction error?

Soutane

It should not be able to. Speculative execution is required to behave as if it wasn't speculative. So all faults (access fault, alignment, illegal instruction etc...) need to be suppressed until you exit speculative execution mode. – Pyrogenous

That's a good argument. I would love some pointers to a reliable source though (e.g. intel manual) – Soutane

Branch prediction just fills the caches? It doesn't actually execute, isn't it? – Stinkweed

The golden rule of out-of-order/speculative execution and memory re-ordering is "don't break single-threaded code". The CPU can do anything it wants under the hood, as long as correct code always runs correctly. – Gnomon

Answer 1

It is explicitly documented for the #UD trap (Invalid Opcode Execution) in the Intel Processor Manuals, Volume 3A, chapter 6.15:

In Intel 64 and IA-32 processors that implement out-of-order execution microarchitectures, this exception is not generated until an attempt is made to retire the result of executing an invalid instruction; that is, decoding and speculatively attempting to execute an invalid opcode does not generate this exception. Likewise, in the Pentium processor and earlier IA-32 processors, this exception is not generated as the result of prefetching and preliminary decoding of an invalid instruction.

Furfuran

Answer 2

Instruction traps, such as the "Illegal instruction" trap, take effect when the instruction is executed, not before. Programs often depend on what state the program is at the moment the trap is thrown, so executing them early would be a serious bug in the architecture.

EDIT: okay, okay. From the Intel software developer's manual, volume 3A:

The ability of a P6 family processor to speculatively execute instructions does not affect the taking of interrupts by the processor. Interrupts are taken at instruction boundaries located during the retirement phase of instruction execution; so they are always taken in the “in-order” instruction stream.

Standard

Answer 3

It depends on how you define "cause". Can it have some observable effect? yes.

As Hans and Sneftel pointed out - exceptions, interrupts, traps, etc.. are all taking place at retirement, to which a wrong-path code would never reach. So a mispredicted branch can't cause a commit / retirement of wrong path instruction, including any illegal opcodes they may attempt to execute.

However, it is possible for a mispredicted code path to affect the micro architectural state in more subtle ways, you may have loads performed and cached, page walks done, and various other micro architectural events - in short, all the hard work your CPU does while you carelessly go and mispredict some branch (which is quite a lot in OOO CPUs). I highly doubt that any CPU out there is going to go and fish all this stuff out of your caches/TLBs, so take into account that these things can later have an effect on the performance of your program (for good or for worse), just not on its functional behavior.

It's probably not worthwhile if you're in the business of optimizing performance (and probably too complicated to make any use of), but it may be relevant to consider if you're in the business of securing your code - some hackers may have ways to sniff such data from your caches.

Vibraharp

This is a good point. For my particular case of illegal instruction though, the CPU hopefully wouldn't be able to do much damage! Except perhaps for loading it into the instruction cache. – Soutane

@mtoossi, funny to observe this 10 years later, this preceded Spectre/Meltdown by about 4 years :) apparently there is a lot of damage that can be done with this kind of bogus code – Vibraharp