How does the instruction decoder differentiate between EVEX prefix and BOUND opcode in 32-bit mode?

Question

In 32-bit mode Intel solves the VEX prefix vs LDS/LES conflict by inverting the high bits of register extension, because the mod field of ModRM byte can't be 11b

The VEX prefix's initial-byte values, C4h and C5h, are the same as the opcodes of the LDS and LES instructions. These instructions are not supported in 64-bit mode. To resolve the ambiguity while in 32-bit mode, VEX's specification exploits the fact that a legal LDS or LES's ModRM byte can not be of the form 11xxxxxx (which would specify a register operand). Various bit-fields in the VEX prefix's second byte are inverted to ensure that the byte is always of this form in 32-bit mode.

https://en.wikipedia.org/wiki/VEX_prefix#Technical_description

However in EVEX the R and X bits aren't inverted, which results in mod=00b, which also indicates a memory operand in BOUND instruction

Four bits R, X, B, and W from the REX prefix. W expands the operand size to 64 bits or serves as an additional opcode, R expands reg, B expands r/m or reg, and X and B expand index and base in the SIB byte. Comparing to the VEX prefix, RXB are provided in non-inversed forms, just like in the REX prefix.

https://en.wikipedia.org/wiki/EVEX_prefix

So how can they decode that instruction overlap cleanly?

---

I checked the Intel manual and they seem to mention about the inversion of the bits only in VEX but not EVEX.

OTOH the table in sandpile says that those RXB bits in EVEX should be inverted as well.

Which of them is correct?

Juliojulis

That's funny, if I assemble some instructions with an EVEX, they come out with RXB inverted, and that would of course work with bound. But the manual doesn't appear to say to do that.. – Weinhardt

The R & X bits aren't used in 32-bit mode, so presumably these bits would have to be set to 1 in order to avoid creating a valid BOUND instruction. – Cycloid

@RossRidge even in 64 bit mode, they are inverted. At least nasm thinks so, would have to test on an actual cpu. – Paramo

Answer 1

The trick with inverted bits works because of two facts:

1. In the 32-bit mode, only 8 of 32 ZMM registers are available, therefore EVEX.R' bit never flips, and thus never aliases with allowed BOUND encodings.
2. In the 64-bit mode, there is no BOUND instruction recognized so the EVEX bits can take any value.

An excerpt from a patent on AVX clarifies this:

[0111] REX′ field —this is the first part of the REX′ field and is the EVEX.R′ bit field (EVEX Byte 1, bit [4]—R′) that is used to encode either the upper 16 or lower 16 of the extended 32 register set. In one embodiment of the invention, this bit, along with others as indicated below, is stored in bit inverted format to distinguish (in the well-known x86 32-bit mode) from the BOUND instruction, whose real opcode byte is 62, but does not accept in the MOD R/M field (described below) the value of 11 in the MOD field; alternative embodiments of the invention do not store this and the other indicated bits below in the inverted format. A value of 1 is used to encode the lower 16 registers. In other words, R′Rrrr is formed by combining EVEX.R′, EVEX.R, and the other RRR from other fields.

However, the Intel SDM is unclear on this fact. I've looked through the SDM and indeed, in the EVEX section there are no traces of mentioning of the complement meaning of EVEX encodings. One must deduce it somehow from the fact that EVEX is an extension of VEX, and for the latter there is a statement of inverted meaning (volume 2A, section 2.3.5, the first bullet):

This field is encoded using 1’s complement form (inverted form), i.e. XMM0/YMM0/R0 is encoded as 1111B, XMM15/YMM15/R15 is encoded as 0000B.

Epistemic

you have misread my question. The Intel manual and wiki article didn't say that the first 2 bits are inverted, so it'll clash with BOUND encoding. That's what I'm wondering about. Those are R and X, not R'. Moreover you only have access to 8 ZMM registers in 32-bit mode, not 16 – Juliojulis

That same patent also says "The EVEX.R, EVEX.X, and EVEX.B bit fields provide the same functionality as the corresponding VEX bit fields, and are encoded using 1s complement form, i.e. ZMM0 is encoded as 1111B, ZMM15 is encoded as 0000B. Other fields of the instructions encode the lower three bits of the register indexes as is known in the art (rrr, xxx, and bbb), so that Rrrr, Xxxx, and Bbbb may be formed by adding EVEX.R, EVEX.X, and EVEX.B." – Weinhardt

@LưuVĩnhPhúc Thanks, I was not sure whether it was 8 or 16. I've looked through the SDM and indeed, in the EVEX section there are no traces of mentioning of the complement meaning of EVEX encodings. One must deduce it somehow from the fact that EVEX is an extension of VEX, and for the latter there is a statement of inverted meaning (volume 2A, section 2.3.5, the first bullet): "This field is encoded using 1’s complement form (inverted form), i.e. XMM0/YMM0/R0 is encoded as 1111B, XMM15/YMM15/R15 is encoded as 0000B." – Epistemic