order for encoding x86 instruction prefix bytes

Question

I know that x86 instructions can have a maximum of 4 bytes of prefixes, e.g Lock, rep, segment overrides etc.

Is there any particular order in which they should appear, in case multiple prefixes are used?

Mia

Answer 1

the order can be found in volume 2A of the Intel Software Developer's Manual.

In a nutshell:

• the F2 and F3 prefixes cancel each other out. The one that comes later has precedence.
• the 66 prefix is ignored if either F2 or F3 are used (as mandatory prefixes in a long instruction). This of course doesn't apply to rep movsw where both those prefixes are simply prefixes, not part of the opcode.
• the REX escape may not be followed by any other prefixes.
• the VEX escape may not be preceded by REX, 66, F2 or F3

for the rest, the order shouldn't matter.

Dearly

66 is ignored if F2 is provided? WTF? Can't you do rep movsw in 32 bit mode? – Hest

The difference between MOVSB and MOVSW is the opcode, not the 66 prefix. MOVSB is A4, and MOVSW is A5. – Dearly

In 32 bit and 64 bit mode, the opcode a5 indicates movsl (resp. movsd in Intel syntax) because the default operand size is 32 bit. So to get movsw, you need to use a 66 prefix. Thus, rep movsw is 66 f3 a5 which you say is invalid. This is strange. – Hest

Note that I cannot find your second rule in the Intel manuals, so let me tentatively downvote this as I believe it is incorrect. – Hest

@Hest if both F2 and F3 are used with an SSE instruction, then the last one "wins", and 66 is ignored if other prefixes are present – Slice

@LưuVĩnhPhúc movsw is not an SSE instruction, so the answer should be updated to make clear, that this only applies to SSE instructions. And again, the answer you link to does not cite an authoritative source and neither does the reddit comment it cites. Could be just implementation defined behaviour. – Hest

@NathanFellman: The issue isn't MOVSB, it's MOVSD vs. MOVSW. 8-bit operand size uses a separate opcode, 16/32/64 use prefixes with 32 being the default in 32/64-bit mode. felixcloutier.com/x86/movs:movsb:movsw:movsd:movsq shows that A5 is the opcode for movsw and movsd, but we know that both of those instructions are usable with REP prefixes in any mode, using a 66 operand-size prefix when necessary. – Sickler

Answer 2

Quote from Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 2A: Instruction Set Reference, A-M

Instruction prefixes are divided into four groups, each with a set of allowable prefix codes. For each instruction, it is only useful to include up to one prefix code from each of the four groups (Groups 1, 2, 3, 4). Groups 1 through 4 may be placed in any order relative to each other.

Kaka

Answer 3

The architecture volume of the intel developer manuals details the layout at lenght, however, from what I remember last time I read it, the order for most didn't matter, except the REX/REX.W prefix which must occupy the slot closest to the start of the actual instruction bytes (aka it takes the slot most on the right)

Darkish