What does the PCOMMIT instruction do?
Asked Answered
A

1

9

In the Intel ISA extension manual the description for pcommit is a bit cryptic:

The PCOMMIT instruction causes certain store-to-memory operations to persistent memory ranges to become persistent (power failure protected). Specifically, PCOMMIT applies to those stores that have been accepted to memory.
[...]
If PCOMMIT is executed after a store to a persistent memory range is accepted to memory, the store becomes persistent when the PCOMMIT becomes globally visible.
[...]
The data in a store to persistent memory becomes persistent (durable) only after it has either been written to the targeted non-volatile device, or to some intermediate power-fail protected storage/buffer.

It names concepts like persistent memory ranges, stores accepted to memory, stores becoming persistent and non-volatile device1.

What is the exact context?


1 This cannot be the classical NV devices like the NOR Flash ROM or NVMe devices (read: new SSDs) as they are behind a variable number of bridges, including subtractive decoding ones, that the CPU has no control over.

Annapolis answered 26/1, 2017 at 16:22 Comment(0)
A
12

First of all pcommit has been deprecated before even shipping to an actual CPU.
Most of this answer is based on the content of the link above.


Intel, in conjunction with Micron, developed a new form of Non-volatile memory (NVM) called 3D XPoint (from its internal structure).
An actual implementation, as a disk cache, is already available and Intel started to prepare for a wider adoption of its NVM technology a while ago.

Particularly Intel imagined that some of the DIMMs could contain a portion made with 3D XPoint technology and thus constitute a non-volatile device.

This would make one or more memory ranges persistent, the collection of these persistent ranges is called the persistent domain.
One of the main features of the persistent domain is its ability to be power-fail safe.

When a store is made it goes through:

  • The store buffer.
    The store is completed/visible locally, but not globally.
    The store buffer can be flushed with different instructions (e.g. sfence).
  • The cache hierarchy.
    The store is globally visible (the cache coherence protocol ensure this).
    The cache can be flushed with different instructions (e.g. clflush, clflushopt, clwb, et al).
  • The memory controller Write Pending Queue (WPQ).
    The store is accepted to memory but it is not written to the DIMMs yet.
    The WPQ can be flushed through specific PCIe configuration registers of the memory controller or with pcommit.
  • The memory.
    The store is committed/written in the memory.

At what point of the data path above the store is in the persistent domain and thus will not be lost in case of power failure?
Some memory controller has a feature called Asynchronous DRAM Refresh that ensures that even in the case of power-loss the WPQ is flushed correctly (thanks to a battery for example).
For these platforms, the persistent domain starts at the WPQ.

Intel, however, was concerned that not all platforms would have had the ADR feature and created the pcommit instruction as a way to be sure that stores entered the persistence domain (pcommit is executable in user mode).

This is how a store was intended to be made persistent

mov [X], rax     ;Store

;Here the store has started moving to the store buffer

clwb [X]

;Here the store has moved to the cache (CLWB is ordered with previous stores) 
;and then starting moving to the memory controller WPQ 
;(the line containing X has been written back)

sfence           ;Wait for CLWB to become globally visible

;Here the store is in the WPQ

pcommit         

;The store is being committed

sfence          ;Wait for pcommit to become globally visible

;The store is committed

It turned out that every platform that is planning to support the new Intel NVM technology is also planning to support ADR, so Intel deprecated pcommit in favour of a simpler programming model:

mov [X], rax
clwb [X]
sfence         

;Here the store is in the WPQ and that's enough
Annapolis answered 26/1, 2017 at 16:22 Comment(8)
Is it for sure that stores are always put in WPQ? looking at hardware counters I observed that only around 6th of the write operations are inserted in WPQ, but I am not sure that makes sense.Impermeable
@AnaKhorguani AFAIK the WPQ is the only ingress port to (NV) memory. But that doesn't mean there's a 1-1 mapping with the stores, as stores can be merged in the SB and will be merged in the cache lines. I've never tested it with counters. You are using the uncore counters right? which one specifically (I don't remember them off the top of my head)? Maybe I can arrange a testAnnapolis
about merging, I am writing in an object which is 64 byte in size so should be entire cache line, that's why I thought they could not be merged more. Yes, I am using uncore counters, skx_unc_imc0::UNC_M_WPQ_INSERTS:cpu=0 this one for example which as I read: counts the number of allocations into the Write Pending Queue.Impermeable
Also, this every 6th writing to WPQ was when writing to DRAM, and when I try to count this event when writing in NVM, it does not get me anything, even though I should be mapping memory to the right numa node and checking correct cpu, but this is another problem. First I was trying to understand why not every store goes to WPQImpermeable
About the test, I did not mean to bother that much :) Already thanks for an interesting question/answer. I was looking some more information about WPQs but there is not that much that I've found, so this is already helpful.Impermeable
@AnaKhorguani It's a very interesting subject, I'd like to be able to write some code about it. I don't have too much time in my hands unfortunately, so I'm not sure I'll be able to code anything. Once useful source of information is Google Patent, have you tried it? I've found this which seems relevant but I've not read it yetAnnapolis
No I have not tried google patents yet, but I will read this one. I've posted it as a question in case if you would be interested, no one has commented yet though: #60822744Impermeable
I think I understood, or at least have an idea why this UNC_M_WPQ_INSERTS counter returned 1/6 of writes as a result, but then I am out of ideas why is it "working" when writing in DRAM and not when writing in NVM.Impermeable

© 2022 - 2024 — McMap. All rights reserved.