RDMA atomic operation's implementation

Question

I heard about that RDMA reads and writes are implemented like, when requests come the nic find the physical page and then using DMA to get the data to nic and then to the target.

This is straightforward for reads and write, but it is odd when it comes to the atomic operations. My question is that is RDMA atomic operations implemented the same way as reads and writes and how? And more specificely, what is it's relationship between the cpu's atomic operations (like compare and swap) and the RDMA's corresponding operations?.

Strephon

Answer 1

RDMA atomic operations are implemented using PCI-express read and write operations. As such they do not provide atomicity with respect to the CPU's atomic operations, nor with respect to other HCAs.

Mestizo

Are there HCAs (say, Mellanox's ConnectX-4) that use PCIe atomics instead of read + write over PCIe? – Corron

Current Mellanox HCAs do not support that feature. – Mestizo

Haggai, what about Connect X-5? mellanox.com/related-docs/user_manuals/ConnectX-5_VPI_IC.pdf "ConnectX-5 delivers ... additional Network Atomic and PCIe Atomic operations support. – PCIe Gen 4 ... – PCIe Atomic". Are host platform required to provide PCIe Atomic to use X-5? – Lezley

Yes, the host CPU must support PCIe atomics. I believe that the PCIe controller on Ivy Bridge and newer Intel server CPUs supports PCIe atomics. Sandy Bridge CPUs support PCIe 3.0, but do not support PCIe atomics AFAIK. – Corron