How do I take advantage of RDMA in windows

Question

How can I copy blocks of memory from one server to another under windows using RDMA? We don't have infiniband, but we do have 10gb network switches. All I need is an example, but I'm not having much luck with google.

EDIT:

Well no one has answered my question so far. I just need a link to an example, or the name of a few functions. Everything I have read so far seems to be in the driver SDK or in sockets. I want to use RDMA in our software to update local copies of data from a main server. The data is up to 10GB in size. Most of the time the updates are about 1GB, but if a server has to reboot, the entire compressed dataset is 10GB. I want to update some of the data nearly continuously and RDMA seems to be the route to take.

Deandreadeane

Network Direct SPI: msdn.microsoft.com/en-us/library/cc904397%28v=vs.85%29.aspx – Deliladelilah

THANKS!! I think that's it! Post it as an answer and I will accept it. – Deandreadeane

Answer 1

You need NetworkDirect API. There's no documentation whatsoever on how to write something using NetworkDirect, only the interface description. The closest thing to the documentation that I could find is starting from here, and then going deeper into every link. But you can check the ND tests source code for the usage examples.

Polymer

Yeah, that's what I was running into. No docs. No examples. I'm going to check out what you sent. Looks good though. – Deandreadeane

Beware. There are 3 or 4 providers of ND. And the only one that has been available right through all the software versions is the IBAL one. Recently I see that they have dropped NDv1 & NDv2 and introduced in the 4.8 series drivers an unsupported ND over WinVerbs. ND over IBAL is still there. It's the only one I can get working consistently. – Crossroad

Answer 2

Isis2 (isis2.codeplex.com) now supports RDMA on Linux and Windows and MPI does too. I suggest that you work with those kinds of existing libraries. RDMA is extremely hard to use directly (I spent much of the past year adding RDMA support to Isis2). Documentation is horrible, only a few devices actually offer RDMA (although the set is increasing) and once you understand what you need to do, actually doing it is hard too. Check out the Isis2 code for INFINIBAND if you are curious -- my stuff is open source.

I don't have any experience with the Windows I/O completion mechanism Trevor mentions, but just to point out the obvious: if you don't limit yourself to aspects common to Linux and Windows, you end up with non-portable code. So this is also a problem -- only some aspects of RDMA work on both platforms, and one finds this out by trial and error!

Louls

Answer 3

The OpenFabrics Enterprise Distribution for Windows provides a libibverbs interface similar to what is commonly used on Linux. The main difference seems to be in handling completion channels: On Linux, you would poll a file descriptor, but on Windows, there's a (sparsely documented) I/O completion port-based helper library. Windows OFED (WinOF) libibverbs is a wrapper on top of winverbs, a COM-based API that mirrors libibverbs and an associated driver. Of course, according to this thread, NetworkDirect is the only API that Microsoft requires hardware vendors to support.

Capture