"Local" RDMA for development

Question

I'm trying to build and run the RDMA examples here. But because I'm just exploring, I don't have any hardware capable of managing RDMA. I get errors like this when I try to run the example code.

librdmacm: couldn't read ABI version.
librdmacm: assuming: 4
CMA: unable to get RDMA device list
error: ec = rdma_create_event_channel() failed (returned zero/null).

Is there any "local" implementation of the RDMA functionality that I can use for development? I understand that the "R" in RDMA means remote, but I thought this might exist for testing/development purposes.

For reference, I'm trying this on an Ubuntu 14.04 box having installed the packages libibverbs-dev and librdmacm-dev in order to get the code to compile.

Wanting

Were you ever successful on solving the error: ec = rdma_create_event_channel() failed (returned zero/null).? I'm stuck on this error as well. – Hatchery

Answer 1

Have you tried using soft-iwarp?

http://voidreflections.blogspot.com/2011/03/how-to-install-soft-iwarp-on-ubuntu.html

https://gitorious.org/softiwarp

Sandblast

I'm kinda new to this stuff, and it's not clear to me how iWarp relates. Additionally, I'm looking for an authoritative answer, i.e. someone who has done it and knows that it works. – Wanting

My understanding is that, RDMA is the abstraction layer, iWARP, InfiniBand and RoCE are implementations of it. – Sandblast

RoCE is RDMA over Ethernet L2, basically just like FCoE, iWARP is RDMA over a TCP – Sandblast

OFED is the software stack and is suppose to work transparently with all of them. – Sandblast

I second @thenewguy617. I've actually used SoftiWARP with all the code by TheGeekInTheCorner. I expect SoftiWARP to work with almost any verbs program. It's also pretty stable. Another possibility is using SoftROCE. – Eider

Man I am amazed SoftRoCE exits. Would be interesting to see what performs better, in theory SoftiWARP could use LSO/LRO, which most Intel NICs now have. – Sandblast

Answer 2

Soft-RoCE and iWARP drivers have been integrated in the Linux kernel after this question was asked. Here is an example of how to configure these drivers on a system with an Ethernet interface that has name eth0:

# apt-get install -y iproute2
# modprobe rdma_rxe
# modprobe siw
# rdma link add eth0_rxe type rxe netdev eth0
# rdma link add eth0_siw type siw netdev eth0
# ls /sys/class/infiniband
eth0_rxe  eth0_siw

Ezaria

Answer 3

Perhaps this is what you were looking for:

https://community.mellanox.com/docs/DOC-2184

This is a Soft RoCE installation guide. Since it's thorough and long it's impossible to copy-paste here. Quoting:

Soft-RoCE is a software implementation of RoCE that allows RoCE to run on any Ethernet network adapter whether it offers hardware acceleration or not.

Soft-RoCE is released as part of upstream kernel 4.8 (or above). Mellanox OFED 4.0 or upstream driver could be used. If you install MLNX_OFED 4.0, you automatically get the Soft-RoCE kernel module and user space libraries.

This post demonstrates how to install and setup upstream Soft-RoCE (aka RXE), and is meant for IT managers and developers who wish to test RDMA on software over any 3rd party adapters.

Lustre

A link to a solution is welcome, but please ensure your answer is useful without it: add context around the link so your fellow users will have some idea what it is and why it’s there, then quote the most relevant part of the page you're linking to in case the target page is unavailable. Answers that are little more than a link may be deleted. – Wernsman