Infiniband addressing - host names to IB address without IBoIP

Question

I've just started getting familiar with infiniband and I'm wanting to understand the methods you can use to address the infiniband nodes.

Based on the code is the example from: RDMA read and write with IB verbs I can address individual nodes by IP or hostname using IPoIB.

Another way is to use a port GUID address directly. But it looks like you'd have to look those up and is more similar to ethernet mac addressing.

Then then is something called an LID address, a 16bit local address assigned by the fabric manager. How do I use and determine at runtime an LID address? for example, I run ibaddr and get GID fe80::1a:4bff:ff0c:34e5 LID start 0x6 end 0x6

Basically, if you're not using IPoIB how do you convert host names to addresses or similar? Is there a hosts file or some equivalent?

Kuykendall

Answer 1

There is a basic difference between the various addressing methods that you are listing:

1. Addressing with pure IB verbs
2. Addressing with some level of abstraction

When a packet is "injected" into the IB fabric, it is routed by LID only, that is part of Local Routing Header of a packet. LID is Local ID, 16 bits, assigned by OpenSM (there's also a case of GID and Global Routing Header, but let's leave this case aside - it won't make the explanation any easier, and you obviously don't need this at this point).

This means that if you're writing your application using pure IB verbs, you will need to address the endpoints by LID. You can obtain the LID of a local port with ibv_query_port() - it is part of the port attribute fields.

But you don't have to do all the dirty work yourself - you can use abstraction libraries such as librdmacm (RDMA Connection Manager) to create connection between endpoints (and by "endpoints" I mean RC QPs), and then use pure verbs to actually send/receive your data.

Basically, if you're not using IPoIB how do you convert host names to addresses or similar? Is there a hosts file or some equivalent?

You can't, and there isn't :( If you go through the earlier post on that blog that you linked to, you see that you need to:

• Determine the queue pair’s address.
• Communicate the address to the other node (through some out-of-band mechanism).

The key item here is "out-of-band". For instance, MPI exchanges all these addresses over SSH (which, BTW, can also run on top of IPoIB), and once this info is exchanged and all the QPs are connected, data starts flowing via these RC QPs.

Router

Thanks for the info. It does seem like running IPoIB is the way to go. – Kuykendall

even tools like qperf seem to use some out-of-band mechanism to exchange such information – Carcinoma