Linux: Setting up RDMA over Ethernet

More detailed info can be found here : https://community.mellanox.com/docs/DOC-2086

Download and install MLNX OFED drivers

Download the driver package from http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers, after choosing appropriate platform and version of OS being used.

Install

# ./mlnxofedinstall

For a diskless installation like chroot

# some other notes on installing in chroot for ohpc / openhpc 
[root@head1 ~]# mount -o bind /proc/ /opt/ohpc/admin/images/centos7.2/proc/
[root@head1 ~]# chroot /opt/ohpc/admin/images/centos7.2/
# then install as usual 
# ./mlnxofedinstall --without-fw-update

Install RDMA

# yum install rdma
# service rdma restart ; chkconfig rdma on

Port verification

Make sure that the ports are configured properly for use as ethernet

# connectx_port_config 

ConnectX PCI devices :
|----------------------------|
| 1             0000:02:00.0 |
|----------------------------|

Before port change:
eth

|----------------------------|
| Possible port modes:       |
| 1: Infiniband              |
| 2: Ethernet                |
| 3: AutoSense               |
|----------------------------|
Select mode for port 1 (1,2,3): 2

After port change:
eth

RDMA Verification

On server, run

# udaddy

On client, run

# udaddy -s <server_ip>