VScaler: CIX Network Node Preparation

Base OS Installation

• Base installation of CentOS 7.2 was performed from our PXE server (pxe.boston.co.uk)
• NB - This could have been done from the OHPC head node
• Nothing more than a vanilla installation

Set a hostname

[root@localhost ~]# hostnamectl set-hostname "network1.cix.vscaler.cloud" --static
[root@localhost ~]# hostname -f
network1.cix.vscaler.cloud

Configure the internal/management/provisioning network interface

• Create an /etc/sysconfig/network-scripts/ifcfg-enp6s0f0 file with the following contents

TYPE=Ethernet
BOOTPROTO=static
IPADDR=172.18.0.2
NETMASK=255.255.0.0
NAME=enp6s0f0
UUID=41d85fef-a915-4a89-a28a-a9cd4dac3de8
DEVICE=enp6s0f0
ONBOOT=yes

Configure the infiniband interface

• Create an /etc/sysconfig/network-scripts/ifcfg-ib0 file with the following contents

DEVICE=ib0
BOOTPROTO=static
IPADDR=172.20.0.2
NETMASK=255.255.0.0
ONBOOT=yes

• Install ibutils and infiniband-diags

[root@head ~]# yum -y install ibutils infiniband-diags

• Stop and disable NetworkManager

[root@head ~]# systemctl stop NetworkManager
[root@head ~]# systemctl disable NetworkManager

• Bring up interface ib0

[root@head ~]# ifup ib0

• Install, start and enable opensm

[root@head ~]# yum -y install opensm
[root@head ~]# systemctl start opensm
[root@head ~]# systemctl enable opensm

• Check the IB link is up and showing 20Gb/s

[root@head ~]# ibstat
CA 'mthca0'
	CA type: MT25204
	Number of ports: 1
	Firmware version: 1.2.0
	Hardware version: a0
	Node GUID: 0x0030487f354a0000
	System image GUID: 0x0030487f354a0003
	Port 1:
		State: Initializing
		Physical state: LinkUp
		Rate: 20
		Base lid: 0
		LMC: 0
		SM lid: 0
		Capability mask: 0x02590a68
		Port GUID: 0x0030487f354a0001
		Link layer: InfiniBand

Update the /etc/hosts file

• It is necessary update the /etc/hosts file so that references to "controller" resolve to the local system

[root@head ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

172.18.0.1  controller head