VScaler: CIX Network Node Preparation

Base OS Installation

• Base installation of CentOS 7.2 was performed from our PXE server (pxe.boston.co.uk)
• NB - This could have been done from the OHPC head node
• Nothing more than a vanilla installation

Set a hostname

[root@localhost ~]# hostnamectl set-hostname "network1.cix.vscaler.cloud" --static
[root@localhost ~]# hostname -f
network1.cix.vscaler.cloud

Set up SSH keys

[root@head ~]# ssh-keygen -t dsa

Configure the internal/management/provisioning network interface

If first ethernet interface is to be used for management only

• Create an /etc/sysconfig/network-scripts/ifcfg-enp6s0f0 file with the following contents

TYPE=Ethernet
BOOTPROTO=static
IPADDR=172.18.0.2
NETMASK=255.255.0.0
NAME=enp6s0f0
UUID=41d85fef-a915-4a89-a28a-a9cd4dac3de8
DEVICE=enp6s0f0
ONBOOT=yes

If first ethernet interface is to be used for management and cluster/cloud traffic

• Edit the /etc/sysconfig/network-scripts/ifcfg-enp6s0f0 file:

DEVICE=enp6s0f0
TYPE=ethernet
BOOTPROTO=none
ONBOOT=yes
BRIDGE=br0
MTU=1500

• Create and edit the /etc/sysconfig/network-scripts/ifcfg-br0 file:

DEVICE=br0
ONBOOT=yes
MTU=1500
BOOTPROTO=none
IPADDR=172.18.1.1
NETMASK=255.255.0.0
GATEWAY=172.18.0.1
TYPE=Bridge
STP=off
DELAY=0

• Create and edit the /etc/sysconfig/network-scripts/ifcfg-br0:vxlan file:

DEVICE=br0\:vxlan
ONBOOT=yes
MTU=1500
BOOTPROTO=none
IPADDR=172.19.1.1
NETMASK=255.255.0.0
TYPE=Bridge
STP=off
DELAY=0

Configure the infiniband interface

• Create an /etc/sysconfig/network-scripts/ifcfg-ib0 file with the following contents

DEVICE=ib0
BOOTPROTO=static
IPADDR=172.20.0.2
NETMASK=255.255.0.0
ONBOOT=yes

• Install ibutils and infiniband-diags

[root@network1 ~]# yum -y install ibutils infiniband-diags

• Stop and disable NetworkManager

[root@network1 ~]# systemctl stop NetworkManager
[root@network1 ~]# systemctl disable NetworkManager

• Bring up interface ib0

[root@network1 ~]# ifup ib0

• Check the IB link is up and showing 20Gb/s

[root@network1 ~]# ibstat
CA 'mthca0'
	CA type: MT25204
	Number of ports: 1
	Firmware version: 1.2.0
	Hardware version: a0
	Node GUID: 0x0030487f35500000
	System image GUID: 0x0030487f35500003
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 20
		Base lid: 3
		LMC: 0
		SM lid: 1
		Capability mask: 0x02590a68
		Port GUID: 0x0030487f35500001
		Link layer: InfiniBand

Update the /etc/hosts file

• It is necessary update the /etc/hosts file so that references to "controller" resolve to the local system

[root@network1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

172.18.0.1  controller head

• NB - If this node had been deployed from the OHPC head node, the hosts file would be synced with the head node