VScaler: CIX Network Node Preparation - Revision history

Jon at 15:42, 2 August 2016

2016-08-02T15:42:02Z

Jon: /* Configure the internal/management/provisioning network interface */

2016-08-02T15:09:21Z

Configure the internal/management/provisioning network interface

Jon: /* Set a hostname */

2016-08-02T15:06:08Z

Set a hostname

Jon: /* Base OS Installation */

2016-08-02T15:04:32Z

Base OS Installation

Jon: /* Configure the internal/management/provisioning network interface */

2016-08-01T14:05:59Z

Configure the internal/management/provisioning network interface

Jon at 11:39, 28 July 2016

2016-07-28T11:39:28Z

Jon: /* Update the /etc/hosts file */

2016-07-28T11:28:55Z

Update the /etc/hosts file

Jon: /* Configure the infiniband interface */

2016-07-28T11:27:51Z

Configure the infiniband interface

Jon: /* Configure the infiniband interface */

2016-07-28T11:26:31Z

Configure the infiniband interface

Jon: Created page with "== Base OS Installation == * Base installation of '''CentOS 7.2''' was performed from our PXE server (pxe.boston.co.uk) * '''NB''' - This could have been done from the OHPC he..."

2016-07-28T11:25:26Z

Created page with "== Base OS Installation == * Base installation of '''CentOS 7.2''' was performed from our PXE server (pxe.boston.co.uk) * '''NB''' - This could have been done from the OHPC he..."

New page

== Base OS Installation ==
* Base installation of '''CentOS 7.2''' was performed from our PXE server (pxe.boston.co.uk)
* '''NB''' - This could have been done from the OHPC head node
* Nothing more than a vanilla installation

== Set a hostname ==
<syntaxhighlight>
[root@localhost ~]# hostnamectl set-hostname "network1.cix.vscaler.cloud" --static
[root@localhost ~]# hostname -f
network1.cix.vscaler.cloud
</syntaxhighlight>

== Configure the internal/management/provisioning network interface ==
* Create an '''<code>/etc/sysconfig/network-scripts/ifcfg-enp6s0f0</code>''' file with the following contents
<syntaxhighlight>
TYPE=Ethernet
BOOTPROTO=static
IPADDR=172.18.0.2
NETMASK=255.255.0.0
NAME=enp6s0f0
UUID=41d85fef-a915-4a89-a28a-a9cd4dac3de8
DEVICE=enp6s0f0
ONBOOT=yes
</syntaxhighlight>

== Configure the infiniband interface ==
* Create an '''<code>/etc/sysconfig/network-scripts/ib0</code>''' file with the following contents
<syntaxhighlight>
DEVICE=ib0
BOOTPROTO=static
IPADDR=172.20.0.2
NETMASK=255.255.0.0
ONBOOT=yes
</syntaxhighlight>

* Install '''ibutils''' and '''infiniband-diags'''
<syntaxhighlight>
[root@head ~]# yum -y install ibutils infiniband-diags
</syntaxhighlight>

* Stop and disable '''<code>NetworkManager</code>'''
<syntaxhighlight>
[root@head ~]# systemctl stop NetworkManager
[root@head ~]# systemctl disable NetworkManager
</syntaxhighlight>

* Bring up interface '''ib0'''
<syntaxhighlight>
[root@head ~]# ifup ib0
</syntaxhighlight>

* Install, start and enable '''<code>opensm</code>'''
<syntaxhighlight>
[root@head ~]# yum -y install opensm
[root@head ~]# systemctl start opensm
[root@head ~]# systemctl enable opensm
</syntaxhighlight>

* Check the IB link is up and showing 20Gb/s
<syntaxhighlight>
[root@head ~]# ibstat
CA 'mthca0'
CA type: MT25204
Number of ports: 1
Firmware version: 1.2.0
Hardware version: a0
Node GUID: 0x0030487f354a0000
System image GUID: 0x0030487f354a0003
Port 1:
State: Initializing
Physical state: LinkUp
Rate: 20
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02590a68
Port GUID: 0x0030487f354a0001
Link layer: InfiniBand
</syntaxhighlight>

== Update the /etc/hosts file ==
* It is necessary update the <code>/etc/hosts</code> file so that references to '''"controller"''' resolve to the local system
<syntaxhighlight>
[root@head ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

172.18.0.1 controller head
</syntaxhighlight>

@@ Line 13: / Line 13: @@
 <syntaxhighlight>
 [root@localhost ~]# hostnamectl set-hostname "networknode01.cix.vscaler.cloud" --static
 </syntaxhighlight>
@@ Line 82: / Line 77: @@
 </syntaxhighlight>
-== Configure the infiniband interface ==
+== Configure the infiniband interface (only if *not* installed from headnode) ==
 * Create an '''<code>/etc/sysconfig/network-scripts/ifcfg-ib0</code>''' file with the following contents
 <syntaxhighlight>
@@ Line 94: / Line 89: @@
 * Install '''ibutils''' and '''infiniband-diags'''
 <syntaxhighlight>
-[root@network1 ~]# yum -y install ibutils infiniband-diags
+[root@networknode01 ~]# yum -y install ibutils infiniband-diags
 </syntaxhighlight>
 * Stop and disable '''<code>NetworkManager</code>'''
 <syntaxhighlight>
-[root@network1 ~]# systemctl stop NetworkManager
+[root@networknode01 ~]# systemctl stop NetworkManager
-[root@network1 ~]# systemctl disable NetworkManager
+[root@networknode01 ~]# systemctl disable NetworkManager
 </syntaxhighlight>
 * Bring up interface '''ib0'''
 <syntaxhighlight>
-[root@network1 ~]# ifup ib0
+[root@networknode01 ~]# ifup ib0
 </syntaxhighlight>
 * Check the IB link is up and showing 20Gb/s
 <syntaxhighlight>
-[root@network1 ~]# ibstat
+[root@networknode01 ~]# ibstat
 CA 'mthca0'
 	CA type: MT25204
@@ Line 130: / Line 125: @@
 </syntaxhighlight>
-== Update the /etc/hosts file ==
+== Update the /etc/hosts file (only if *not* installed from headnode) ==
 * It is necessary update the <code>/etc/hosts</code> file so that references to '''"controller"''' resolve to the local system
 <syntaxhighlight>
-[root@network1 ~]# cat /etc/hosts
+[root@networknode01 ~]# cat /etc/hosts
 .0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
 ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

@@ Line 3: / Line 3: @@
 * Nothing more than a vanilla installation
-== Set a hostname ==
+== Confirm hostname ==
 <syntaxhighlight>
-[root@localhost ~]# hostnamectl set-hostname "network1.cix.vscaler.cloud" --static
+[root@networknode01 ~]# hostname -f
-[root@localhost ~]# hostname -f
+networknode01.cix.vscaler.cloud
-network1.cix.vscaler.cloud
+</syntaxhighlight>
 </syntaxhighlight>

@@ Line 1: / Line 1: @@
 == Base OS Installation ==
-* Base installation of '''CentOS 7.2''' was performed from our PXE server (pxe.boston.co.uk)
+* Base installation of '''CentOS 7.2''' was performed the controller node '''controller01.cix.vscaler.cloud'''
 * Nothing more than a vanilla installation

@@ Line 17: / Line 17: @@
 == Configure the internal/management/provisioning network interface ==
 * Create an '''<code>/etc/sysconfig/network-scripts/ifcfg-enp6s0f0</code>''' file with the following contents
 <syntaxhighlight>
@@ Line 27: / Line 28: @@
 DEVICE=enp6s0f0
 ONBOOT=yes
 </syntaxhighlight>

@@ Line 36: / Line 36: @@
 * Install '''ibutils''' and '''infiniband-diags'''
 <syntaxhighlight>
-[root@head ~]# yum -y install ibutils infiniband-diags
+[root@network1 ~]# yum -y install ibutils infiniband-diags
 </syntaxhighlight>
 * Stop and disable '''<code>NetworkManager</code>'''
 <syntaxhighlight>
-[root@head ~]# systemctl stop NetworkManager
+[root@network1 ~]# systemctl stop NetworkManager
-[root@head ~]# systemctl disable NetworkManager
+[root@network1 ~]# systemctl disable NetworkManager
 </syntaxhighlight>
 * Bring up interface '''ib0'''
 <syntaxhighlight>
-[root@head ~]# ifup ib0
+[root@network1 ~]# ifup ib0
 </syntaxhighlight>
 * Check the IB link is up and showing 20Gb/s
 <syntaxhighlight>
-[root@head ~]# ibstat
+[root@network1 ~]# ibstat
 CA 'mthca0'
 	CA type: MT25204
@@ Line 65: / Line 58: @@
 	Firmware version: 1.2.0
 	Hardware version: a0
-	Node GUID: 0x0030487f354a0000
+	Node GUID: 0x0030487f35500000
-	System image GUID: 0x0030487f354a0003
+	System image GUID: 0x0030487f35500003
 	Port 1:
-		State: Initializing
+		State: Active
 		Physical state: LinkUp
 		Rate: 20
-		Base lid: 0
+		Base lid: 3
 		LMC: 0
-		SM lid: 0
+		SM lid: 1
 		Capability mask: 0x02590a68
-		Port GUID: 0x0030487f354a0001
+		Port GUID: 0x0030487f35500001
 		Link layer: InfiniBand
 </syntaxhighlight>