Difference between revisions of "VScaler: CIX Network Node Preparation"
Jump to navigation
Jump to search
| (6 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
== Base OS Installation == | == Base OS Installation == | ||
| − | * Base installation of '''CentOS 7.2''' was performed | + | * Base installation of '''CentOS 7.2''' was performed the controller node '''controller01.cix.vscaler.cloud''' |
| − | |||
* Nothing more than a vanilla installation | * Nothing more than a vanilla installation | ||
| − | == | + | == Confirm hostname == |
<syntaxhighlight> | <syntaxhighlight> | ||
| − | [root@ | + | [root@networknode01 ~]# hostname -f |
| − | [root@localhost ~]# hostname | + | networknode01.cix.vscaler.cloud |
| − | + | </syntaxhighlight> | |
| + | |||
| + | Correct if necessary: | ||
| + | |||
| + | <syntaxhighlight> | ||
| + | [root@localhost ~]# hostnamectl set-hostname "networknode01.cix.vscaler.cloud" --static | ||
</syntaxhighlight> | </syntaxhighlight> | ||
== Configure the internal/management/provisioning network interface == | == Configure the internal/management/provisioning network interface == | ||
| + | === If first ethernet interface is to be used for management only === | ||
* Create an '''<code>/etc/sysconfig/network-scripts/ifcfg-enp6s0f0</code>''' file with the following contents | * Create an '''<code>/etc/sysconfig/network-scripts/ifcfg-enp6s0f0</code>''' file with the following contents | ||
<syntaxhighlight> | <syntaxhighlight> | ||
| Line 24: | Line 29: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
| − | == Configure the infiniband interface == | + | === If first ethernet interface is to be used for management and cluster/cloud traffic === |
| + | * Edit the '''<code>/etc/sysconfig/network-scripts/ifcfg-enp6s0f0</code>''' file: | ||
| + | <syntaxhighlight> | ||
| + | DEVICE=enp6s0f0 | ||
| + | TYPE=ethernet | ||
| + | BOOTPROTO=none | ||
| + | ONBOOT=yes | ||
| + | BRIDGE=br0 | ||
| + | MTU=1500 | ||
| + | </syntaxhighlight> | ||
| + | |||
| + | * Create and edit the '''<code>/etc/sysconfig/network-scripts/ifcfg-br0</code>''' file: | ||
| + | <syntaxhighlight> | ||
| + | DEVICE=br0 | ||
| + | ONBOOT=yes | ||
| + | MTU=1500 | ||
| + | BOOTPROTO=none | ||
| + | IPADDR=172.18.1.1 | ||
| + | NETMASK=255.255.0.0 | ||
| + | GATEWAY=172.18.0.1 | ||
| + | TYPE=Bridge | ||
| + | STP=off | ||
| + | DELAY=0 | ||
| + | </syntaxhighlight> | ||
| + | |||
| + | * Create and edit the '''<code>/etc/sysconfig/network-scripts/ifcfg-br0:vxlan</code>''' file: | ||
| + | <syntaxhighlight> | ||
| + | DEVICE=br0\:vxlan | ||
| + | ONBOOT=yes | ||
| + | MTU=1500 | ||
| + | BOOTPROTO=none | ||
| + | IPADDR=172.19.1.1 | ||
| + | NETMASK=255.255.0.0 | ||
| + | TYPE=Bridge | ||
| + | STP=off | ||
| + | DELAY=0 | ||
| + | </syntaxhighlight> | ||
| + | |||
| + | == Configure the external network interface == | ||
| + | * The external interface needs to be up but have no IP address | ||
| + | * Create/edit the '''<code>/etc/sysconfig/network-scripts/ifcfg-enp6s0f1</code>''' file: | ||
| + | <syntaxhighlight> | ||
| + | DEVICE=enp6s0f1 | ||
| + | BOOTPROTO=none | ||
| + | ONBOOT=yes | ||
| + | HWADDR=00:30:48:7f:35:51 | ||
| + | </syntaxhighlight> | ||
| + | |||
| + | == Configure the infiniband interface (only if *not* installed from headnode) == | ||
* Create an '''<code>/etc/sysconfig/network-scripts/ifcfg-ib0</code>''' file with the following contents | * Create an '''<code>/etc/sysconfig/network-scripts/ifcfg-ib0</code>''' file with the following contents | ||
<syntaxhighlight> | <syntaxhighlight> | ||
| Line 36: | Line 89: | ||
* Install '''ibutils''' and '''infiniband-diags''' | * Install '''ibutils''' and '''infiniband-diags''' | ||
<syntaxhighlight> | <syntaxhighlight> | ||
| − | [root@ | + | [root@networknode01 ~]# yum -y install ibutils infiniband-diags |
</syntaxhighlight> | </syntaxhighlight> | ||
* Stop and disable '''<code>NetworkManager</code>''' | * Stop and disable '''<code>NetworkManager</code>''' | ||
<syntaxhighlight> | <syntaxhighlight> | ||
| − | [root@ | + | [root@networknode01 ~]# systemctl stop NetworkManager |
| − | [root@ | + | [root@networknode01 ~]# systemctl disable NetworkManager |
</syntaxhighlight> | </syntaxhighlight> | ||
* Bring up interface '''ib0''' | * Bring up interface '''ib0''' | ||
<syntaxhighlight> | <syntaxhighlight> | ||
| − | [root@ | + | [root@networknode01 ~]# ifup ib0 |
</syntaxhighlight> | </syntaxhighlight> | ||
* Check the IB link is up and showing 20Gb/s | * Check the IB link is up and showing 20Gb/s | ||
<syntaxhighlight> | <syntaxhighlight> | ||
| − | [root@ | + | [root@networknode01 ~]# ibstat |
CA 'mthca0' | CA 'mthca0' | ||
CA type: MT25204 | CA type: MT25204 | ||
| Line 72: | Line 125: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
| − | == Update the /etc/hosts file == | + | == Update the /etc/hosts file (only if *not* installed from headnode) == |
* It is necessary update the <code>/etc/hosts</code> file so that references to '''"controller"''' resolve to the local system | * It is necessary update the <code>/etc/hosts</code> file so that references to '''"controller"''' resolve to the local system | ||
<syntaxhighlight> | <syntaxhighlight> | ||
| − | [root@ | + | [root@networknode01 ~]# cat /etc/hosts |
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 | 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 | ||
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 | ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 | ||
| Line 81: | Line 134: | ||
172.18.0.1 controller head | 172.18.0.1 controller head | ||
</syntaxhighlight> | </syntaxhighlight> | ||
| + | |||
| + | * '''NB''' - If this node had been deployed from the OHPC head node, the hosts file would be synced with the head node | ||
Latest revision as of 15:42, 2 August 2016
Base OS Installation
- Base installation of CentOS 7.2 was performed the controller node controller01.cix.vscaler.cloud
- Nothing more than a vanilla installation
Confirm hostname
[root@networknode01 ~]# hostname -f
networknode01.cix.vscaler.cloudCorrect if necessary:
[root@localhost ~]# hostnamectl set-hostname "networknode01.cix.vscaler.cloud" --staticConfigure the internal/management/provisioning network interface
If first ethernet interface is to be used for management only
- Create an
/etc/sysconfig/network-scripts/ifcfg-enp6s0f0file with the following contents
TYPE=Ethernet
BOOTPROTO=static
IPADDR=172.18.0.2
NETMASK=255.255.0.0
NAME=enp6s0f0
UUID=41d85fef-a915-4a89-a28a-a9cd4dac3de8
DEVICE=enp6s0f0
ONBOOT=yesIf first ethernet interface is to be used for management and cluster/cloud traffic
- Edit the
/etc/sysconfig/network-scripts/ifcfg-enp6s0f0file:
DEVICE=enp6s0f0
TYPE=ethernet
BOOTPROTO=none
ONBOOT=yes
BRIDGE=br0
MTU=1500- Create and edit the
/etc/sysconfig/network-scripts/ifcfg-br0file:
DEVICE=br0
ONBOOT=yes
MTU=1500
BOOTPROTO=none
IPADDR=172.18.1.1
NETMASK=255.255.0.0
GATEWAY=172.18.0.1
TYPE=Bridge
STP=off
DELAY=0- Create and edit the
/etc/sysconfig/network-scripts/ifcfg-br0:vxlanfile:
DEVICE=br0\:vxlan
ONBOOT=yes
MTU=1500
BOOTPROTO=none
IPADDR=172.19.1.1
NETMASK=255.255.0.0
TYPE=Bridge
STP=off
DELAY=0Configure the external network interface
- The external interface needs to be up but have no IP address
- Create/edit the
/etc/sysconfig/network-scripts/ifcfg-enp6s0f1file:
DEVICE=enp6s0f1
BOOTPROTO=none
ONBOOT=yes
HWADDR=00:30:48:7f:35:51Configure the infiniband interface (only if *not* installed from headnode)
- Create an
/etc/sysconfig/network-scripts/ifcfg-ib0file with the following contents
DEVICE=ib0
BOOTPROTO=static
IPADDR=172.20.0.2
NETMASK=255.255.0.0
ONBOOT=yes- Install ibutils and infiniband-diags
[root@networknode01 ~]# yum -y install ibutils infiniband-diags- Stop and disable
NetworkManager
[root@networknode01 ~]# systemctl stop NetworkManager
[root@networknode01 ~]# systemctl disable NetworkManager- Bring up interface ib0
[root@networknode01 ~]# ifup ib0- Check the IB link is up and showing 20Gb/s
[root@networknode01 ~]# ibstat
CA 'mthca0'
CA type: MT25204
Number of ports: 1
Firmware version: 1.2.0
Hardware version: a0
Node GUID: 0x0030487f35500000
System image GUID: 0x0030487f35500003
Port 1:
State: Active
Physical state: LinkUp
Rate: 20
Base lid: 3
LMC: 0
SM lid: 1
Capability mask: 0x02590a68
Port GUID: 0x0030487f35500001
Link layer: InfiniBandUpdate the /etc/hosts file (only if *not* installed from headnode)
- It is necessary update the
/etc/hostsfile so that references to "controller" resolve to the local system
[root@networknode01 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.18.0.1 controller head- NB - If this node had been deployed from the OHPC head node, the hosts file would be synced with the head node