David: Created page with "== Hardware Spec == At the time of writing, the current spec host system is based around: * AS-1024US-TRT (or similar H12 Ultra) * 2 x AMD 7402 CPUs * 16 x 32GB DIMMs * 2 x MC..."

2021-02-02T22:25:06Z

Created page with "== Hardware Spec == At the time of writing, the current spec host system is based around: * AS-1024US-TRT (or similar H12 Ultra) * 2 x AMD 7402 CPUs * 16 x 32GB DIMMs * 2 x MC..."

New page

== Hardware Spec ==
At the time of writing, the current spec host system is based around:
* AS-1024US-TRT (or similar H12 Ultra)
* 2 x AMD 7402 CPUs
* 16 x 32GB DIMMs
* 2 x MCX556A-EDAT (though 1 x single port version should suffice)
* At least on local drive for O/S and software install

== Operating System ==
At the time of writing, the latest support version of Linux is '''Ubuntu 18.04.4 LTS'''. Install this as normal (minimal installation is fine). '''CentOS 7.2''' and '''CentOS 8''' are also supported but, as yet, untested at Boston.

== Prerequisite Packages ==
A number of packages need to be installed using ''aptitude'':
<pre>
root@ipu-host-2:~# apt-get install apt-transport-https ibverbs-utils openjdk-8-jdk python3-virtualenv autoconf ipmitool php-cli python3-wheel automake jq php-curl qtcreator bc kcachegrind policykit-1 rdma-core build-essential libaio-dev protobuf-compiler screen ccache libboost-all-dev python-boto3 software-properties-common clang libeigen3-dev python-dev sshpass cmake libjson-c-dev python-lxml subversion curl libjson-c-doc python-numpy swig direnv libpci-dev python-pip sysfsutils dkms libpixman-1-dev python-pytest tar emacs libprotobuf-dev python-recommonmark tmux ethtool libtool python-requests u-boot-tools exuberant-ctags lldpad python-setuptools unzip flex m4 python-wheel valgrind g++ minicom python-yaml vim gawk moreutils python2 virtualenv gcc net-tools python3 wdiff gdb netcat python3-dev wget git parallel python3-numpy zip golang-go pciutils python3-pip htop perl python3-setuptools
</pre>

And a few python packages which can be installed via ''pip'':
<pre>
root@ipu-host-2:~# pip install autograd paramiko pylint scp jstyleson pep8 pyyaml yapf mock pexpect requests
</pre>

== User Accounts ==
=== Account Overview ===
The following user accounts are required on the host system:
{| class="wikitable"
|- valign="top"
! style="background: grey"| Accounts
! style="background: grey"| Function
|- valign="middle"
! style="background: white"| root
! style="background: white"| A root user account secured with a password is recommended.
|- valign="middle"
! style="background: white"| itadmin
! style="background: white"| An admin account secured with a password is recommended. Home folder located at /home/itadmin using bash shell.
|- valign="middle"
! style="background: white"| ipuuser
! style="background: white"| An account dedicated to IPU software and IPU-M2000 management software is mandatory. Home folder located at /home/ipuuser using bash shell.
|- valign="middle"
! style="background: white"| poplaruser
! style="background: white"| An account dedicated to Poplar software is mandatory. Home folder located at /home/poplaruser using bash shell.
|}
 
The following user accounts are present on the IPU M2000 system:
{| class="wikitable"
|- valign="top"
! style="background: grey"| Login to
! style="background: grey"| Username
! style="background: grey"| Password
|- valign="middle"
! style="background: white"| IPU-M2000 BMC OS
! style="background: white"| root
! style="background: white"| 0penBmc
|- valign="middle"
! style="background: white"| IPU-M2000 GW OS
! style="background: white"| itadmin
! style="background: white"| ChangeMeFdh5P
|}

=== Create Users on Host system ===
Create users with ''useradd'':
<pre>
root@ipu-host-2:~# useradd -m itadmin
root@ipu-host-2:~# useradd -m ipuuser
root@ipu-host-2:~# useradd -m poplaruser
</pre>

And set passwords with '''passwd''' (repeat for all users):
<pre>
root@ipu-host-2:~# passwd itadmin
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
</pre>

Double check that the default shell for each of these users is ''bash''. Edit the <code>/etc/passwd</code> file as appropriate, but each of these new users should look something like:
<pre>
ipuuser@ipu-host-2:~$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
.
.
.
itadmin:x:1001:1001::/home/itadmin:/bin/bash
ipuuser:x:1002:1002::/home/ipuuser:/bin/bash
poplaruser:x:1003:1003::/home/poplaruser:/bin/bash
</pre>

=== Add sudo rights ===
Edit the <code>/etc/sudoers</code> file to look something like:
<pre>
ipuuser@ipuhost:~$ sudo cat /etc/sudoers
#
# This file MUST be edited with the 'visudo' command as root.
#
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#
Defaults env_reset
Defaults mail_badpass
Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"

# Host alias specification

# User alias specification

# Cmnd alias specification

# User privilege specification
root ALL=(ALL:ALL) ALL
itadmin ALL=(ALL:ALL) ALL
ipuuser ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "#include" directives:

#includedir /etc/sudoers.d
</pre>

== Networking ==
=== Network Configuration Overview ===
The Host system should have interfaces configured as follows:
{| class="wikitable"
|- valign="top"
! style="background: grey"| Port
! style="background: grey"| Role
! style="background: grey"| Link Speed
! style="background: grey"| IP address
! style="background: grey"| Configured From
|- valign="middle"
! style="background: white"| enp65s0f0
! style="background: white"| External connectivity to IT infrastructure
! style="background: white"| 1GbE
! style="background: white"| 192.168.8.20x/20
! style="background: white"| Static or DHCP lease
|- valign="middle"
! style="background: white"| enp65s0f1
! style="background: white"| Management of IPU-M2000
! style="background: white"| 1GbE
! style="background: white"| 10.1.3.101/22
! style="background: white"| Static
|- valign="middle"
! style="background: white"| enp129s0f0
! style="background: white"| Unused
! style="background: white"| 100GbE
! style="background: white"| N/A
! style="background: white"| N/A
|- valign="middle"
! style="background: white"| enp129s0f1
! style="background: white"| RDMA interface to IPU-M2000
! style="background: white"| 100GbE
! style="background: white"| 10.1.5.5/24
! style="background: white"| Static
|}

The IPU M2000 system should have interfaces configured as follows:
{| class="wikitable"
|- valign="top"
! style="background: grey"| Port
! style="background: grey"| Role
! style="background: grey"| Link Speed
! style="background: grey"| IP address
! style="background: grey"| Configured From
|- valign="middle"
! style="background: white"| IPU Mgmt Port
! style="background: white"| BMC+GW management ports
! style="background: white"| 1GbE
! style="background: white"| BMC: 10.1.1.1/22 GW: 10.1.2.1/22
! style="background: white"| Static lease from DHCP server
|- valign="middle"
! style="background: white"| IPU 100GbE #1
! style="background: white"| Host-link data-plane link to IPU-M2000s
! style="background: white"| 100GbE
! style="background: white"| 10.1.5.2/30
! style="background: white"| Static lease from DHCP server
|}

=== Onboard Intel Network Drivers ===
==== Build Intel NIC drivers ====
'''''It may be necessary to install a temporary NIC in order to pull down build tools and drivers'''''
 
Install <code>build-essentials</code>
<pre>
ipuuser@ipuhost:~$ sudo apt-get install build-essentials
</pre>
Copy the Intel NIC driver to the system and decompress the archive:
<pre>
ipuuser@ipuhost:~$ tar zxvf i40e-2.13.10.tar.gz
</pre>
Switch to the source directory:
<pre>
ipuuser@ipuhost:~$ cd nic_temp/i40e-2.13.10/src/
</pre>
Make the module:
<pre>
ipuuser@ipuhost:~$ make install
</pre>
<code>Modprobe</code> the new driver module:
<pre>
ipuuser@ipuhost:~$ sudo modprobe i40e
</pre>

==== Configure IP address for onboard NIC ====
Edit the <code>/etc/netplan/01-netcfg.yaml</code> file to look something like this:
<pre>
ipuuser@ipuhost:~$ cat /etc/netplan/01-netcfg.yaml
# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
version: 2
renderer: networkd
ethernets:
enp65s0f0:
addresses:
- 192.168.8.201/20
gateway4: 192.168.5.1
nameservers:
addresses: [192.168.5.3, 192.168.5.2]
</pre>

Apply the Netplan config:
<pre>
ipuuser@ipuhost:~$ sudo netplay apply
</pre>

=== Mellanox Network Drivers ===
==== Add Mellanox Repos ====
Add the Mellanox repositories so we can install the necessary Mellanox drivers:
<pre>
cd /etc/apt/sources.list.d/
curl -LO https://linux.mellanox.com/public/repo/mlnx_ofed/latest/ubuntu18.04/mellanox_mlnx_ofed.list
</pre>

Edit the '''.list''' file so it looks like this:
<pre>
ipuuser@ipu-host-2:/etc/apt/sources.list.d$ cat mellanox_mlnx_ofed.list
#
# Mellanox Technologies Ltd. public repository configuration file.
# For more information, refer to http://linux.mellanox.com
#

# [mlnx_ofed_5.2-1.0.4.0_base]
deb [trusted=yes] http://linux.mellanox.com/public/repo/mlnx_ofed/5.2-1.0.4.0/ubuntu18.04/$(ARCH) ./
</pre>

Pull down a copy of the Mellanox GPG key:
<pre>
root@ipu-host-3:/etc/apt/sources.list.d# wget -qO - https://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add -
</pre>

Update the aptitude database and install the Mellanox OFED package:
<pre>
root@ipu-host-2:~$ apt update
</pre>

==== Install and configure Mellanox drivers ====
<pre>
root@ipu-host-2:~$ apt-get install mlnx-ofed-all
</pre>

Ensure the interfaces are configured to '''Ethernet''' rather than '''Infiniband''':
<pre>
root@ipu-host-2:~$ mlxconfig -d /dev/mst/mt4121_pciconf0 query
root@ipu-host-2:~$ mlxconfig -d /dev/mst/mt4121_pciconf0 set LINK_TYPE_P1=2
root@ipu-host-2:~$ mlxconfig -d /dev/mst/mt4121_pciconf0 set LINK_TYPE_P2=2
</pre>

'''Reboot the server'''

=== Netplan configuration ===
Networking on the Host system is configured by '''Netplan'''. The <code>/etc/netplan/01-netcfg.yaml</code> configuration file should look something like this (to reflect the table above):
<pre>
root@ipu-host-2:~# cat /etc/netplan/01-netcfg.yaml
# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
version: 2
renderer: networkd
ethernets:
enp65s0f0:
addresses:
- 192.168.8.202/20
gateway4: 192.168.5.1
nameservers:
addresses: [192.168.5.3, 192.168.5.2]
enp65s0f1:
addresses:
- 10.1.3.101/22
enp129s0f1:
addresses:
- 10.1.5.5/24
</pre>
Apply the Netplan configuration:
<pre>
root@ipu-host-2:~# netplan apply
</pre>

== DHCP Server ==
=== Installation ===
An '''isc-dhcp-server''' is required on the host system to provide the IPU M2000 system with the appropriate IP address (detailed above). It can be installed from the standard repos"
<pre>
root@ipu-host-2:~# apt-get install isc-dhcp-server
</pre>

=== Interface Configuration file ===
The <code>/etc/default/isc-dhcp-server</code> file dictates which interfaces will be serviced by the DHCP server service. It should look something like:
<pre>
root@ipu-host-2:~# cat /etc/default/isc-dhcp-server
INTERFACESv4="enp65s0f1 enp129s0f1"
INTERFACESv6=""
</pre>

=== Main DHCP configuration file ===
The <code>/etc/dhcp/dhcpd.conf</code> file provides configuration information for the DHCP server service. It should look something like:
<pre>
root@ipu-host-2:~# cat /etc/dhcp/dhcpd.conf
default-lease-time 600;
max-lease-time 1200;
ddns-update-style none;
authoritave;
log-facility local7;

subnet 10.1.5.0 netmask 255.255.255.0 {
option subnet-mask 255.255.255.0;
range 10.1.5.2 10.1.5.2;
}
subnet 10.1.0.0 netmask 255.255.252.0 {
option subnet-mask 255.255.252.0;
}
host ipum1bmc { hardware ethernet 70:69:79:20:13:b4; fixed-address 10.1.1.1; }
host ipum1gw { hardware ethernet 70:69:79:20:13:b5; fixed-address 10.1.2.1; }

host ipum1mx { hardware ethernet 0c:42:a1:78:89:cd; fixed-address 10.1.5.2; }
</pre>

=== Start and enable the DHCP Service ===
Start the DHCP service with:
<pre>
root@ipu-host-2:~# systemctl start isc-dhcp-server
</pre>
Enable the DHCP service with:
<pre>
root@ipu-host-2:~# systemctl enable isc-dhcp-server
</pre>
Check the status of the DHCP service with:
<pre>
root@ipu-host-2:~# systemctl status isc-dhcp-server
● isc-dhcp-server.service - ISC DHCP IPv4 server
Loaded: loaded (/lib/systemd/system/isc-dhcp-server.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-01-19 16:43:43 GMT; 1 day 22h ago
Docs: man:dhcpd(8)
Main PID: 3474 (dhcpd)
Tasks: 1 (limit: 19660)
CGroup: /system.slice/isc-dhcp-server.service
└─3474 dhcpd -user dhcpd -group dhcpd -f -4 -pf /run/dhcp-server/dhcpd.pid -cf /etc/dhcp/dhcpd.conf

Jan 21 15:36:08 ipu-host-2 dhcpd[3474]: DHCPREQUEST for 10.1.1.1 from 70:69:79:20:13:b4 via enp65s0f1
Jan 21 15:36:08 ipu-host-2 dhcpd[3474]: DHCPACK on 10.1.1.1 to 70:69:79:20:13:b4 via enp65s0f1
Jan 21 15:36:11 ipu-host-2 dhcpd[3474]: DHCPREQUEST for 10.1.2.1 from 70:69:79:20:13:b5 via enp65s0f1
Jan 21 15:36:11 ipu-host-2 dhcpd[3474]: DHCPACK on 10.1.2.1 to 70:69:79:20:13:b5 via enp65s0f1
Jan 21 15:36:24 ipu-host-2 dhcpd[3474]: DHCPREQUEST for 10.1.5.2 from 0c:42:a1:78:89:cd via enp129s0f1
Jan 21 15:36:24 ipu-host-2 dhcpd[3474]: DHCPACK on 10.1.5.2 to 0c:42:a1:78:89:cd via enp129s0f1
Jan 21 15:41:07 ipu-host-2 dhcpd[3474]: DHCPREQUEST for 10.1.1.1 from 70:69:79:20:13:b4 via enp65s0f1
Jan 21 15:41:07 ipu-host-2 dhcpd[3474]: DHCPACK on 10.1.1.1 to 70:69:79:20:13:b4 via enp65s0f1
Jan 21 15:41:10 ipu-host-2 dhcpd[3474]: DHCPREQUEST for 10.1.2.1 from 70:69:79:20:13:b5 via enp65s0f1
Jan 21 15:41:10 ipu-host-2 dhcpd[3474]: DHCPACK on 10.1.2.1 to 70:69:79:20:13:b5 via enp65s0f1
</pre>

== NTP Server Service ==
=== Installation ===
NTP service is recommended to provide network time configuration to IPU-M2000 systems. It can be installed from the Ubuntu repositories:
<pre>
root@ipu-host-2:~# apt-get install ntp
</pre>

=== Configuration ===
The <code>etc/ntp.conf </code> file details the configuration for the NTP server; it should look something like this:
<pre>
root@ipu-host-2:~# grep -v "#" /etc/ntp.conf

driftfile /var/lib/ntp/ntp.drift

leapfile /usr/share/zoneinfo/leap-seconds.list

includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
fudge 127.127.1.0 stratum 10
pool 0.ubuntu.pool.ntp.org iburst
pool 1.ubuntu.pool.ntp.org iburst
pool 2.ubuntu.pool.ntp.org iburst
pool 3.ubuntu.pool.ntp.org iburst
pool ntp.ubuntu.com

restrict 127.0.0.1
restrict ::1
restrict source notrap nomodify noquery
</pre>

=== Start and enable the NTP Service ===
Start the NTP service with:
<pre>
root@ipu-host-2:~# systemctl start ntp
</pre>
Enable the NTP service with:
<pre>
root@ipu-host-2:~# systemctl enable ntp
</pre>
Check the status of the NTP service with:
<pre>
root@ipu-host-2:~# systemctl status ntp
● ntp.service - Network Time Service
Loaded: loaded (/lib/systemd/system/ntp.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:ntpd(8)
</pre>

== Syslog ==
Syslog is a software utility for forwarding log messages in an IP network.
=== Configuration ===
The <code>/etc/rsyslog.conf</code> file should look like this:
<pre>
root@ipu-host-2:~# grep "^[^#;]" /etc/rsyslog.conf
module(load="imuxsock") # provides support for local system logging
module(load="imudp")
input(type="imudp" port="514")
module(load="imtcp")
input(type="imtcp" port="514")
module(load="imklog" permitnonkernelfacility="on")
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$RepeatedMsgReduction on
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
$WorkDirectory /var/spool/rsyslog
$IncludeConfig /etc/rsyslog.d/*.conf
</pre>

The <code>/etc/rsyslog.d/99_ipum.conf</code> file should look like this:
<pre>
root@ipu-host-2:~# grep "^[^#;]" /etc/rsyslog.d/99_ipum.conf
$template precise,"%fromhost-ip%,%HOSTNAME%,%syslogpriority%,%syslogfacility%,%timegenerated::fulltime%,%syslogtag%,%msg%\n"
:HOSTNAME, contains, "ipum" /var/log/ipulogs/ipulogs;precise
& ~
</pre>

The <code>/etc/rsyslog.d/99_dhcpd.conf</code> file should look like this:
<pre>
root@ipu-host-2:~# grep "^[^#;]" /etc/rsyslog.d/99_dhcpd.conf
local7.* /var/log/dhcpd.log
</pre>

== Graphcore Software Installation ==
The following Graphcore software packages need to be installed on the server: 
* '''V-IPU server''' contains management and control software for IPU resource control, built-in self-test (BIST) and monitoring of the IPU-M2000s and IPUs. There is a V-IPU Admin Guide and a V-IPU User Guide available.
* '''IPU-M2000 system software''' contains the latest IPU-M2000 resident software for update, if required. It also includes the server resident tool <code>rack_tool</code> which is required for updating the IPU-M2000s resident software and testing the system hardware.

=== V-IPU server installation ===
Both the release notes and the V-IPU software release tarball are available from the Graphcore download portal https://downloads.graphcore.ai 

An installation script called <code>install.sh</code> is included with the V-IPU tarball. The installation script has been tested and verified to work with Ubuntu and CentOS distros that
use systemd as the default service manager. The installation script needs to be executed with root privileges (sudo ./install.sh) as it copies the '''vipu-server''', '''vipu-admin''' and '''vipu''' binaries to <code>/usr/local/bin</code>.
 
'''''Ensure you are logged in as ipuuser'''''
 
The script will configure and start <code>vipu-server.service</code>. 

In the following example, the system is cabled according to the standard instructions where “enp65s0f0” is the host server interface that connects to the top IPU-M2000 management port at the top of the stack.
<pre>
ipuuser@ipu-host-2:~$ sudo ./install.sh
Do you want to start the vipu-server as a service in this host? Note that you should have vipu-server running only in one host and use vipu/vipu-admin to connect to it from all other hosts.

(N/y) y

Choose an interface to use for agent auto-discovery: eno0

enp65s0f0
enp65s0f1
lo

Enter disable to deactivate the auto-discovery
Which interface should be used for auto-discovery? enp65s0f0

- vipu-server will be configured to be run as a service in this host
- Initialising /etc/vipu/config.hcl
</pre>

=== IPU-M2000 Server Software Installation ===
The IPU-M2000 system software contains a set of upgradable software and FPGA sub- components that are targeted to be executed on the IPU-M2000 units. The release also contains the tool <code>rack_tool</code> which is used for the software upgrade and other rack related tasks targeting the IPU-M2000s.
 
'''''Ensure you are logged in as ipuuser'''''
 

Go to the Graphcore download portal https://downloads.graphcore.ai and download the latest release into your home directory 

Unpack the tarball:
<pre>
ipuuser@ipu-host-2:~$ tar xvf IPU_M_SW-2.0.0.rc.3.tar
</pre>

Install the software/tools:
<pre>
ipuuser@ipu-host-2:~$ cd ~/IPU_M_SW-2.0.0-rc.3+a51e75a/maintenance_tools/
ipuuser@ipu-host-2:~$ ./install.sh
</pre>

=== rack_tool Configuration ===
<code>rack_tool</code> requires a config file which contains information on all the IPU-M2000s it will control. The information in the config file defines all IP addresses of the BMC, GW and RNIC interfaces. 
Create a directory for the configuration file:
<pre>
ipuuser@ipu-host-2:~$ mkdir -p $HOME/.rack_tool
</pre>
Your <code>/home/ipuuser/.rack_tool/rack_config.json</code> should look something like this:
<pre>
ipuuser@ipu-host-2:~$ cat /home/ipuuser/.rack_tool/rack_config.json
{
"global_credentials": {
"bmc_username": "root",
"bmc_passwd": "0penBmc",
"gw_username": "itadmin",
"gw_passwd": "ChangeMeFdh5P"
},
"gw_root_overlay": "/home/ipuuser/IPU-M_releases/IPU_M_SW-2.0.0-rc.3+a51e75a/maintenance_tools/ipu_pod_config/root-overlay/",
"machines": [
{
"name": "m01",
"bmc_ip": "10.1.1.1",
"gw_ip": "10.1.2.1",
"mx_ip": "10.1.5.2"
}
]
}
</pre>

=== Copy root-overlay file system ===
A root-overlay file system is used to pass configuration of the NTP and syslog into the IPU- M2000 software. The rack_config.json file above refers to the path of these files. The path is either relative to the location of the rack_config.json or an absolute path. The easiest is to copy over the files to the default location:
<pre>
ipuuser@ipu-host-2:~$ cd /home/ipuuser/IPU-M_releases/IPU_M_SW-2.0.0-rc.3+a51e75a/maintenance_tools/ipu_pod_config
ipuuser@ipu-host-2:~$ cp -r root-overlay /home/ipuuser/.rack_tool/
</pre>

Graphcore M2000 Direct Connect Bringup - Revision history

David: Created page with "== Hardware Spec == At the time of writing, the current spec host system is based around: * AS-1024US-TRT (or similar H12 Ultra) * 2 x AMD 7402 CPUs * 16 x 32GB DIMMs * 2 x MC..."