Iomart - Revision history

Mariusz: Add TOC

2020-09-29T12:10:46Z

Add TOC

Mariusz: Fix formatting for one of the listings

2020-09-09T15:35:58Z

Fix formatting for one of the listings

Antony c: /* Bartech */

2019-11-19T16:20:38Z

Bartech

Mariusz: Add description for adding/replacing a node in Bartech from scratch

2019-11-19T15:52:09Z

Add description for adding/replacing a node in Bartech from scratch

New page

== Bartech ==
=== Adding a compute node from scratch ===
First add your node to Ironic and provision it with a regular <code>centos7</code> image.

Copy public SSH key of the root user from the controller to <code>/root/.ssh/authorized_keys</code> on the node so Ansible can log into the node from the controller.

Install Quobyte client:

<nowiki>
yum -y install java-1.8.0-openjdk-headless wget
cd /etc/yum.repos.d/
wget https://packages.quobyte.com/repo/3/8acxjFCHCQ7YMvxKmNEzhYTQ1kr9xA2e/rpm/CentOS_7/quobyte.repo
yum -y install quobyte-client
</nowiki>
Grab <code>/etc/quobyte/client.cfg</code> from one of the existing compute nodes (log in to it though the controller) and copy the file to the new node.

Make sure the <code>ib0</code> interface is set up with a static IP by editing/creating <code>/etc/sysconfig/network-scripts/ifcfg-ib0</code> like the following:

<nowiki>
DEVICE=ib0
BOOTPROTO=static
IPADDR=192.168.100.<node-number>
NETMASK=255.255.255.0
ONBOOT=yes
NM_CONTROLLED=no
</nowiki>
where <node-number> is the number in the name of the node, e.g. for <code>node0004</code> this will be 4.

Bring the interface up by running <code>ifup ib0</code>

Create a mount point and mount tell Quobyte to mount the volume by restarting the client:

<nowiki>
mkdir /vscaler/home
systemctl restart quobyte-client.service
systemctl enable quobyte-client.service
</nowiki>

You should see the volume being mounted and with files and directories in it:

<nowiki>
# mount | grep vscaler/home
quobyte@192.168.100.201:7861|192.168.100.202:7861|192.168.100.203:7861 on /vscaler/home type fuse.quobyte (rw,nosuid,nodev,noatime,user_id=0,group_id=0,allow_other)
# ls -lah /vscaler/home/
total 428G
drwxr-xr-x. 1 root root 0 Nov 18 10:00 .
drwxr-xr-x. 4 root root 32 Nov 15 16:41 ..
drwx------. 1 nfsnobody nfsnobody 0 Nov 15 17:49 acaldas
drwx------. 1 nfsnobody nfsnobody 0 Nov 15 17:49 ccairoli
drwx------. 1 7872 7872 0 Aug 28 13:57 cfd_biosit
drwx--x---. 1 nfsnobody nfsnobody 0 Nov 15 17:49 rems
drwx------. 1 nfsnobody nfsnobody 0 Nov 18 09:51 rsupport
drwxrwx---. 1 nfsnobody nfsnobody 0 Nov 7 15:59 shared-BARTech
-rw-r--r--. 1 centos centos 416G Oct 15 01:47 shared-BARTech.tgz
-rw-r--r--. 1 root root 10G Nov 6 09:16 testfile
drwx------. 1 nfsnobody nfsnobody 0 Nov 15 17:48 tgratton
</nowiki>

Log into the controller and add your node's IP with its appropriate nodeXXXX name to <code>/etc/hosts</code>. Then run deployment playbooks:

<nowiki>
cd /opt/vScaler/site/
ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook controller.yml --skip-tags=luna -t bind
ansible-playbook static-compute.yml -l nodeXXXX
</nowiki>
where <code>nodeXXXX</code> is node name added to hosts.

When this is done, log back to the node and make sure OFED is installed. A specific version, for the specific node kernel, has to be installed. For <code>3.10.0-957.27.2.el7.x86_64</code> add this repo:

<nowiki>
cat /etc/yum.repos.d/mellanox_mlnx_ofed.repo
#
# Mellanox Technologies Ltd. public repository configuration file.
# For more information, refer to http://linux.mellanox.com
#
[mlnx_ofed_latest]
name=Mellanox Technologies rhel7.6-$basearch mlnx_ofed latest version
baseurl=http://linux.mellanox.com/public/repo/mlnx_ofed/latest/rhel7.6/$basearch
enabled=0
gpgkey=http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox
gpgcheck=1
[mlnx_ofed_4.6-1.0.1.1_base]
name=Mellanox Technologies rhel7.6-$basearch mlnx_ofed 4.6-1.0.1.1 GA
baseurl=http://repomirror/mlnx_ofed_4.6-1.0.1.1_base/
enabled=1
gpgkey=http://repomirror/RPM-GPG-KEY-Mellanox
gpgcheck=1
</nowiki>
and then install the <code>mlnx-ofed-hpc</code> package.

Umount the <code>/vscaler/home</code> share and restart the Quobyte client so it'll start using LDAP:

<nowiki>
umount /vscaler/home/
systemctl restart quobyte-client.service
</nowiki>
At this point you should see proper users (instead of nfsnobody) on files in this shared home directory, like so:

<nowiki>
# ls -lah /vscaler/home/
total 428G
drwxr-xr-x 1 root root 0 Nov 18 10:00 .
drwxr-xr-x. 4 root root 32 Oct 11 14:15 ..
drwx------ 1 acaldas BARTech 0 Nov 15 17:49 acaldas
drwx------ 1 ccairoli BARTech 0 Nov 15 17:49 ccairoli
drwx------ 1 7872 7872 0 Aug 28 13:57 cfd_biosit
drwx--x--- 1 rems BARTech 0 Nov 15 17:49 rems
drwx------ 1 rsupport admins 0 Nov 18 09:51 rsupport
drwxrwx--- 1 rems BARTech 0 Nov 7 15:59 shared-BARTech
-rw-r--r-- 1 centos centos 416G Oct 15 01:47 shared-BARTech.tgz
-rw-r--r-- 1 root root 10G Nov 6 09:16 testfile
drwx------ 1 tgratton BARTech 0 Nov 15 17:48 tgratton
</nowiki>

Set up StarCCM+:

<nowiki>
ln -s /vscaler/home/shared-BARTech/SIEMENS /opt/SIEMENS
yum install redhat-lsb-core -y
</nowiki>
Finally, add this line with IP of the license server to <code>/etc/hosts</code> on the node:

<nowiki>
62.7.66.229 BARTech-lic
</nowiki>

==== Testing ====
A job for testing OFED and RDMA:

<nowiki>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module add gnu8/8.3.0 openmpi3/3.1.4 imb/2018.1
mpirun IMB-MPI1 pingpong
</nowiki>

TODO: Add a basic Slurm job checking if StarCCM+ is set up correctly.

← Older revision		Revision as of 12:10, 29 September 2020
Line 1:		Line 1:
		+	__TOC__
		+
	== Bartech ==		== Bartech ==
	=== Adding a compute node from scratch ===		=== Adding a compute node from scratch ===

@@ Line 39: / Line 39: @@
 You should see the volume being mounted and with files and directories in it:
-<nowiki>
+ <nowiki>
 # mount | grep vscaler/home
-quobyte@192.168.100.201:7861|192.168.100.202:7861|192.168.100.203:7861 on /vscaler/home type fuse.quobyte (rw,nosuid,nodev,noatime,user_id=0,group_id=0,allow_other)
+quobyte @ 192.168.100.201:7861|192.168.100.202:7861|192.168.100.203:7861 on /vscaler/home type fuse.quobyte (rw,nosuid,nodev,noatime,user_id=0,group_id=0,allow_other)
 # ls -lah /vscaler/home/
 total 428G

@@ Line 66: / Line 66: @@
 where <code>nodeXXXX</code> is node name added to hosts.
-When this is done, log back to the node and make sure OFED is installed. A specific version, for the specific node kernel, has to be installed. For <code>3.10.0-957.27.2.el7.x86_64</code> add this repo:
+When this is done, log back to the node and make sure OFED is installed. A specific version, for the specific node kernel, has to be installed. SCP the <code>MLNX_OFED_LINUX-4.7-1.0.0.1-rhel7.6-x86_64.tgz</code> from the /root on the head node to the compute node. SSH in, extract, cd into the new dir and run <code>./mlnxofedinstall</code> and reboot
 Umount the <code>/vscaler/home</code> share and restart the Quobyte client so it'll start using LDAP: