Genomics England - Revision history

Mariusz: /* Dragen migration to P2 (Helix) */ Use the -vv flag with ansible-playbook commands so they are more verbose

2020-07-02T09:35:07Z

Dragen migration to P2 (Helix): Use the -vv flag with ansible-playbook commands so they are more verbose

Mariusz: /* Dragen migration to P2 (Helix) */ Add a note on installing Datadog on the Dragen node

2020-07-02T09:30:56Z

Dragen migration to P2 (Helix): Add a note on installing Datadog on the Dragen node

Mariusz: /* Dragen migration to P2 (Helix) */ Move sssd section before the LSF part as the LSF installer requires the lsfadmin user to be present

2020-07-02T09:19:13Z

Dragen migration to P2 (Helix): Move sssd section before the LSF part as the LSF installer requires the lsfadmin user to be present

Mariusz: /* Add node to LSF */ Add commands for setting up LSF on the node

2020-07-02T09:17:09Z

Add node to LSF: Add commands for setting up LSF on the node

Mariusz: /* Dragen migration to P2 (Helix) */ Use a different name for Dragen Ironic nodes

2020-07-02T08:46:03Z

Dragen migration to P2 (Helix): Use a different name for Dragen Ironic nodes

Mariusz: /* Dragen migration to P2 (Helix) */ Add a note on changing boot order for the node

2020-07-02T08:30:44Z

Dragen migration to P2 (Helix): Add a note on changing boot order for the node

Mariusz: /* Dragen migration to P2 (Helix) */ Add subsections for LSF and SSSD

2020-07-02T08:23:57Z

Dragen migration to P2 (Helix): Add subsections for LSF and SSSD

Mariusz: /* Dragen migration from P1 to P2 */ Change title of this section in case of Dragen migrations from a different place than P1

2020-07-02T08:16:57Z

Dragen migration from P1 to P2: Change title of this section in case of Dragen migrations from a different place than P1

Mariusz: /* Dragen migration from P1 to P2 */ Add commands for setting up SSSD

2020-07-01T14:50:03Z

Dragen migration from P1 to P2: Add commands for setting up SSSD

Mariusz: Add the first version of docs for Dragen migration - a description of Dragen provisioning and deployment but without LSF components

2020-07-01T14:28:43Z

Add the first version of docs for Dragen migration - a description of Dragen provisioning and deployment but without LSF components

New page

= Dragen nodes =

== General info ==

* also known as "Edico" nodes
* no raid in Dragen nodes (only one disk)
* a local NVMe drive in each (used as cache for data)
* boot interface on <code>enp134s0f0</code>

== Dragen migration from P1 to P2 ==

First, wait for GEL to physically move Dragen boxes from P1 to P2.

To proceed you will need the following:

* iDRAC IP addresses of nodes accessible from P2
* provisioning network interfaces connected to the <code>public304</code> provisioning network
* a confirmation of whether the node has to go to the dev or prod cluster

When the above requirements are fulfilled, you can add Dragen nodes to Ironic:

<nowiki>
# openstack image list
...
| 36a1c5fc-ff1c-40dc-9b87-75197e73257a | ironic-deploy_kernel | active |
| e73198d9-605b-45a8-84cb-c828599e59ca | ironic-deploy_ramdisk | active |
...
# openstack network list
...
| ab24b469-e07d-44ca-8bee-5c24d6c455e4 | public304 | a00b07c1-0a1b-4a36-91cb-5e0d51ec9258 |
...
# openstack baremetal node create --name edico01 --driver idrac --driver-info drac_address=10.6.6.44 --driver-info drac_username=<idrac-user> --driver-info drac_password=<idrac-password> --driver-info cleaning_network=ab24b469-e07d-44ca-8bee-5c24d6c455e4 --driver-info provisioning_network=ab24b469-e07d-44ca-8bee-5c24d6c455e4 --driver-info deploy_kernel=36a1c5fc-ff1c-40dc-9b87-75197e73257a --driver-info deploy_ramdisk=e73198d9-605b-45a8-84cb-c828599e59ca --resource-class baremetal --network-interface flat
# openstack baremetal node list | grep edico
# openstack baremetal port create 98:03:9b:8c:61:b2 --node ca6bd4f3-54f2-4877-8fba-db86f691a849
# openstack baremetal node manage edico01
# openstack baremetal node provide edico01
</nowiki>

Then provision the newly added node with an operating system:

<nowiki>
# openstack server create --image centos7-1907-dhcp-on-enp134s0f0-raid --flavor baremetal.small --security-group ping-and-ssh --key-name mykey --network public304 <t/p>hpgridzdragXXX
</nowiki>
After provisioning, log into the instance as the <code>centos</code> user and add this public key to root's <code>~/.ssh/authorized_keys</code> for passwordless SSH from the HPC controller:

<nowiki>
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBDEKTyKSRBpHcjgG16LF5mav11lEwbot1lmTPjvZPr6 cluster key
</nowiki>

The next step is to run Dragen deployment using the Dragen role (https://gitlab.vscaler.com/mkarpiarz/ansible-dragen-role) and trinityX playbooks residing on the HPC controller (<code>vcontroller</code>) in the main environment.

For this, you will have to add the new baremetal instance to DNS first. From GEL's <code>controller001</code> get into the HPC controller and run the following commands:

<nowiki>
# ssh vc
$ cd /opt/vScaler/site/
$ vim /etc/hosts
</nowiki>
Here add the IP and the name of the new Dragen node in the "Dragen boxes" section.

<nowiki>
$ vim hosts
</nowiki>
Here add only the name of the node in the <code>dragen_dev</code> (dev cluster) or <code>dragen</code> (production cluster) group.
And then run this playbook to update the DNS server:

<nowiki>
$ ansible-playbook controller.yml -t bind
</nowiki>
Ping the new node from the controller using its name to confirm the new DNS entry.

With the node added to DNS, run the Dragen deployment with this command:
<nowiki>
$ ansible-playbook dragen.yml
</nowiki>
Then run the compute playbook to add mounts, slurm workers, etc on Dragen nodes:

<nowiki>
$ ansible-playbook static-compute.yml -l dragen,dragen_dev
</nowiki>

Next up, set up LSF on the node -- this is currently a manual procedure and there is no playbook for it:

TODO: Add commands for this.

Finally, set up SSSD (also a manual step):

TODO: Add commands for this.

@@ Line 86: / Line 86: @@
 === Add node to LSF ===
-Next up, set up LSF on the node -- this is currently a manual procedure and there is no playbook for it:
+Next up, set up LSF on the node -- this is currently a manual procedure and there is no playbook for it.
-TODO: Add commands for this.
+ <nowiki>
 === Set up SSSD for AD ===

← Older revision		Revision as of 08:16, 2 July 2020
Line 9:		Line 9:


−	== Dragen migration ~~from P1~~ to P2 ==	+	== Dragen migration to P2 (Helix) ==

	First, wait for GEL to physically move Dragen boxes from P1 to P2.		First, wait for GEL to physically move Dragen boxes from P1 to P2.

@@ Line 70: / Line 70: @@
   <nowiki>
-$ ansible-playbook controller.yml -t bind
+$ ansible-playbook -vv controller.yml -t bind
 </nowiki>
 Ping the new node from the controller using its name to confirm the new DNS entry.
@@ Line 76: / Line 76: @@
 With the node added to DNS, run the Dragen deployment with this command:
   <nowiki>
-$ ansible-playbook dragen.yml
+$ ansible-playbook -vv dragen.yml
 </nowiki>
 Then run the compute playbook to add mounts, slurm workers, etc on Dragen nodes:
   <nowiki>
-$ ansible-playbook static-compute.yml -l dragen,dragen_dev
+$ ansible-playbook -vv static-compute.yml -l dragen,dragen_dev
 </nowiki>
 Also, remember to install Datadog on the new node by running this playbook:
   <nowiki>
-$ ansible-playbook datadog-agent-install.yaml
+$ ansible-playbook -vv datadog-agent-install.yaml
 </nowiki>
 Limit this command to the name of the new node to speed the execution of this playbook up.

@@ Line 33: / Line 33: @@
 | ab24b469-e07d-44ca-8bee-5c24d6c455e4 | public304                                          | a00b07c1-0a1b-4a36-91cb-5e0d51ec9258 |
 ...
-# openstack baremetal node create --name edico01 --driver idrac --driver-info drac_address=10.6.6.44 --driver-info drac_username=<idrac-user> --driver-info drac_password=<idrac-password> --driver-info cleaning_network=ab24b469-e07d-44ca-8bee-5c24d6c455e4 --driver-info provisioning_network=ab24b469-e07d-44ca-8bee-5c24d6c455e4 --driver-info deploy_kernel=36a1c5fc-ff1c-40dc-9b87-75197e73257a --driver-info deploy_ramdisk=e73198d9-605b-45a8-84cb-c828599e59ca --resource-class baremetal --network-interface flat
+# openstack baremetal node create --name edico-dragen016 --driver idrac --driver-info drac_address=10.6.6.44 --driver-info drac_username=<idrac-user> --driver-info drac_password=<idrac-password> --driver-info cleaning_network=ab24b469-e07d-44ca-8bee-5c24d6c455e4 --driver-info provisioning_network=ab24b469-e07d-44ca-8bee-5c24d6c455e4 --driver-info deploy_kernel=36a1c5fc-ff1c-40dc-9b87-75197e73257a --driver-info deploy_ramdisk=e73198d9-605b-45a8-84cb-c828599e59ca --resource-class baremetal --network-interface flat
 # openstack baremetal node list | grep edico
-# openstack baremetal port create 98:03:9b:8c:61:b2 --node ca6bd4f3-54f2-4877-8fba-db86f691a849
+| ca6bd4f3-54f2-4877-8fba-db86f691a849 | edico-dragen016 | None                                 | None        | enroll             | False       |
-# openstack baremetal node manage edico01
+# openstack baremetal port create <10g-interface-mac> --node ca6bd4f3-54f2-4877-8fba-db86f691a849
-# openstack baremetal node provide edico01
+# openstack baremetal node manage edico-dragen016
 </nowiki>

@@ Line 19: / Line 19: @@
 * a confirmation of whether the node has to go to the dev or prod cluster
-When the above requirements are fulfilled, you can add Dragen nodes to Ironic:
+When you get the iDRAC address and credentials, log in to <nowiki>https://<idrac-address></nowiki> and change the boot order so that PXE/network boot from a 10G card is first on the list. Also, write down the MAC address of the first PCI network interface (the one this node will be booting from).
   <nowiki>