DeepOps on OpenStack POC - Revision history

Mariusz: /* Enable monitoring (Prometheus + Grafana) */ Add more script-ready commands

2019-03-07T17:27:49Z

Enable monitoring (Prometheus + Grafana): Add more script-ready commands

Mariusz: /* DeepOps on vScaler POC */ Checkout to a specific revision of the DeepOps repo

2019-03-07T14:05:08Z

DeepOps on vScaler POC: Checkout to a specific revision of the DeepOps repo

Mariusz: Add a note on using the image UUID when accepting it

2019-03-07T13:39:07Z

Add a note on using the image UUID when accepting it

Mariusz: /* DeepOps on vScaler POC */ Add commands for deploying monitoring on GPU nodes

2019-01-10T14:50:05Z

DeepOps on vScaler POC: Add commands for deploying monitoring on GPU nodes

Mariusz: /* DeepOps on vScaler POC */ Use the `-o wide` flag with kubectl get pods

2019-01-09T13:26:01Z

DeepOps on vScaler POC: Use the `-o wide` flag with kubectl get pods

Mariusz: Add description on how to use the POC deployment on vScaler

2018-12-21T10:46:50Z

Add description on how to use the POC deployment on vScaler

New page

== DeepOps on vScaler POC ==

The Ubuntu driver and all the components required for a simple Kubernetes with GPU deployment (1 master, 1 worker) are available in the <code>ireland.south1</code> region. Below is a description how users can use it.

First, as an admin, share the DGX image with your project:

<nowiki>
[root@lhc-headnode ~]# source /etc/kolla/admin-openrc.sh
[root@lhc-headnode ~]# openstack image add project ubuntu-software-config-dgx <your-project-name>
[root@lhc-headnode ~]# openstack image member list ubuntu-software-config-dgx
</nowiki>
The last command will tell you the image in "pending" state. Log as your regular user and accept the image either through Horizon or with the command:

<nowiki>
$ openstack image set --accept ubuntu-software-config-dgx
</nowiki>
All of this is done this way for now because the vGPU driver and settings for connecting to our license server are baked into the image.

Next, head to the dashboard to <code>Container Infra -> Cluster Templates</code>, where you can see the "deepops-poc-template" template. This template was created using the following command:

<nowiki>
[root@lhc-headnode ~]# openstack coe cluster template create --coe kubernetes --image ubuntu-software-config-dgx --external-network public1 --flavor g1.large.1xk80 --master-flavor m1.medium --docker-storage-driver overlay --public --floating-ip-disabled deepops-poc-template
</nowiki>
Create a cluster off of the template by clicking on "Create Cluster" next to the template and specifying the name of the cluster and your keypair (leave all other parameters with their defaults).
You can also create a cluster with these commands:

<nowiki>
$ openstack coe cluster template list
$ openstack coe cluster create --cluster-template deepops-poc-template --keypair <your-keypair> <cluster-name>
$ openstack coe cluster list
</nowiki>
Now, head to <code>Orchestration -> Stacks</code> and wait for the stack associated with the cluster to transition into the "Create Complete" state. When this is done, check the list of your instances in <code>Compute -> Instances</code> and find the master node of the cluster. Assign a floating IP to it and SSH into it.

Confirm Kubernetes works by launching a test pod:

<nowiki>
kubectl get nodes
git clone https://github.com/NVIDIA/deepops
cd deepops/
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.12/nvidia-device-plugin.yml
kubectl apply -f tests/gpu-test-job.yml
kubectl get pods
</nowiki>
Wait for the pod to become ready and run these commands to confirm the pod can access the vGPU:

<nowiki>
kubectl describe pod gpu-pod
kubectl exec -ti gpu-pod -- nvidia-smi
</nowiki>

== Resources ==
* https://github.com/NVIDIA/deepops

@@ Line 62: / Line 62: @@
 cd deepops/
 cp -r config.example/ config
-</nowiki>
+KUBE_VERSION=$(kubectl version --short | grep -i server | cut -d ":" -f 2)
-Edit <code>config/kube.yml</code> and change <code>kube_version</code> to whatever <code>kubectl version --short</code> reports as the Server version.
+sed -i.bak "s/kube_version:.*$/kube_version:${KUBE_VERSION}/" config/kube.yml
+kubectl taint nodes --all node-role.kubernetes.io/master- | true
-Then continue:
+kubectl taint nodes --all kubeadmNode- | true
 MASTER=$(kubectl get nodes -l node-role.kubernetes.io/master= -o jsonpath='{.items[].metadata.name}')
 kubectl label node $MASTER node-role.kubernetes.io/master=true --overwrite
 ./scripts/helm_install_linux.sh
 kubectl create sa tiller --namespace kube-system
 kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
 helm init --service-account tiller --node-selectors node-role.kubernetes.io/master=true
 </nowiki>
-Wait for the tiller pod to become ready (check on it with <code>kubectl get pods -n kube-system | grep tiller</code> and then proceed:
+Wait for the tiller pod to become ready. Here is a quick script that waits for the tiller pod:
   <nowiki>
 helm repo add coreos https://s3-eu-west-1.amazonaws.com/coreos-charts/stable/
@@ Line 86: / Line 94: @@
   <nowiki>
-kubectl label nodes <gpu-node-name> hardware-type=NVIDIAGPU
+MINIONS=$(kubectl get nodes -l node-role.kubernetes.io/node= -o jsonpath='{.items[].metadata.name}')
 </nowiki>
 and deploy the explorer:

@@ Line 40: / Line 40: @@
 git clone https://github.com/NVIDIA/deepops
 cd deepops/
 kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.12/nvidia-device-plugin.yml
 kubectl apply -f tests/gpu-test-job.yml

@@ Line 13: / Line 13: @@
   <nowiki>
-$ openstack image set --accept ubuntu-software-config-dgx
+$ openstack image set --accept <uuid-of-the-image>
 </nowiki>
 All of this is done this way for now because the vGPU driver and settings for connecting to our license server are baked into the image.

@@ Line 48: / Line 48: @@
 kubectl exec -ti gpu-pod -- nvidia-smi
 </nowiki>
 == Resources ==
 * https://github.com/NVIDIA/deepops