OpenStack:Magnum

Deploy and test Magnum

Due to a bug in the Queens version of Magnum, use the Rocky version instead. If the platform is not on Rocky, add this to your /etc/kolla/globals.yml:

magnum_tag: "7.0.2"

and deploy.

After the deployment is done, grab a Fedora Atomic image, either version 27 (fedora-atomic-latest) from our platform, https://demo.vscaler.com/ngdetails/OS::Glance::Image/23714b64-6017-4841-87a2-8f31664da5c1, or the latest version directly from the Fedora Atomic project page, https://getfedora.org/en/atomic/download/ (section "Atomic Host Images for Cloud Environments"). Both Fedora Atomic 27 and 29 were tested and confirmed working in Rocky.

Add the image to Glance, making sure the os_distro property is set:

openstack image create --container-format bare --disk-format qcow2 --file <path-to-the-image>.qcow2 --property os_distro=fedora-atomic <image-name>

To add the property to an existing image run:

openstack image set --property os_distro=fedora-atomic <your-fedora-image>

Next up create a cluster template in Magnum. Here is a sample Kubernetes cluster template: https://demo.vscaler.com/ngdetails/OS::Magnum::ClusterTemplate/3e32b6c6-6217-45f3-91c1-4ddfa7675c2b. This template was created using the following command:

openstack coe cluster template create --coe kubernetes --image fedora-atomic-latest --external-network public1 --flavor m1.medium --master-flavor m1.medium --docker-storage-driver overlay --public --floating-ip-disabled kubernetes-fedora

Finally, create a cluster using your template:

openstack coe cluster create --cluster-template <template-name> --master-count 1 --node-count 1 --keypair <your-keypair> <cluster-name>

Adding custom driver

How to create and test a custom Magnum driver - based on the work done on k8_ubuntu_v1.

Build the driver

Clone Magnum's repo from https://github.com/openstack/magnum. Checkout a stable branch that's the same as the version of deployed OpenStack (you should be good with just using master as templates don't change much). Copy an existing driver from either magnum/drivers or contrib/drivers.

Copy one of the *template_def.py available in magnum/drivers/heat/, rename it so it starts with your driver name and replace all references to the old driver. For example:

drivers/heat/k8s_ubuntu_template_def.py

from magnum.drivers.heat import k8s_template_def
...
class UbuntuK8sTemplateDefinition(k8s_template_def.K8sTemplateDefinition):
    """Kubernetes template for an Ubuntu."""

    def __init__(self):
        super(UbuntuK8sTemplateDefinition, self).__init__()
...

In the driver's driver.py script replace references to the old driver, for example this one is for the Ubuntu driver:

drivers/k8s_ubuntu_v1/driver.py
...
from magnum.drivers.k8s_ubuntu_v1 import template_def
...
    @property
    def provides(self):
        return [
            {'server_type': 'vm',
             'os': 'ubuntu',
             'coe': 'kubernetes'},
        ]

    def get_template_definition(self):
        return template_def.UbuntuK8sTemplateDefinition()
...

There will also be references to the old driver in template_def.py, for example:

drivers/k8s_ubuntu_v1/template_def.py
...
class UbuntuK8sTemplateDefinition(kctd.UbuntuK8sTemplateDefinition):
    """Kubernetes template for an Ubuntu VM."""
...

Add your driver to the list of magnum.drivers in setup.cfg

setup.cfg
...
magnum.drivers =
...
    k8s_ubuntu_v1 = magnum.drivers.k8s_ubuntu_v1.driver:Driver
...

Now for Heat templates in the templates directory of the driver: we want to make them as simple as possible for now. There shouldn't be a need to kubecluster.yaml, at least not initially. In kubemaster.yaml remove all OS::Heat::SoftwareConfig objects from the software configs section of the template and only leave write_heat_params and wc_notify. Same for kubeminion.yaml. Remove scripts from templates/fragments directory - all but fragments/write-heat-params-master.yaml and fragments/write-heat-params-minion.yaml.

Your minimal wc_notify should look like this one:

  wc_notify:
    type: OS::Heat::SoftwareConfig
    properties:
      group: ungrouped
      config:
        str_replace:
          template: |
            #!/bin/bash -v
            wc_notify --data-binary '{"status": "SUCCESS"}'
          params:
            wc_notify: {get_attr: [master_wait_handle, curl_cli]}

Make sure that under software configs there is this MultipartMime:

  kube_master_init:
    type: OS::Heat::MultipartMime
    properties:
      parts:
        - config: {get_resource: write_heat_params}
        - config: {get_resource: wc_notify}

(this is for masters, use kube_minion_init for the minions)

Test the driver

You can build a whole Magnum from the git repo by creating a virutalenv and running python setup.py install. And then run magnum-api and magnum-conductor that you get from the installation.

If you already have a working OpenStack cluster with Magnum deployed with Kolla, it's easier to use it as a test environment by following these steps:

(do this once) Add your driver to the [magnum.drivers] list in package entry points:

docker cp magnum_api:/usr/lib/python2.7/site-packages/magnum-6.2.0-py2.7.egg-info/entry_points.txt ~/
vim ~/entry_points.txt
# Add your driver to [magnum.drivers] then proceed
[root@vscaler-vgpu ~]# docker cp ~/entry_points.txt magnum_conductor:/usr/lib/python2.7/site-packages/magnum-6.2.0-py2.7.egg-info/entry_points.txt
[root@vscaler-vgpu ~]# docker cp ~/entry_points.txt magnum_api:/usr/lib/python2.7/site-packages/magnum-6.2.0-py2.7.egg-info/entry_points.txt

If you've changed heat/*template_def.py, copy it to both containers:

docker cp ~/magnum/magnum/drivers/heat/k8s_ubuntu_template_def.py magnum_conductor:/usr/lib/python2.7/site-packages/magnum/drivers/heat/
docker cp ~/magnum/magnum/drivers/heat/k8s_ubuntu_template_def.py magnum_api:/usr/lib/python2.7/site-packages/magnum/drivers/heat/

Copy your driver to /usr/lib/python2.7/site-packages/magnum/drivers/ for both API and conductor containers like so:

docker cp ~/magnum/magnum/drivers/k8s_ubuntu_v1 magnum_conductor:/usr/lib/python2.7/site-packages/magnum/drivers/
docker cp ~/magnum/magnum/drivers/k8s_ubuntu_v1 magnum_api:/usr/lib/python2.7/site-packages/magnum/drivers/

Restart both Magnum containers:

docker restart magnum_api
docker restart magnum_conductor

The 2 last steps should be repeated every time you make changes to the driver.

To test the driver works with Magnum, add appropriate os_distro property to the image you want to use, for example

openstack image set --property os_distro=ubuntu ubuntu-software-config-dgx

then create a simple Magnum cluster template and launch a cluster specifying the COE and the other parameters.

Going further

Send data from host to Heat

You can send back a JSON with the "data" key containing a string parameter, for example a variable:

wc_notify --data-binary '{"status": "SUCCESS", "id": "token", "data": "'$TOKEN'"}'

This variable can then be put in outputs of the template like so:

outputs:
...
  kubeadm_token:
    value: { 'Fn::Select': ['token', {get_attr: [master_wait_condition, data] }] }

and used in the kubecluster.yaml like this:

kubeadm_token: {get_attr: [kube_masters, resource.0.kubeadm_token]}

This allows you to pass data from a master to minions.

Resources

https://docs.openstack.org/heat/queens/template_guide/software_deployment.html

Baking heat components into the image

TODO

User guides

How to create a cluster template and use it to deploy Kubernetes on vGPU instances: http://wiki.bostonlabs.co.uk/w/index.php?title=DeepOps_on_OpenStack_POC#DeepOps_on_vScaler_POC