OpenStack:Magnum - Revision history

Mariusz: Add docs on deploying and testing Magnum

2019-03-19T13:52:40Z

Add docs on deploying and testing Magnum

Mariusz: Add link to DeepOps POC

2019-01-08T11:34:48Z

Add link to DeepOps POC

Mariusz: Fix copy-paste typo

2018-12-20T17:41:51Z

Fix copy-paste typo

Mariusz: Add description on creating and testing custom Magnum drivers

2018-11-30T15:53:10Z

Add description on creating and testing custom Magnum drivers

New page

== Adding custom driver ==
How to create and test a custom Magnum driver - based on the work done on <code>k8_ubuntu_v1</code>.

=== Build the driver ===
Clone Magnum's repo from https://github.com/openstack/magnum. Checkout a stable branch that's the same as the version of deployed OpenStack (you should be good with just using master as templates don't change much).
Copy an existing driver from either <code>magnum/drivers</code> or <code>contrib/drivers</code>.

Copy one of the <code>*template_def.py</code> available in <code>magnum/drivers/heat/</code>, rename it so it starts with your driver name and replace all references to the old driver. For example:

<nowiki>
drivers/heat/k8s_ubuntu_template_def.py

from magnum.drivers.heat import k8s_template_def
...
class UbuntuK8sTemplateDefinition(k8s_template_def.K8sTemplateDefinition):
"""Kubernetes template for an Ubuntu."""

def __init__(self):
super(UbuntuK8sTemplateDefinition, self).__init__()
...
</nowiki>
In the driver's <code>driver.py</code> script replace references to the old driver, for example this one is for the Ubuntu driver:

<nowiki>
drivers/k8s_ubuntu_v1/driver.py
...
from magnum.drivers.k8s_ubuntu_v1 import template_def
...
@property
def provides(self):
return [
{'server_type': 'vm',
'os': 'ubuntu',
'coe': 'kubernetes'},
]

def get_template_definition(self):
return template_def.UbuntuK8sTemplateDefinition()
...
</nowiki>
There will also be references to the old driver in <code>template_def.py</code>, for example:
<nowiki>
drivers/k8s_ubuntu_v1/template_def.py
...
class UbuntuK8sTemplateDefinition(kctd.UbuntuK8sTemplateDefinition):
"""Kubernetes template for an Ubuntu VM."""
...
</nowiki>
Add your driver to the list of <code>magnum.drivers</code> in <code>setup.cfg</code>
<nowiki>
setup.cfg
...
magnum.drivers =
...
k8s_ubuntu_v1 = magnum.drivers.k8s_ubuntu_v1.driver:Driver
...
</nowiki>

Now for Heat templates in the <code>templates</code> directory of the driver: we want to make them as simple as possible for now.
There shouldn't be a need to <code>kubecluster.yaml</code>, at least not initially.
In <code>kubemaster.yaml</code> remove all <code>OS::Heat::SoftwareConfig</code> objects from the <code>software configs</code> section of the template and only leave <code>write_heat_params</code> and <code>wc_notify</code>. Same for <code>kubeminion.yaml</code>.
Remove scripts from <code>templates/fragments</code> directory - all but <code>fragments/write-heat-params-master.yaml</code> and <code>fragments/write-heat-params-minion.yaml</code>.

Your minimal <code>wc_notify</code> should look like this one:
<nowiki>
wc_notify:
type: OS::Heat::SoftwareConfig
properties:
group: ungrouped
config:
str_replace:
template: |
#!/bin/bash -v
wc_notify --data-binary '{"status": "SUCCESS"}'
params:
wc_notify: {get_attr: [master_wait_handle, curl_cli]}
</nowiki>

Make sure that under software configs there is this <code>MultipartMime</code>:

<nowiki>
kube_master_init:
type: OS::Heat::MultipartMime
properties:
parts:
- config: {get_resource: write_heat_params}
- config: {get_resource: wc_notify}
</nowiki>
(this is for masters, use <code>kube_minion_init</code> for the minions)

=== Test the driver ===
You can build a whole Magnum from the git repo by creating a virutalenv and running <code>python setup.py install</code>. And then run <code>magnum-api</code> and <code>magnum-conductor</code> that you get from the installation.

If you already have a working OpenStack cluster with Magnum deployed with Kolla, it's easier to use it as a test environment by following these steps:

* (do this once) Add your driver to the <code>[magnum.drivers]</code> list in package entry points:
<nowiki>
docker cp magnum_api:/usr/lib/python2.7/site-packages/magnum-6.2.0-py2.7.egg-info/entry_points.txt ~/
vim ~/entry_points.txt
# Add your driver to [magnum.drivers] then proceed
[root@vscaler-vgpu ~]# docker cp ~/entry_points.txt magnum_conductor:/usr/lib/python2.7/site-packages/magnum-6.2.0-py2.7.egg-info/entry_points.txt
[root@vscaler-vgpu ~]# docker cp ~/entry_points.txt magnum_api:/usr/lib/python2.7/site-packages/magnum-6.2.0-py2.7.egg-info/entry_points.txt
</nowiki>
* If you've changed <code>heat/*template_def.py</code>, copy it to both containers:
<nowiki>
docker cp ~/magnum/magnum/drivers/heat/k8s_ubuntu_template_def.py magnum_conductor:/usr/lib/python2.7/site-packages/magnum/drivers/heat/
docker cp ~/magnum/magnum/drivers/heat/k8s_ubuntu_template_def.py magnum_conductor:/usr/lib/python2.7/site-packages/magnum/drivers/heat/
</nowiki>
* Copy your driver to <code>/usr/lib/python2.7/site-packages/magnum/drivers/</code> for both API and conductor containers like so:
<nowiki>
docker cp ~/magnum/magnum/drivers/k8s_ubuntu_v1 magnum_conductor:/usr/lib/python2.7/site-packages/magnum/drivers/
docker cp ~/magnum/magnum/drivers/k8s_ubuntu_v1 magnum_api:/usr/lib/python2.7/site-packages/magnum/drivers/
</nowiki>
* Restart both Magnum containers:
<nowiki>
docker restart magnum_api
docker restart magnum_conductor
</nowiki>

The 2 last steps should be repeated every time you make changes to the driver.

To test the driver works with Magnum, add appropriate <code>os_distro</code> property to the image you want to use, for example
<nowiki>
openstack image set --property os_distro=ubuntu ubuntu-software-config-dgx
</nowiki>
then create a simple Magnum cluster template and launch a cluster specifying the COE and the other parameters.

=== Going further ===

==== Send data from host to Heat ====
You can send back a JSON with the "data" key containing a string parameter, for example a variable:

<nowiki>
wc_notify --data-binary '{"status": "SUCCESS", "id": "token", "data": "'$TOKEN'"}'
</nowiki>
This variable can then be put in <code>outputs</code> of the template like so:
<nowiki>
outputs:
...
kubeadm_token:
value: { 'Fn::Select': ['token', {get_attr: [master_wait_condition, data] }] }
</nowiki>
and used in the <code>kubecluster.yaml</code> like this:
<nowiki>
kubeadm_token: {get_attr: [kube_masters, resource.0.kubeadm_token]}
</nowiki>
This allows you to pass data from a master to minions.

=== Resources ===
# https://docs.openstack.org/heat/queens/template_guide/software_deployment.html

== Baking heat components into the image ==
TODO

← Older revision		Revision as of 13:52, 19 March 2019
Line 1:		Line 1:
		+	== Deploy and test Magnum ==
		+	Due to a bug in the Queens version of Magnum, use the Rocky version instead. If the platform is not on Rocky, add this to your <code>/etc/kolla/globals.yml</code>:
		+	<nowiki>
		+	magnum_tag: "7.0.2"
		+	</nowiki>
		+	and deploy.
		+
		+	After the deployment is done, grab a Fedora Atomic image, either version 27 (<code>fedora-atomic-latest</code>) from our platform, https://demo.vscaler.com/ngdetails/OS::Glance::Image/23714b64-6017-4841-87a2-8f31664da5c1, or the latest version directly from the Fedora Atomic project page, https://getfedora.org/en/atomic/download/ (section "Atomic Host Images for Cloud Environments"). Both Fedora Atomic 27 and 29 were tested and confirmed working in Rocky.
		+
		+	Add the image to Glance, making sure the <code>os_distro</code> property is set:
		+	<nowiki>
		+	openstack image create --container-format bare --disk-format qcow2 --file <path-to-the-image>.qcow2 --property os_distro=fedora-atomic <image-name>
		+	</nowiki>
		+	To add the property to an existing image run:
		+	<nowiki>
		+	openstack image set --property os_distro=fedora-atomic <your-fedora-image>
		+	</nowiki>
		+
		+	Next up create a cluster template in Magnum. Here is a sample Kubernetes cluster template: https://demo.vscaler.com/ngdetails/OS::Magnum::ClusterTemplate/3e32b6c6-6217-45f3-91c1-4ddfa7675c2b. This template was created using the following command:
		+	<nowiki>
		+	openstack coe cluster template create --coe kubernetes --image fedora-atomic-latest --external-network public1 --flavor m1.medium --master-flavor m1.medium --docker-storage-driver overlay --public --floating-ip-disabled kubernetes-fedora
		+	</nowiki>
		+
		+	Finally, create a cluster using your template:
		+	<nowiki>
		+	openstack coe cluster create --cluster-template <template-name> --master-count 1 --node-count 1 --keypair <your-keypair> <cluster-name>
		+	</nowiki>
		+
	== Adding custom driver ==		== Adding custom driver ==
	How to create and test a custom Magnum driver - based on the work done on <code>k8_ubuntu_v1</code>.		How to create and test a custom Magnum driver - based on the work done on <code>k8_ubuntu_v1</code>.

@@ Line 149: / Line 149: @@
 === Resources ===
 # https://docs.openstack.org/heat/queens/template_guide/software_deployment.html
 == Baking heat components into the image ==
 TODO

@@ Line 105: / Line 105: @@
   <nowiki>
 docker cp ~/magnum/magnum/drivers/heat/k8s_ubuntu_template_def.py magnum_conductor:/usr/lib/python2.7/site-packages/magnum/drivers/heat/
-docker cp ~/magnum/magnum/drivers/heat/k8s_ubuntu_template_def.py magnum_conductor:/usr/lib/python2.7/site-packages/magnum/drivers/heat/
+docker cp ~/magnum/magnum/drivers/heat/k8s_ubuntu_template_def.py magnum_api:/usr/lib/python2.7/site-packages/magnum/drivers/heat/
 </nowiki>
 * Copy your driver to <code>/usr/lib/python2.7/site-packages/magnum/drivers/</code> for both API and conductor containers like so: