Difference between revisions of "OpenStack:Ironic"

From Define Wiki
Jump to navigation Jump to search
m (→‎Troubleshooting: Output from `docker ps` redacted for brewity)
m (fix raid node with 1 remaining disk)
 
(23 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
== Update ==
 +
much of this is still pure gold in antelope (9 releases on) updates shortly
 +
 +
scroll down for "hacking in a new switch driver"
 +
 +
== Reference diagram ==
 +
Taken from docs for Genomics England:
 +
 +
[[File:Genomics_England_Ironic_diagram.png|1024px]]
 
== Recommended kolla-ansible overrides ==
 
== Recommended kolla-ansible overrides ==
 
NOTE: Stein or newer is recommended for this, but when multitenancy won't be enabled, Rocky will also work.
 
NOTE: Stein or newer is recommended for this, but when multitenancy won't be enabled, Rocky will also work.
Line 32: Line 41:
 
Also set "sshkey" to the public key of your deploy node or headnode to enable SSH access to the node that's being deployed.
 
Also set "sshkey" to the public key of your deploy node or headnode to enable SSH access to the node that's being deployed.
  
−
=== Multitenancy
+
Add these overrides to <code>/etc/kolla/config/neutron/ml2_conf.ini</code> to make sure that Neutron's ML2 agent can create VLAN networks on Ironic's bridge:
 +
 
 +
<nowiki>
 +
[ml2_type_vlan]
 +
network_vlan_ranges = physnet1:300:400
 +
</nowiki>
 +
Replace <code>physnet1</code> with the provider interface that's set up on the interface leading to baremetal nodes and the numbers -- with VLAN ID range that you want your VLAN networks to be in (or simply remove the numbers to allow for all VLAN IDs to be used).
 +
 
 +
=== Multitenancy ===
 
Deployments with multitenancy require a few more overrides.
 
Deployments with multitenancy require a few more overrides.
  
Line 40: Line 57:
 
[DEFAULT]
 
[DEFAULT]
 
enabled_network_interfaces = noop,flat
 
enabled_network_interfaces = noop,flat
−
</nowkik>
+
</nowiki>
 
with
 
with
  
Line 77: Line 94:
 
</nowiki>
 
</nowiki>
 
Here <code>mechanism_drivers</code> adds <code>genericswitch</code> to the list of enabled drivers, which is then configured in the <code>[genericswitch:]</code> subsection. The existing netmiko driver for Cisco IOS is used and the driver excepts SSH access to be configured on the switch. Notice that password specified in this config is plain text (not encrypted). Lastly, a range of VLANs is specified on the provider interface that connects to baremetal nodes, so only a subset of VLAN IDs is used for tenant networks.
 
Here <code>mechanism_drivers</code> adds <code>genericswitch</code> to the list of enabled drivers, which is then configured in the <code>[genericswitch:]</code> subsection. The existing netmiko driver for Cisco IOS is used and the driver excepts SSH access to be configured on the switch. Notice that password specified in this config is plain text (not encrypted). Lastly, a range of VLANs is specified on the provider interface that connects to baremetal nodes, so only a subset of VLAN IDs is used for tenant networks.
−
 
  
 
== Recommended post-deployment configuration ==
 
== Recommended post-deployment configuration ==
Line 94: Line 110:
 
$ sudo make
 
$ sudo make
 
</nowiki>
 
</nowiki>
 +
Resulting kernel image and ramdisk will be available in the <code>UPLOAD</code> subdirectory.
 +
 +
NOTE: Ready to use deploy images are available here: http://185.93.31.43/ironic-images/
  
−
Transfer/mount images from the <code>UPLOAD</code> subdirectory to your controller/headnode and add them to Glance:
+
Transfer/mount deploy images and add them to Glance:
  
 
  <nowiki>
 
  <nowiki>
Line 101: Line 120:
 
$ openstack image create --public --container-format ari --disk-format ari --file ~/coreos_production_pxe_image-oem.cpio.gz ironic-deploy_ramdisk
 
$ openstack image create --public --container-format ari --disk-format ari --file ~/coreos_production_pxe_image-oem.cpio.gz ironic-deploy_ramdisk
 
</nowiki>
 
</nowiki>
−
Also add regular operating system images - no special options are required here and regular cloud images can be used. Here is an example for a CentOS image:
+
Also add regular operating system images - no special options are required here and regular cloud images can be used. Here is an example for a CentOS image downloaded from https://cloud.centos.org/centos/7/images/:
  
 
  <nowiki>
 
  <nowiki>
 
$ openstack image create --public --container-format bare --disk-format qcow2 --file ~/CentOS-7-x86_64-GenericCloud-1907.qcow2 centos7-1907
 
$ openstack image create --public --container-format bare --disk-format qcow2 --file ~/CentOS-7-x86_64-GenericCloud-1907.qcow2 centos7-1907
 
</nowiki>
 
</nowiki>
 +
 +
NOTE: For security reasons vanilla CentOS 7 cloud images don't have root password set. However, being able to log in as root to baremetal instances is very useful for debugging problems caused, for example, by network interfaces being down inside of the instance. Images available on http://185.93.31.43/ironic-images/ with "rootpass" in their names can be used for baremetal provisioning in the initial phase of testing a new Ironic deployment (root password for these images is <code>vScaler2019?!</code>). Then, when all the initial problems are resolved, it's recommended to switch to using vanilla cloud images instead.
  
 
Create a flavour for your baremetal nodes:
 
Create a flavour for your baremetal nodes:
Line 121: Line 142:
 
  <nowiki>
 
  <nowiki>
 
$ openstack network create public304 --provider-physical-network physnet1 --provider-network-type vlan --provider-segment 304
 
$ openstack network create public304 --provider-physical-network physnet1 --provider-network-type vlan --provider-segment 304
−
$ openstack subnet create --dhcp --allocation-pool start=10.6.44.101,end=10.6.47.254 --network public304 --subnet-range 10.6.44.0/22 --gateway 10.6.44.1 public304-subnet
+
$ openstack subnet create --dhcp --allocation-pool start=10.6.44.101,end=10.6.44.254 --network public304 --subnet-range 10.6.44.0/24 --gateway 10.6.44.1 public304-subnet
 
</nowiki>
 
</nowiki>
  
Line 140: Line 161:
  
 
  <nowiki>
 
  <nowiki>
−
$ openstack server create --image <image-to-provision-node-with> --flavor baremetal.small --security-group ping-and-ssh --key-name mykey --network <name-of-the-provisioning-network> <instance-name>
+
$ openstack server create --image <image-to-provision-node-with> --flavor baremetal.small --key-name <your-keypair> --network <name-of-the-provisioning-network> <instance-name>
 +
</nowiki>
 +
 
 +
The images available on our webserver come with a service exposing journald messages out of the instance though an HTTP endpoint. You can see the messages by running this command from your deployment machine:
 +
 
 +
<nowiki>
 +
curl 'http://<provisioning-IP-assigned-to-the-instance>:19531/entries?follow'
 
</nowiki>
 
</nowiki>
 +
Where the instance IP is the one assigned to the instance by the Neutron DHCP server.
 +
  
−
=== Multitenancy
+
=== Multitenancy ===
 
When adding a node to Ironic, use <code>--network-interface neutron</code> instead of <code>--network-interface flat</code>.
 
When adding a node to Ironic, use <code>--network-interface neutron</code> instead of <code>--network-interface flat</code>.
  
Line 165: Line 194:
 
# https://docs.openstack.org/ironic/stein/admin/multitenancy.html
 
# https://docs.openstack.org/ironic/stein/admin/multitenancy.html
  
 +
== Software RAID ==
 +
Support for software RAID has been added in the Train release, so if you're on an older release, you'll have to add the following parameters to your <code>globals.yml</code>:
 +
 +
<nowiki>
 +
ironic_tag: "train"
 +
ironic_install_type: "source"
 +
</nowiki>
 +
and then run a kolla-ansible deploy with the "ironic" tag.
 +
Make sure that support for software RAID is present in the revision of the Ironic Python Agent that you're using for building deploy images (or use images available on http://185.93.31.43/ironic-images/).
 +
 +
Reconfigure your Ironic node(s) for software RAID:
 +
 +
<nowiki>
 +
# openstack baremetal node set <node-name> --raid-interface agent
 +
# cat ~/raid_1_basic.json
 +
{
 +
"logical_disks": [
 +
                  {
 +
                    "size_gb": "MAX",
 +
                    "raid_level": "1",
 +
                    "controller": "software"
 +
                  }
 +
                  ]
 +
}
 +
 +
# openstack baremetal node set <node-name> --target-raid-config ~/raid_1_basic.json
 +
</nowiki>
 +
Feel free to edit the JSON file (the above is just a basic one for setting up a RAID-1 on 2 disks), but bear in mind limitations of the software RAID driver (check links in References).
 +
Here is a more advanced config - with 2 RAID-1 arrays on 2 disks - the first array (the one of size 50GB) is where the operating system will go:
 +
 +
<nowiki>
 +
{
 +
"logical_disks": [
 +
                  {
 +
                    "size_gb": "50",
 +
                    "raid_level": "1",
 +
                    "controller": "software",
 +
                    "is_root_volume": true
 +
                  }
 +
                  {
 +
                    "size_gb": "MAX",
 +
                    "raid_level": "1",
 +
                    "controller": "software"
 +
                  }
 +
                  ]
 +
}
 +
</nowiki>
 +
 +
Now the node is ready for RAID creation. In Ironic this is done during the "cleaning" stage.
 +
 +
<nowiki>
 +
# openstack baremetal node manage <node-name>
 +
# cat ~/raid_cleaning_steps.json
 +
[{
 +
  "interface": "raid",
 +
  "step": "delete_configuration"
 +
},
 +
{
 +
  "interface": "deploy",
 +
  "step": "erase_devices_metadata"
 +
},
 +
{
 +
  "interface": "raid",
 +
  "step": "create_configuration"
 +
}]
 +
# openstack baremetal node clean <node-name> --clean-steps ~/raid_cleaning_steps.json
 +
</nowiki>
 +
There is no need to modify this "cleaning steps" JSON, it should work for any RAID configuration.
 +
 +
This should put the node in the "cleaning" state. Wait for the node get back to the "manageable" state (feel free to fire up the virtual console of this node to keep an eye on the process). Check ironic-conductor logs for errors if the node gets into "clean failed" instead.
 +
 +
When the node is "manageable" again, add this root_device hint specifying your array as the root device and give the node back to Nova:
 +
 +
<nowiki>
 +
# openstack baremetal node set <node-name> --property 'root_device={"name": "/dev/md127"}'
 +
# openstack baremetal node provide <node-name>
 +
</nowiki>
 +
 +
To provision baremetal instances capable of booting from a software RAID, you'll need your target image (the one that you're flashing the system with) to include mdadm. This is already done in Ubuntu cloud images (at least in Xenial and Bionic), but requires baking an new image when using CentOS 7 cloud images (at least for build 1907 and older).
 +
Here are all the mdadm-related commands do use when baking a custom image for CentOS 7:
 +
 +
<nowiki>
 +
# yum install mdadm -y
 +
# vi /etc/dracut.conf
 +
# grep hostonly /etc/dracut.conf
 +
hostonly="no"
 +
# dracut -f --add mdraid --add-drivers xfs -v
 +
</nowiki>
 +
When baking this image you can also add in your custom cloud-init networking config and set root password if you want.
 +
 +
Instead of baking your own images, you can use qcow2 images available here: http://185.93.31.43/ironic-images/
 +
 +
Finally, launch an instance off of this new image and using the baremetal flavour.
 +
 +
=== References ===
 +
# https://docs.openstack.org/ironic/train/admin/raid.html
 +
# https://techblog.web.cern.ch/techblog/post/ironic_software_raid/
  
 
== Troubleshooting ==
 
== Troubleshooting ==
Line 239: Line 365:
 
</nowiki>
 
</nowiki>
 
This is normal and nothing to worry about. {TODO: Why is this port marked as DOWN?}
 
This is normal and nothing to worry about. {TODO: Why is this port marked as DOWN?}
 +
 +
If you know your node will end in ERROR state and you want to peek inside the deploy image during provisioning, here is how to do this:
 +
 +
# Launch a baremetal instance on the node.
 +
# As soon as the instance get scheduled to the node (check that instance UUID shows up on the output from <code>openstack baremetal node list</code>), put your node in maintenance mode by running <code>openstack baremetal node maintenance set <node-name></code>.
 +
# Get into the console of the node (for example using SOL) and stop the IPA service: <code>systemctl stop ironic-python-agent</code>.
 +
# Check <code>journald</code> logs, make changes, etc. In CoreOS-based deploy images all IPA-related files are in <code>/opt/ironic-python-agent</code>.
 +
# When you're done, remove your node from maintenance (<code>openstack baremetal node maintenance unset <node-name></code>) and start the IPA service (<code>systemctl start ironic-python-agent</code>).
 +
 +
=== Common error messages ===
 +
<nowiki>
 +
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: 2019-12-13 13:30:06.652 1699 ERROR root [-] Command execution error: Command execution failed: Installing GRUB2 boot loader to device /dev/md127 failed with Unexpected error while running command.
 +
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Command: chroot /tmp/tmpbMmsW1 /bin/sh -c "grub2-install /dev/sda"
 +
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Exit code: 1
 +
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Stdout: u''
 +
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Stderr: u"Installing for i386-pc platform.\ngrub2-install: error: disk `md127,1' not found.\n".: CommandExecutionError: Command execution failed: Installing GRUB2 boot loader to device /dev/md127 failed with Unexpected error while running command.
 +
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Command: chroot /tmp/tmpbMmsW1 /bin/sh -c "grub2-install /dev/sda"
 +
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Exit code: 1
 +
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Stdout: u''
 +
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Stderr: u"Installing for i386-pc platform.\ngrub2-install: error: disk `md127,1' not found.\n".
 +
</nowiki>
 +
Source: ironic-conductor logs
 +
 +
Your target image doesn't support software RAID. Use/bake an image with the mdadm package installed and mdadm support in the initrd enabled.
 +
 +
------------------------------------------------------------------------------
 +
<nowiki>
 +
2019-12-13 13:04:53.424 6 ERROR ironic.drivers.modules.agent_base_vendor [req-e55e913d-1216-4072-b1cc-94cdca5b72b1 - - - - -] Asynchronous exception: Node failed to deploy. Exception: Deploy failed for instance 66956849-b57c-49b7-ac78-9a0b4f2117b7. Error: Unexpected error while running command.
 +
Command: sudo ironic-rootwrap /etc/ironic/rootwrap.conf dd if=/var/lib/ironic/images/baf22641-4698-450f-a26a-ecca012b4a24/disk of=/dev/disk/by-path/ip-192.168.202.51:3260-iscsi-iqn.2008-10.org.openstack:baf22641-4698-450f-a26a-ecca012b4a24-lun-1 bs=1M oflag=direct
 +
Exit code: 1
 +
Stdout: u''
 +
Stderr: u"/bin/dd: error writing '/dev/disk/by-path/ip-192.168.202.51:3260-iscsi-iqn.2008-10.org.openstack:baf22641-4698-450f-a26a-ecca012b4a24-lun-1': No space left on device\n8192+0 records in\n8191+0 records out\n8588886016 bytes (8.6 GB) copied, 17.2107 s, 499 MB/s\n" for node baf22641-4698-450f-a26a-ecca012b4a24: InstanceDeployFailure: Deploy failed for instance 66956849-b57c-49b7-ac78-9a0b4f2117b7. Error: Unexpected error while running command.
 +
</nowiki>
 +
Source: ironic-conductor logs
 +
 +
Ironic Python Agents tries to dd your image to a partition instead of a block device. Either clean (<code>openstack baremetal node clean...</code>) the node to get rid of the partition or set the <code>device_root</code> hint [2] on the node to your block device (for example <code>/dev/md127</code> for software RAID).
 +
 +
------------------------------------------------------------------------------
 +
<nowiki>
 +
2019-12-16 09:57:00.598 6 ERROR ironic.drivers.modules.agent_base_vendor [-] Agent returned error for clean step {u'interface': u'raid', u'priority': 0, u'step': u'create_configuration', u'abortable': False} on node c3ebcf94-a4db-49bf-927e-1bd186b01a71 : {u'message': u"Clean step failed: Error performing clean_step create_configuration: Software RAID caused unknown error: Failed to create partitions on /dev/sda: Unexpected error while running command.\nCommand: parted /dev/sda -s -a optimal -- mkpart primary 2048s -1\nExit code: 1\nStdout: u''\nStderr: u'Error: Partition(s) 1 on /dev/sda have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes.\\n'", u'code': 500, u'type': u'CleaningError', u'details': u"Error performing clean_step create_configuration: Software RAID caused unknown error: Failed to create partitions on /dev/sda: Unexpected error while running command.\nCommand: parted /dev/sda -s -a optimal -- mkpart primary 2048s -1\nExit code: 1\nStdout: u''\nStderr: u'Error: Partition(s) 1 on /dev/sda have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes.\\n'"}.
 +
</nowiki>
 +
Source: ironic-conductor logs
 +
 +
Cleanup has failed. Run the <code>openstack baremetal node clean...</code> command again.
 +
 +
------------------------------------------------------------------------------
 +
<nowiki>
 +
2019-10-22 12:32:41.312 6 ERROR ironic.conductor.utils [req-37fbce85-8a88-4a61-9a4a-22d6f37dc398 - - - - -] Asynchronous exception: Node failed to deploy. Exception: Connection to agent failed: Failed to connect to the agent running on node d4b3434e-047c-4715-8e89-dfd849477cdc for invoking command image.install_bootloader. Error: HTTPConnectionPool(host='172.16.100.97', port=9999): Read timed out. (read timeout=60) for node: AgentConnectionFailed: Connection to agent failed: Failed to connect to the agent running on node d4b3434e-047c-4715-8e89-dfd849477cdc for invoking command image.install_bootloader. Error: HTTPConnectionPool(host='172.16.100.97', port=9999): Read timed out. (read timeout=60)
 +
</nowiki>
 +
Source: ironic-conductor logs
 +
 +
Root cause of this error is unknown, but the problem disappears after a short while.
 +
 +
------------------------------------------------------------------------------
 +
<nowiki>
 +
# openstack server show test_ironic | grep fault
 +
| fault                              | {u'message': u"Host 'controller003-ironic' is not mapped to any cell", u'code': 400, u'created': u'2019-12-11T12:10:15Z'} |
 +
</nowiki>
 +
Source: the openstack CLI
 +
 +
This can happen if <code>nova-compute-ironic</code> containers are not up when you add baremetal nodes to Ironic. Make sure these containers are fully functional and then run <code>nova-manage cell_v2 map_cell_and_hosts</code> from one of the <nowiki>nova-api</nowiki> container to fix the problem.
 +
 +
------------------------------------------------------------------------------
 +
<nowiki>
 +
CLIENT MADDR: XX XX XX XX XX XX  GUID: 00000000 0000 0000 0000 002590F950BE
 +
CLIENT IP: 10.6.44.249  MASK: 255.255.255.0  DHCP IP: 10.6.44.101
 +
GATEWAY IP: 10.6.44.1
 +
PXE-E32: TFTP open timeout
 +
PXE-M0F: Exiting Intel Boot Agent.
 +
</nowiki>
 +
The above message seen on the remote console indicates problems with L3 connection to the TFTP server. Make sure your switch knows how to get to the IP specified in <code>my_ip</code> or <code>tftp_server</code> parameter in Ironic conductor's config (in this example TFTP server has the 192.168.88.233 IP). Also, check if you can reach the IP assigned by Neutron DHCP to this node (in this example, 10.6.44.249) from the machine on which the TFTP server runs. If the default route on this machine doesn't go though the Ironic provisioning switch, you will need to add static routes for all your provisioning networks. For example:
 +
 +
<nowiki>
 +
# ip route add 10.6.44.0/24 via 192.168.88.253
 +
</nowiki>
  
 
=== References ===
 
=== References ===
 
# https://cloudinit.readthedocs.io/en/18.4/topics/network-config-format-v2.html
 
# https://cloudinit.readthedocs.io/en/18.4/topics/network-config-format-v2.html
 +
# https://docs.openstack.org/ironic/train/install/advanced.html#specifying-the-disk-for-deployment-root-device-hints
  
 
== Ironic POC basic test environment ==
 
== Ironic POC basic test environment ==
Line 435: Line 637:
  
 
Network interface config is exactly the same as in the previous iteration of the deployment (without multi-tenancy).
 
Network interface config is exactly the same as in the previous iteration of the deployment (without multi-tenancy).
 +
 +
=== Hacking in a new networking generic switch driver ===
 +
 +
every now and then for a POC you will come across a switch which isn't supported. This is NOT the end of the world as long as there is a way to ssh in and run a list of cli commands to control vlans.
 +
 +
the way I did tis was to stop the neutron_server container, and corresponding systemd service (thanks anelope), on all but one controller. The reason for this is so we don't have to inject 3x.
 +
 +
the next step is to pick victim driver to overwrite. DO NOT pick the netmiko fake driver!!! This one exits really early. Also DO NOT waste time trying to inject an new one uunless you understand pyhon module packaging a lot better than I.
 +
 +
I picked the pluribus driver as I know it works and there is zero chance of me encountering another pluribus switch in this POC. Probably best not to pick a common cisco in case you forget and try to use it. For those who care you can see the original pluribus driver at the following link https://opendev.org/openstack/networking-generic-switch/src/branch/master/networking_generic_switch/devices/netmiko_devices/pluribus.py.
 +
 +
here is my updated driver for a FS switch in a file called pluribus.py in my local dir
 +
 +
<pre>
 +
# Copyright 2022 define-technology.
 +
#
 +
#    Licensed under the Apache License, Version 2.0 (the "License"); you may
 +
#    not use this file except in compliance with the License. You may obtain
 +
#    a copy of the License at
 +
#
 +
#        http://www.apache.org/licenses/LICENSE-2.0
 +
#
 +
#    Unless required by applicable law or agreed to in writing, software
 +
#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 +
#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 +
#    License for the specific language governing permissions and limitations
 +
#    under the License.
 +
 +
from networking_generic_switch.devices import netmiko_devices
 +
 +
 +
class Pluribus(netmiko_devices.NetmikoSwitch):
 +
    ADD_NETWORK = (
 +
        'configure',
 +
        'vlan {segmentation_id}',
 +
        'exit'
 +
        'exit'
 +
    )
 +
 +
    DELETE_NETWORK = (
 +
        'configure',
 +
        'no vlan {segmentation_id}',
 +
        'exit'
 +
        'exit'
 +
    )
 +
 +
    PLUG_PORT_TO_NETWORK = (
 +
        'configure',
 +
        'interface {port}',
 +
        'no port hybrid vlan all',
 +
        'port hybrid vlan {segmentation_id} untagged',
 +
        'port hybrid pvid {segmentation_id}',
 +
        'exit'
 +
        'exit'
 +
    )
 +
 +
    DELETE_PORT = (
 +
        'configure',
 +
        'interface {port}',
 +
        'port hybrid pvid 1',
 +
        'no port hybrid vlan all',
 +
        'exit'
 +
        'exit'
 +
    )
 +
    ADD_NETWORK_TO_TRUNK = (
 +
        'configure',
 +
        'interface {port}',
 +
        'port hybrid vlan {segmentation_id} tagged',
 +
        'exit'
 +
        'exit'
 +
    )
 +
 +
    REMOVE_NETWORK_FROM_TRUNK = (
 +
        'configure',
 +
        'interface {port}',
 +
        'no port hybrid vlan {segmentation_id} tagged',
 +
        'exit'
 +
        'exit'
 +
    )
 +
</pre>
 +
 +
this has properly implemented MOST  but not all of the full functionality. I'm missing a error regex so it has no way to know if a command has failed and will just plow on regardless assuming all is good buuuuut who cares about that? I am also missing a save function so the debug logs will end up being full of save not implemented but again it still works so nerrrr. It is REALLY handy to have that add/remove from trunk support as this allows you to add the vlan to uplink ports and your openstack config should be modified to include controllers and hypervisors (the latter only id DVR or similar is enabled).
 +
 +
the next thing we need to do is to find the **right** pluribus.py file (or whatever file you have decided to sacrifice) to overwrite in the neuron server container.
 +
 +
now lets have a look around in there hints are
 +
 +
# nothing with build in the path
 +
# it has always been in the /var/lib/kolla/venv/ in my experience and there should be only the one
 +
when you have found it exit out of the container and copy your file in an restart neutron_server
 +
 +
<pre>
 +
docker exec -it neutron_server bash
 +
find -iname pluribus.py | grep "kolla/venv"
 +
/var/lib/kolla/venv/lib/python3.10/site-packages/networking_generic_switch/devices/netmiko_devices/pluribus.py
 +
</pre>
 +
 +
<pre>
 +
docker cp pluribus.py neutron_server:/var/lib/kolla/venv/lib/python3.10/site-packages/networking_generic_switch/devices/netmiko_devices/
 +
docker restart neutron_server
 +
</pre>
 +
 +
if you have no typos then it should not go into a restart loop and you can try it out in your multi-tenant config
 +
 +
I modified my ml2_conf.ini on the controller NOT IN KOLLA CONIFG here it is in full
 +
<pre>
 +
root@control0002:~/scratch/neutron-ngs-patch# cat /etc/kolla/neutron-server/ml2_conf.ini
 +
[ml2]
 +
type_drivers = flat,vlan,vxlan,geneve
 +
tenant_network_types = geneve
 +
mechanism_drivers = genericswitch,baremetal,ovn
 +
extension_drivers = port_security
 +
 +
[ml2_type_vlan]
 +
network_vlan_ranges = physnet1:102:110,physnet1:200:1200
 +
 +
[ml2_type_flat]
 +
flat_networks = *
 +
 +
[ml2_type_vxlan]
 +
vni_ranges = 1:1000
 +
 +
[ml2_type_geneve]
 +
vni_ranges = 1001:2000
 +
max_header_size = 38
 +
 +
[ovn]
 +
ovn_nb_connection = tcp:192.168.60.11:6641,tcp:192.168.60.12:6641,tcp:192.168.60.13:6641
 +
ovn_sb_connection = tcp:192.168.60.11:6642,tcp:192.168.60.12:6642,tcp:192.168.60.13:6642
 +
ovn_metadata_enabled = True
 +
enable_distributed_floating_ip = True
 +
 +
[genericswitch:switch01]
 +
#device_type = netmiko_fs_n_series
 +
device_type = netmiko_pluribus
 +
ip = 192.168.30.1
 +
ngs_mac_address = 64:9d:99:3a:3a:58
 +
username = admin
 +
password = password
 +
ngs_network_name_format = neutron-{network_id}-{segmentation_id}
 +
ngs_save_configuration = False
 +
ngs_trunk_ports = 25gigaethernet 1/0/14, 25gigaethernet 1/0/15, 25gigaethernet 1/0/17, 25gigaethernet 1/0/18, 25gigaethernet 1/0/20, 25gigaethernet 1/0/21, 25gigaethernet 1/0/27, 10gigaethernet 1/0/58
 +
</pre>
 +
 +
I then created a new vlan provider net and the vlan was added to the switch and all the expected trunk ports were updated!
 +
 +
=== how to do properly install a new switch driver ===
 +
first step make a fork of networking generic switch and push it somewhere reachable for example here:
 +
 +
https://gitlab.define-technology.com/pub/networking-generic-switch/-/compare/stable%2F2023.1...feat-add-fs-switch?from_project_id=159&straight=false
 +
 +
this is the final working driver (for a given definition of working)
 +
 +
this driver overrides the device type in netmiko to use the existing pluribus driver. This is because of two things
 +
# it exists in the standard release
 +
# error checking is not implemented
 +
 +
The latter point is most important with this driver as I don't want to spend ages writing a full netmiko driver for this switch and it's not cisco like enough to allow those to work. It either errors going into or exiting from configure mode. If we use the pluribus driver. It never needs to go into configure mode at all and just runs the commands as is without checking the output. We know they work that is good enough for this.
 +
 +
it does NOT need to be a public repo it can be a private github repo but you need to make sure you can access it using a ssh key and that your key is on the build node or in your ssh agent
 +
 +
finally we need to buid it in to the kolla containers I made a custom kolla-build-neutron.conf file
 +
 +
<pre>
 +
[DEFAULT]
 +
namespace = kolla
 +
base = ubuntu
 +
registry = maas.pxeboot.internal:5000
 +
openstack_release = 2023.1
 +
 +
[neutron-base-plugin-networking-generic-switch]
 +
type = git
 +
location = https://gitlab.define-technology.com/pub/networking-generic-switch.git
 +
reference = feat-add-fs-switch
 +
</pre>
 +
 +
that is it
 +
 +
to build JUST the neutron-server and the prerequisite beutron-base container run
 +
<pre>
 +
kolla-build --config-file ./kolla-build-neutron.conf neutron-base neutron-server
 +
</pre>
 +
 +
then push the resulting images by hand
 +
 +
when you update releases you should update your fork or if you haven't done a terrible job like above push to upstream
 +
 +
==Make a node that was configured in RAID back into a non-RAIDed instance==
 +
<pre>
 +
openstack baremetal node manage compute2
 +
openstack baremetal node set compute2 --raid-interface no-raid
 +
openstack baremetal node set compute2 --property root_device='{"size": "<= 2000", "rotational": false}'
 +
openstack baremetal node maintenance unset compute2
 +
openstack baremetal node provide compute2
 +
</pre>

Latest revision as of 18:42, 21 February 2024

Update

much of this is still pure gold in antelope (9 releases on) updates shortly

scroll down for "hacking in a new switch driver"

Reference diagram

Taken from docs for Genomics England:

Error creating thumbnail: File missing

Recommended kolla-ansible overrides

NOTE: Stein or newer is recommended for this, but when multitenancy won't be enabled, Rocky will also work.

Overrides in /etc/kolla/globals.yml enabling Ironic and its Neutron agent (OpenStack Stein): 

enable_ironic: "yes"
enable_ironic_neutron_agent: "yes"

Overrides in /etc/kolla/config/ironic/ironic-conductor.conf: 

[DEFAULT]
enabled_network_interfaces = noop,flat
default_network_interface = flat

[pxe]
tftp_server = <ip-of-a-dedicated-interface>
pxe_append_params = nofb nomodeset vga=normal console=ttyS1,115200 console=tty0 sshkey="ssh-rsa AAAA..." ipa-debug=1 coreos.autologin

[deploy]
default_boot_option = local

[agent]
deploy_logs_collect = on_failure

[conductor]
clean_callback_timeout = 300

The IP for "tftp_server" can be the same as for the interface on which internal OpenStack APIs are running on the host that's hosting Ironic (only setup with one controller hosting all Ironic components has been tested). Also set "sshkey" to the public key of your deploy node or headnode to enable SSH access to the node that's being deployed.

Add these overrides to /etc/kolla/config/neutron/ml2_conf.ini to make sure that Neutron's ML2 agent can create VLAN networks on Ironic's bridge: 

[ml2_type_vlan]
network_vlan_ranges = physnet1:300:400

Replace physnet1 with the provider interface that's set up on the interface leading to baremetal nodes and the numbers -- with VLAN ID range that you want your VLAN networks to be in (or simply remove the numbers to allow for all VLAN IDs to be used).

Multitenancy

Deployments with multitenancy require a few more overrides.

First, enable the "neutron" network interface by replacing 

[DEFAULT]
enabled_network_interfaces = noop,flat

with 

[DEFAULT]
enabled_network_interfaces = noop,flat,neutron

in /etc/kolla/config/ironic/ironic-conductor.conf. You can also change default_network_interface to neutron if you want the multitenant driver to be the default for all nodes. The network driver can be set on each node individually, so you might as well leave the default to be flat and only enable the driver on selected (or all) nodes.

The "neutron" Ironic driver requires specifying provisioning and cleaning networks upfront in the config file, so add these lines to /etc/kolla/config/ironic/ironic-conductor.conf: 

[neutron]
cleaning_network = <provisioning-network-UUID>
provisioning_network = <provisioning-network-UUID>

and add UUIDs of your provisioning network. The provisioning network should be a network (preferably flat, but can also be VALN tagged) with external router set to a gateway for this network set on the physical switch.

There are also extra overrides for the ML2 driver which should be added to /etc/kolla/config/neutron/ml2_conf.ini. Here is an example for a Supermicro switch (confirmed working on SSE-G24-TG4 and MBM-GEM-004): 

[ml2]
mechanism_drivers = openvswitch,baremetal,l2population,genericswitch

[genericswitch:supermicro_1]
#device_type = netmiko_supermicro
device_type = netmiko_cisco_s300
ip = <IP-for-ssh-access-to-baremetal-switch>
username = <admin-user-on-switch>
password = <admin-password-on-switch>

[ml2_type_vlan]
network_vlan_ranges = physnet1:100:200

Here mechanism_drivers adds genericswitch to the list of enabled drivers, which is then configured in the [genericswitch:] subsection. The existing netmiko driver for Cisco IOS is used and the driver excepts SSH access to be configured on the switch. Notice that password specified in this config is plain text (not encrypted). Lastly, a range of VLANs is specified on the provider interface that connects to baremetal nodes, so only a subset of VLAN IDs is used for tenant networks.

Recommended post-deployment configuration

NOTE: Steps described here are for localboot nodes and Ironic without multitenancy only.

First, build CoreOS-based Ironic Python Agent (IPA) deploy images. Here are commands to set up your IPA image building environment on a Ubuntu Xenial: 

$ sudo apt-get update
$ sudo apt-get install docker.io gzip uuid-runtime cpio findutils grep gnupg cgroup-lite git build-essential python-pip python-dev -y
$ sudo service docker start
$ git clone https://git.openstack.org/openstack/ironic-python-agent
$ cd ironic-python-agent/imagebuild/coreos
$ git checkout cf30024f96e798f5607c9664b0b6236db3232119
$ sudo pip install -r ~/ironic-python-agent/requirements.txt
$ sudo make

Resulting kernel image and ramdisk will be available in the UPLOAD subdirectory.

NOTE: Ready to use deploy images are available here: http://185.93.31.43/ironic-images/

Transfer/mount deploy images and add them to Glance: 

$ openstack image create --public --container-format aki --disk-format aki --file ~/coreos_production_pxe.vmlinuz ironic-deploy_kernel
$ openstack image create --public --container-format ari --disk-format ari --file ~/coreos_production_pxe_image-oem.cpio.gz ironic-deploy_ramdisk

Also add regular operating system images - no special options are required here and regular cloud images can be used. Here is an example for a CentOS image downloaded from https://cloud.centos.org/centos/7/images/: 

$ openstack image create --public --container-format bare --disk-format qcow2 --file ~/CentOS-7-x86_64-GenericCloud-1907.qcow2 centos7-1907

NOTE: For security reasons vanilla CentOS 7 cloud images don't have root password set. However, being able to log in as root to baremetal instances is very useful for debugging problems caused, for example, by network interfaces being down inside of the instance. Images available on http://185.93.31.43/ironic-images/ with "rootpass" in their names can be used for baremetal provisioning in the initial phase of testing a new Ironic deployment (root password for these images is vScaler2019?!). Then, when all the initial problems are resolved, it's recommended to switch to using vanilla cloud images instead.

Create a flavour for your baremetal nodes: 

$ openstack flavor create --ram 1024 --vcpus 2 --disk 100 baremetal.small
$ openstack flavor set --property resources:CUSTOM_BAREMETAL=1 baremetal.small
$ openstack flavor set --property resources:VCPU=0 baremetal.small
$ openstack flavor set --property resources:MEMORY_MB=0 baremetal.small
$ openstack flavor set --property resources:DISK_GB=0 baremetal.small

Create a provisioning network and a subnet. These can be just regular flat provider networks, or can be using VLANs. Here is an example for an interface on a VLAN: 

$ openstack network create public304 --provider-physical-network physnet1 --provider-network-type vlan --provider-segment 304
$ openstack subnet create --dhcp --allocation-pool start=10.6.44.101,end=10.6.44.254 --network public304 --subnet-range 10.6.44.0/24 --gateway 10.6.44.1 public304-subnet

Finaly, add your nodes to Ironic's database: 

$ openstack baremetal node create --name <node-name> \
--driver ipmi --driver-info ipmi_username=<ipmi-username> --driver-info ipmi_password=<ipmi-password> --driver-info ipmi_address=<ipmi-address> \
--driver-info cleaning_network=<uuid-of-the-provisioning-network> --driver-info provisioning_network=<uuid-of-the-provisioning-network> \
--driver-info deploy_kernel=<uuid-of-the-ironic-deploy_kernel-image> --driver-info deploy_ramdisk=<uuid-of-the-ironic-deploy_ramdisk-image> \
--resource-class baremetal --network-interface flat
$ openstack baremetal port create <mac-address-of-the-node's-provisioning-interface> --node <node-uuid>
$ openstack baremetal node manage <node-name>
$ openstack baremetal node provide <node-name>

If everything went well, a deploy by launching instances, like so: 

$ openstack server create --image <image-to-provision-node-with> --flavor baremetal.small --key-name <your-keypair> --network <name-of-the-provisioning-network> <instance-name>

The images available on our webserver come with a service exposing journald messages out of the instance though an HTTP endpoint. You can see the messages by running this command from your deployment machine: 

curl 'http://<provisioning-IP-assigned-to-the-instance>:19531/entries?follow'

Where the instance IP is the one assigned to the instance by the Neutron DHCP server.


Multitenancy

When adding a node to Ironic, use --network-interface neutron instead of --network-interface flat.

Next up, when adding a port, you'll need to specify details about the switch port this baremetal node is connected to: 

openstack baremetal port set --local-link-connection switch_info="supermicro_1" --local-link-connection switch_id="<MAC-address-of-the-switchport>" --local-link-connection port_id="gi 0/<port-number>" <port-UUID>

Here switch_info is the name given to the genericswitch config section in ironic-conductor.conf, port_id is the name of the port, for example on Supermicro switches this can be gi 0/1, gi 0/2, etc. (this is basically the name that you pass when running the interface <intname> command on the switch). The switch_id is not really that important (as it's not part of configuration of the switch port), but it has to be a MAC address -- makes sense for it to be the MAC of the switch port the node is connected to.

With these in place, you can now create a tenant network on the same provider as the provisioning network and with VLAN as provider type. You don't need to specify VLAN ID, as this one will be selected by Neutron (from the range of VLANs in the ML2 config) and created on the baremetal switch. Also, add to this network a subnet with whatever IP range and DHCP enabled.

Then when launching a baremetal instance, in the --network parameter you specify the tenant network (instead of the provisioning network). Ironic will first provision the node in the provisioning network and when this is done, Neutron will add switchport of the node to the VLAN of the tenant network, so after a reboot the instance will be available in the tenant network.

NOTE: In Stein there is a bug where genericdriver (for Cisco IOS) skips one of the switch command. As a workaround, make sure that all baremetal switchports are set to "access" mode.

References

  1. https://docs.openstack.org/kolla-ansible/rocky/reference/ironic-guide.html#post-deployment-configuration
  2. https://docs.openstack.org/ironic/rocky/install/configure-glance-images.html
  3. https://github.com/openstack/ironic-python-agent/tree/cf30024f96e798f5607c9664b0b6236db3232119/imagebuild/coreos
  4. https://docs.openstack.org/ironic/stein/admin/multitenancy.html

Software RAID

Support for software RAID has been added in the Train release, so if you're on an older release, you'll have to add the following parameters to your globals.yml: 

ironic_tag: "train"
ironic_install_type: "source"

and then run a kolla-ansible deploy with the "ironic" tag. Make sure that support for software RAID is present in the revision of the Ironic Python Agent that you're using for building deploy images (or use images available on http://185.93.31.43/ironic-images/).

Reconfigure your Ironic node(s) for software RAID: 

# openstack baremetal node set <node-name> --raid-interface agent
# cat ~/raid_1_basic.json
{
 "logical_disks": [
                   {
                    "size_gb": "MAX",
                    "raid_level": "1",
                    "controller": "software"
                   }
                  ]
}

# openstack baremetal node set <node-name> --target-raid-config ~/raid_1_basic.json

Feel free to edit the JSON file (the above is just a basic one for setting up a RAID-1 on 2 disks), but bear in mind limitations of the software RAID driver (check links in References). Here is a more advanced config - with 2 RAID-1 arrays on 2 disks - the first array (the one of size 50GB) is where the operating system will go: 

{
 "logical_disks": [
                   {
                    "size_gb": "50",
                    "raid_level": "1",
                    "controller": "software",
                    "is_root_volume": true
                   }
                   {
                    "size_gb": "MAX",
                    "raid_level": "1",
                    "controller": "software"
                   }
                  ]
}

Now the node is ready for RAID creation. In Ironic this is done during the "cleaning" stage. 

# openstack baremetal node manage <node-name>
# cat ~/raid_cleaning_steps.json
[{
  "interface": "raid",
  "step": "delete_configuration"
},
{
  "interface": "deploy",
  "step": "erase_devices_metadata"
},
{
  "interface": "raid",
  "step": "create_configuration"
}]
# openstack baremetal node clean <node-name> --clean-steps ~/raid_cleaning_steps.json

There is no need to modify this "cleaning steps" JSON, it should work for any RAID configuration.

This should put the node in the "cleaning" state. Wait for the node get back to the "manageable" state (feel free to fire up the virtual console of this node to keep an eye on the process). Check ironic-conductor logs for errors if the node gets into "clean failed" instead.

When the node is "manageable" again, add this root_device hint specifying your array as the root device and give the node back to Nova: 

# openstack baremetal node set <node-name> --property 'root_device={"name": "/dev/md127"}'
# openstack baremetal node provide <node-name>

To provision baremetal instances capable of booting from a software RAID, you'll need your target image (the one that you're flashing the system with) to include mdadm. This is already done in Ubuntu cloud images (at least in Xenial and Bionic), but requires baking an new image when using CentOS 7 cloud images (at least for build 1907 and older). Here are all the mdadm-related commands do use when baking a custom image for CentOS 7: 

# yum install mdadm -y
# vi /etc/dracut.conf
# grep hostonly /etc/dracut.conf
hostonly="no"
# dracut -f --add mdraid --add-drivers xfs -v

When baking this image you can also add in your custom cloud-init networking config and set root password if you want.

Instead of baking your own images, you can use qcow2 images available here: http://185.93.31.43/ironic-images/

Finally, launch an instance off of this new image and using the baremetal flavour.

References

  1. https://docs.openstack.org/ironic/train/admin/raid.html
  2. https://techblog.web.cern.ch/techblog/post/ironic_software_raid/

Troubleshooting

First, make sure all Ironic containers are up and running. Here is a list from a working Stein-based environment: 

# docker ps | grep ironic
...        registry.vscaler.com:5000/kolla/centos-source-ironic-pxe:stein                  "dumb-init --single-…"   ...   ironic_pxe
...        registry.vscaler.com:5000/kolla/centos-source-ironic-api:stein                  "dumb-init --single-…"   ...   ironic_api
...        registry.vscaler.com:5000/kolla/centos-source-ironic-conductor:stein            "dumb-init --single-…"   ...   ironic_conductor
...        registry.vscaler.com:5000/kolla/centos-binary-ironic-neutron-agent:stein        "dumb-init --single-…"   ...   ironic_neutron_agent
...        registry.vscaler.com:5000/kolla/centos-binary-nova-compute-ironic:stein         "dumb-init --single-…"   ...   nova_compute_ironic

Re-run kolla-ansible deploy with tags "ironic" if ironic_pxe, ironic_api or ironic_conductor are missing, "neutron" if ironic_neutron_agent is missing or "nova" if nova_compute_ironic is missing.

Compare the list of baremetal nodes outputted by openstack baremetal node list with the list of hypervisors from openstack hypervisor list and make sure each node's UUID is on the list of hypervisors. Check ironic-conductor logs if they're not. Fix the problem, remove the problematic nodes and add them back in.

Keep an eye on the iKVM virtual console when building an instance and deal with any DHCP, TFTP or PXE errors that show up. {TODO: Add examples of such problems and solutions to them} If the machine PXE boots successfully, the machine will boot the deploy image that gives you access to the coreos user. Make sure that the machine can reach internal OpenStack API.

If your instance shows up as ACTIVE, but you can't ping it, you should delete the instance and create it again using an image that has root password enabled so you can log into the machine through its OOB console. To add a root password to regular cloud images, you can use the virt-customize tool. Here is a command adding a root password to a CentOS 7 image: 

virt-customize -a CentOS-7-x86_64-GenericCloud-1907.qcow2 --root-password password:vScaler2019?!

Cloud-init only brings up the first connected interface, so in most cases this means an onboard interface. If this is not the interface you PXE boot from, you may want to bake a custom config for cloud-init, like this one: 

network:
  version: 2
  ethernets:
    enp59s0f0:
      dhcp4: true

or this more advanced one - with a bond: 

network:
  version: 2
  ethernets:
    eno1:
      dhcp4: true
  bonds:
    bond0:
      interfaces:
        - enp175s0f0
        - enp175s0f1
      dhcp4: true
      parameters:
        mode: 802.3ad
        mii-monitor-interval: 100

Put your config in /etc/cloud/cloud.cfg.d/custom-networking.cfg

Baremetal instances have their ports marked as DOWN in Neutron, like this: 

# openstack server list
+--------------------------------------+------------+--------+-----------------------------+--------------+-----------------+
| ID                                   | Name       | Status | Networks                    | Image        | Flavor          |
+--------------------------------------+------------+--------+-----------------------------+--------------+-----------------+
| 8314b478-310b-4d2a-a316-180c380ab5cf | compute001 | ACTIVE | ironic-prov=192.168.202.104 | centos7-1907 | baremetal.small |
+--------------------------------------+------------+--------+-----------------------------+--------------+-----------------+
# openstack port list | grep 192.168.202.104
| 2d43f9c2-e2a1-420c-8bb3-63a2d01e73c0 |      | b8:59:9f:e2:37:0e | ip_address='192.168.202.104', subnet_id='c54051ef-6344-4026-80da-529f57df6213' | DOWN   |
# openstack baremetal port list | grep b8:59:9f:e2:37:0e
| 957ed1c1-416e-468e-a87b-f14a7eaf5255 | b8:59:9f:e2:37:0e |
# openstack baremetal port show 957ed1c1-416e-468e-a87b-f14a7eaf5255 | grep node
| node_uuid             | 1907746d-4663-48d5-9b93-bd63165db382                             |
# openstack baremetal node list | grep 1907746d-4663-48d5-9b93-bd63165db382
| 1907746d-4663-48d5-9b93-bd63165db382 | node011 | 8314b478-310b-4d2a-a316-180c380ab5cf | power on    | active             | False       |

This is normal and nothing to worry about. {TODO: Why is this port marked as DOWN?}

If you know your node will end in ERROR state and you want to peek inside the deploy image during provisioning, here is how to do this:

  1. Launch a baremetal instance on the node.
  2. As soon as the instance get scheduled to the node (check that instance UUID shows up on the output from openstack baremetal node list), put your node in maintenance mode by running openstack baremetal node maintenance set <node-name>.
  3. Get into the console of the node (for example using SOL) and stop the IPA service: systemctl stop ironic-python-agent.
  4. Check journald logs, make changes, etc. In CoreOS-based deploy images all IPA-related files are in /opt/ironic-python-agent.
  5. When you're done, remove your node from maintenance (openstack baremetal node maintenance unset <node-name>) and start the IPA service (systemctl start ironic-python-agent).

Common error messages

Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: 2019-12-13 13:30:06.652 1699 ERROR root [-] Command execution error: Command execution failed: Installing GRUB2 boot loader to device /dev/md127 failed with Unexpected error while running command.
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Command: chroot /tmp/tmpbMmsW1 /bin/sh -c "grub2-install /dev/sda"
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Exit code: 1
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Stdout: u''
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Stderr: u"Installing for i386-pc platform.\ngrub2-install: error: disk `md127,1' not found.\n".: CommandExecutionError: Command execution failed: Installing GRUB2 boot loader to device /dev/md127 failed with Unexpected error while running command.
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Command: chroot /tmp/tmpbMmsW1 /bin/sh -c "grub2-install /dev/sda"
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Exit code: 1
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Stdout: u''
Dec 13 13:30:06 host-192-168-202-51 chroot[1699]: Stderr: u"Installing for i386-pc platform.\ngrub2-install: error: disk `md127,1' not found.\n".

Source: ironic-conductor logs

Your target image doesn't support software RAID. Use/bake an image with the mdadm package installed and mdadm support in the initrd enabled. 

2019-12-13 13:04:53.424 6 ERROR ironic.drivers.modules.agent_base_vendor [req-e55e913d-1216-4072-b1cc-94cdca5b72b1 - - - - -] Asynchronous exception: Node failed to deploy. Exception: Deploy failed for instance 66956849-b57c-49b7-ac78-9a0b4f2117b7. Error: Unexpected error while running command.
Command: sudo ironic-rootwrap /etc/ironic/rootwrap.conf dd if=/var/lib/ironic/images/baf22641-4698-450f-a26a-ecca012b4a24/disk of=/dev/disk/by-path/ip-192.168.202.51:3260-iscsi-iqn.2008-10.org.openstack:baf22641-4698-450f-a26a-ecca012b4a24-lun-1 bs=1M oflag=direct
Exit code: 1
Stdout: u''
Stderr: u"/bin/dd: error writing '/dev/disk/by-path/ip-192.168.202.51:3260-iscsi-iqn.2008-10.org.openstack:baf22641-4698-450f-a26a-ecca012b4a24-lun-1': No space left on device\n8192+0 records in\n8191+0 records out\n8588886016 bytes (8.6 GB) copied, 17.2107 s, 499 MB/s\n" for node baf22641-4698-450f-a26a-ecca012b4a24: InstanceDeployFailure: Deploy failed for instance 66956849-b57c-49b7-ac78-9a0b4f2117b7. Error: Unexpected error while running command.

Source: ironic-conductor logs

Ironic Python Agents tries to dd your image to a partition instead of a block device. Either clean (openstack baremetal node clean...) the node to get rid of the partition or set the device_root hint [2] on the node to your block device (for example /dev/md127 for software RAID). 

2019-12-16 09:57:00.598 6 ERROR ironic.drivers.modules.agent_base_vendor [-] Agent returned error for clean step {u'interface': u'raid', u'priority': 0, u'step': u'create_configuration', u'abortable': False} on node c3ebcf94-a4db-49bf-927e-1bd186b01a71 : {u'message': u"Clean step failed: Error performing clean_step create_configuration: Software RAID caused unknown error: Failed to create partitions on /dev/sda: Unexpected error while running command.\nCommand: parted /dev/sda -s -a optimal -- mkpart primary 2048s -1\nExit code: 1\nStdout: u''\nStderr: u'Error: Partition(s) 1 on /dev/sda have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes.\\n'", u'code': 500, u'type': u'CleaningError', u'details': u"Error performing clean_step create_configuration: Software RAID caused unknown error: Failed to create partitions on /dev/sda: Unexpected error while running command.\nCommand: parted /dev/sda -s -a optimal -- mkpart primary 2048s -1\nExit code: 1\nStdout: u''\nStderr: u'Error: Partition(s) 1 on /dev/sda have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes.\\n'"}.

Source: ironic-conductor logs

Cleanup has failed. Run the openstack baremetal node clean... command again. 

2019-10-22 12:32:41.312 6 ERROR ironic.conductor.utils [req-37fbce85-8a88-4a61-9a4a-22d6f37dc398 - - - - -] Asynchronous exception: Node failed to deploy. Exception: Connection to agent failed: Failed to connect to the agent running on node d4b3434e-047c-4715-8e89-dfd849477cdc for invoking command image.install_bootloader. Error: HTTPConnectionPool(host='172.16.100.97', port=9999): Read timed out. (read timeout=60) for node: AgentConnectionFailed: Connection to agent failed: Failed to connect to the agent running on node d4b3434e-047c-4715-8e89-dfd849477cdc for invoking command image.install_bootloader. Error: HTTPConnectionPool(host='172.16.100.97', port=9999): Read timed out. (read timeout=60)

Source: ironic-conductor logs

Root cause of this error is unknown, but the problem disappears after a short while. 

# openstack server show test_ironic | grep fault
| fault                               | {u'message': u"Host 'controller003-ironic' is not mapped to any cell", u'code': 400, u'created': u'2019-12-11T12:10:15Z'} |

Source: the openstack CLI

This can happen if nova-compute-ironic containers are not up when you add baremetal nodes to Ironic. Make sure these containers are fully functional and then run nova-manage cell_v2 map_cell_and_hosts from one of the nova-api container to fix the problem. 

CLIENT MADDR: XX XX XX XX XX XX  GUID: 00000000 0000 0000 0000 002590F950BE
CLIENT IP: 10.6.44.249  MASK: 255.255.255.0  DHCP IP: 10.6.44.101
GATEWAY IP: 10.6.44.1
PXE-E32: TFTP open timeout
PXE-M0F: Exiting Intel Boot Agent.

The above message seen on the remote console indicates problems with L3 connection to the TFTP server. Make sure your switch knows how to get to the IP specified in my_ip or tftp_server parameter in Ironic conductor's config (in this example TFTP server has the 192.168.88.233 IP). Also, check if you can reach the IP assigned by Neutron DHCP to this node (in this example, 10.6.44.249) from the machine on which the TFTP server runs. If the default route on this machine doesn't go though the Ironic provisioning switch, you will need to add static routes for all your provisioning networks. For example: 

# ip route add 10.6.44.0/24 via 192.168.88.253

References

  1. https://cloudinit.readthedocs.io/en/18.4/topics/network-config-format-v2.html
  2. https://docs.openstack.org/ironic/train/install/advanced.html#specifying-the-disk-for-deployment-root-device-hints

Ironic POC basic test environment

Error creating thumbnail: File missing

Revision of kolla-ansible used (branch stable/rocky): 

commit 668da3c332fcd58fa2b023e8bb74ca8225e222bc
Author: Jeffrey Zhang <zhang.lei.fly@gmail.com>
Date:   Tue Dec 11 16:01:03 2018 +0800

    Add cache configuration for ceilometer project

    when using ceilometer+gnocchi, for every notification sample, ceilometer
    will update the resource even if is not updated.

    We should add [cache] section to make ceilometer cache the resource, and
    stop send the useless update request.

    Closes-Bug: #1807841
    Change-Id: Ic33b4cd5ba8165c20878cab068f38a3948c9d31d
    (cherry picked from commit 55bf29ec6c459dc46cefdee69acb8e427763e409)

Standard all-in-one inventory has been used.

kolla-ansible config (/etc/kolla/globals.yml): 

---
config_strategy: "COPY_ALWAYS"
kolla_base_distro: "centos"
kolla_install_type: "binary"
openstack_release: "7.0.2"
kolla_internal_vip_address: "192.168.10.254"
kolla_external_vip_address: "172.28.128.254"
docker_registry: "registry.vscaler.com:5000"
network_interface: "enp131s0f1.10"
kolla_external_vip_interface: "eno1"
neutron_external_interface: "enp131s0f0"
neutron_bridge_name: "br-ironic"
neutron_plugin_agent: "openvswitch"
enable_cinder_backup: "no"
enable_haproxy: "yes"
enable_heat: "yes"
enable_horizon: "yes"
enable_horizon_ironic: "{{ enable_ironic | bool }}"
enable_ironic: "yes"
enable_ironic_neutron_agent: "yes"
enable_swift: "no"
tempest_image_id:
tempest_flavor_ref_id:
tempest_public_network_id:
tempest_floating_network_name:
neutron_tenant_network_types: "vlan,flat"
enable_neutron_provider_networks: yes

Config overrides for Ironic (/etc/kolla/config/ironic/ironic-conductor.conf): 

[DEFAULT]
my_ip=192.168.10.10
enabled_network_interfaces=noop,flat,neutron
default_network_interface=flat

[deploy]
default_boot_option = netboot

Here, eno1 is the interface providing access to the Ironic host from inside the Labs: 

# cat /etc/sysconfig/network-scripts/ifcfg-eno1
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="eno1"
UUID="7e04ebd9-2e2a-4297-b417-b9cb5e498259"
DEVICE="eno1"
ONBOOT="yes"
IPADDR="172.28.128.1"
PREFIX="16"
GATEWAY="172.28.0.2"
DNS1="172.28.0.2"
DNS2="8.8.8.8"
IPV6_PRIVACY="no"

enp131s0f0 is an interface that is up, but has no IP set (this will be used by Neutron to put external bridge on) and enp131s0f1.10 is a tagged secondary interface used for hosting internal OpenStack APIs and Ironic's TFTP server. 

# cat /etc/sysconfig/network-scripts/ifcfg-enp131s0f1.10
DEVICE=enp131s0f1.10
NAME=enp131s0f1.10
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.10.10
PREFIX=24
NETWORK=192.168.10.0
VLAN=yes

IPMI to the baremetal host is available through another tagged interface, enp131s0f1.201: 

# cat /etc/sysconfig/network-scripts/ifcfg-enp131s0f1.201
DEVICE=enp131s0f1.201
NAME=enp131s0f1.201
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.201.254
PREFIX=24
NETWORK=192.168.201.0
VLAN=yes

Note that because OpenStack here is deployed on a single machine, HAProxy is not strictly required.

If your playbook creates the ironic_dnsmasq container, stop/remove it, so you don't run into potential problems with 2 DHCPs on the same network.

Post deployment setup

Follow https://docs.openstack.org/kolla-ansible/rocky/reference/ironic-guide.html#post-deployment-configuration

For building baremetal images follow https://docs.openstack.org/ironic/rocky/install/configure-glance-images.html Here is an example of an Ubuntu image with a heat agent allowing to run script-based software deployments: 

$ disk-image-create baremetal ubuntu dhcp-all-interfaces os-collect-config os-refresh-config os-apply-config heat-config heat-config-cfn-init heat-config-script -o ubuntu-software-config-ironic.qcow2

Add images to Glance: 

$ openstack image create --container-format aki --disk-format aki --file ~/ubuntu-software-config-ironic.vmlinuz ubuntu-software-config-ironic_kernel
$ openstack image create --container-format ari --disk-format ari --file ~/ubuntu-software-config-ironic.initrd ubuntu-software-config-ironic_initramfs
$ openstack image create --public --container-format bare --disk-format qcow2 --file ~/ubuntu-software-config-dgx.qcow2 \
--property kernel_id=a8743614-38dc-43ed-8d3b-f4e1b4240eb0 --property ramdisk_id=1d370924-762b-49a0-8619-7f13aa8dafe1 ubuntu-software-config-dgx

In the last command kernel_id and ramdisk_id point to UUIDs of kernel and ramdisk images assigned them by Glance.

Note on localboot: The above setup is susceptible to this bug: https://storyboard.openstack.org/#!/story/2002929. To avoid the problem you can set default_boot_option = local in Ironic overrides, so that your baremetal servers will be able to boot from their local disk after they are done provisioning. More importantly, with local boot you can use regular cloud images - without having to extract kernel and ramdisk out of them first (you'll still need the kernel and ramdisk for initial deploy). This approach will be used in the next section.

Ironic POC with multi-tenancy

TODO: Add diagram


kolla-ansible config (/etc/kolla/globals.yml): 

---
config_strategy: "COPY_ALWAYS"
kolla_base_distro: "centos"
kolla_install_type: "binary"
openstack_release: "7.0.2"
kolla_internal_vip_address: "192.168.10.254"
kolla_external_vip_address: "172.28.128.254"
docker_registry: "registry.vscaler.com:5000"
network_interface: "enp131s0f1.10"
kolla_external_vip_interface: "eno1"
neutron_external_interface: "enp131s0f0"
neutron_bridge_name: "br-ironic"
neutron_plugin_agent: "openvswitch"
enable_cinder_backup: "no"
enable_haproxy: "yes"
enable_heat: "yes"
enable_horizon: "yes"
enable_horizon_ironic: "{{ enable_ironic | bool }}"
enable_ironic: "yes"
enable_ironic_neutron_agent: "yes"
enable_swift: "no"
neutron_tenant_network_types: "vlan,flat"
neutron_server_image: "registry.vscaler.com:5000/kolla/centos-source-neutron-server-with-genericswitch"
neutron_server_tag: "7.1.0"
tempest_image_id:
tempest_flavor_ref_id:
tempest_public_network_id:
tempest_floating_network_name:
enable_neutron_provider_networks: yes

Ironic-specific overrides (/etc/kolla/config/ironic/ironic-conductor.conf): 

[DEFAULT]
my_ip=192.168.10.10
enabled_network_interfaces=noop,flat,neutron
default_network_interface=neutron

[deploy]
default_boot_option = local

Network interface config is exactly the same as in the previous iteration of the deployment (without multi-tenancy).

Hacking in a new networking generic switch driver

every now and then for a POC you will come across a switch which isn't supported. This is NOT the end of the world as long as there is a way to ssh in and run a list of cli commands to control vlans.

the way I did tis was to stop the neutron_server container, and corresponding systemd service (thanks anelope), on all but one controller. The reason for this is so we don't have to inject 3x.

the next step is to pick victim driver to overwrite. DO NOT pick the netmiko fake driver!!! This one exits really early. Also DO NOT waste time trying to inject an new one uunless you understand pyhon module packaging a lot better than I.

I picked the pluribus driver as I know it works and there is zero chance of me encountering another pluribus switch in this POC. Probably best not to pick a common cisco in case you forget and try to use it. For those who care you can see the original pluribus driver at the following link https://opendev.org/openstack/networking-generic-switch/src/branch/master/networking_generic_switch/devices/netmiko_devices/pluribus.py.

here is my updated driver for a FS switch in a file called pluribus.py in my local dir 

# Copyright 2022 define-technology.
#
#    Licensed under the Apache License, Version 2.0 (the "License"); you may
#    not use this file except in compliance with the License. You may obtain
#    a copy of the License at
#
#         http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#    License for the specific language governing permissions and limitations
#    under the License.

from networking_generic_switch.devices import netmiko_devices


class Pluribus(netmiko_devices.NetmikoSwitch):
    ADD_NETWORK = (
        'configure',
        'vlan {segmentation_id}',
        'exit'
        'exit'
    )

    DELETE_NETWORK = (
        'configure',
        'no vlan {segmentation_id}',
        'exit'
        'exit'
    )

    PLUG_PORT_TO_NETWORK = (
        'configure',
        'interface {port}',
        'no port hybrid vlan all',
        'port hybrid vlan {segmentation_id} untagged',
        'port hybrid pvid {segmentation_id}',
        'exit'
        'exit'
    )

    DELETE_PORT = (
        'configure',
        'interface {port}',
        'port hybrid pvid 1',
        'no port hybrid vlan all',
        'exit'
        'exit'
    )
    ADD_NETWORK_TO_TRUNK = (
        'configure',
        'interface {port}',
        'port hybrid vlan {segmentation_id} tagged',
        'exit'
        'exit'
    )

    REMOVE_NETWORK_FROM_TRUNK = (
        'configure',
        'interface {port}',
        'no port hybrid vlan {segmentation_id} tagged',
        'exit'
        'exit'
    )

this has properly implemented MOST but not all of the full functionality. I'm missing a error regex so it has no way to know if a command has failed and will just plow on regardless assuming all is good buuuuut who cares about that? I am also missing a save function so the debug logs will end up being full of save not implemented but again it still works so nerrrr. It is REALLY handy to have that add/remove from trunk support as this allows you to add the vlan to uplink ports and your openstack config should be modified to include controllers and hypervisors (the latter only id DVR or similar is enabled).

the next thing we need to do is to find the **right** pluribus.py file (or whatever file you have decided to sacrifice) to overwrite in the neuron server container.

now lets have a look around in there hints are

  1. nothing with build in the path
  2. it has always been in the /var/lib/kolla/venv/ in my experience and there should be only the one

when you have found it exit out of the container and copy your file in an restart neutron_server 

docker exec -it neutron_server bash
find -iname pluribus.py | grep "kolla/venv"
/var/lib/kolla/venv/lib/python3.10/site-packages/networking_generic_switch/devices/netmiko_devices/pluribus.py

docker cp pluribus.py neutron_server:/var/lib/kolla/venv/lib/python3.10/site-packages/networking_generic_switch/devices/netmiko_devices/
docker restart neutron_server

if you have no typos then it should not go into a restart loop and you can try it out in your multi-tenant config

I modified my ml2_conf.ini on the controller NOT IN KOLLA CONIFG here it is in full 

root@control0002:~/scratch/neutron-ngs-patch# cat /etc/kolla/neutron-server/ml2_conf.ini
[ml2]
type_drivers = flat,vlan,vxlan,geneve
tenant_network_types = geneve
mechanism_drivers = genericswitch,baremetal,ovn
extension_drivers = port_security

[ml2_type_vlan]
network_vlan_ranges = physnet1:102:110,physnet1:200:1200

[ml2_type_flat]
flat_networks = *

[ml2_type_vxlan]
vni_ranges = 1:1000

[ml2_type_geneve]
vni_ranges = 1001:2000
max_header_size = 38

[ovn]
ovn_nb_connection = tcp:192.168.60.11:6641,tcp:192.168.60.12:6641,tcp:192.168.60.13:6641
ovn_sb_connection = tcp:192.168.60.11:6642,tcp:192.168.60.12:6642,tcp:192.168.60.13:6642
ovn_metadata_enabled = True
enable_distributed_floating_ip = True

[genericswitch:switch01]
#device_type = netmiko_fs_n_series
device_type = netmiko_pluribus
ip = 192.168.30.1
ngs_mac_address = 64:9d:99:3a:3a:58
username = admin
password = password
ngs_network_name_format = neutron-{network_id}-{segmentation_id}
ngs_save_configuration = False
ngs_trunk_ports = 25gigaethernet 1/0/14, 25gigaethernet 1/0/15, 25gigaethernet 1/0/17, 25gigaethernet 1/0/18, 25gigaethernet 1/0/20, 25gigaethernet 1/0/21, 25gigaethernet 1/0/27, 10gigaethernet 1/0/58

I then created a new vlan provider net and the vlan was added to the switch and all the expected trunk ports were updated!

how to do properly install a new switch driver

first step make a fork of networking generic switch and push it somewhere reachable for example here:

https://gitlab.define-technology.com/pub/networking-generic-switch/-/compare/stable%2F2023.1...feat-add-fs-switch?from_project_id=159&straight=false

this is the final working driver (for a given definition of working)

this driver overrides the device type in netmiko to use the existing pluribus driver. This is because of two things

  1. it exists in the standard release
  2. error checking is not implemented

The latter point is most important with this driver as I don't want to spend ages writing a full netmiko driver for this switch and it's not cisco like enough to allow those to work. It either errors going into or exiting from configure mode. If we use the pluribus driver. It never needs to go into configure mode at all and just runs the commands as is without checking the output. We know they work that is good enough for this.

it does NOT need to be a public repo it can be a private github repo but you need to make sure you can access it using a ssh key and that your key is on the build node or in your ssh agent

finally we need to buid it in to the kolla containers I made a custom kolla-build-neutron.conf file 

[DEFAULT]
namespace = kolla
base = ubuntu
registry = maas.pxeboot.internal:5000
openstack_release = 2023.1

[neutron-base-plugin-networking-generic-switch]
type = git
location = https://gitlab.define-technology.com/pub/networking-generic-switch.git
reference = feat-add-fs-switch

that is it

to build JUST the neutron-server and the prerequisite beutron-base container run 

kolla-build --config-file ./kolla-build-neutron.conf neutron-base neutron-server

then push the resulting images by hand

when you update releases you should update your fork or if you haven't done a terrible job like above push to upstream

Make a node that was configured in RAID back into a non-RAIDed instance

openstack baremetal node manage compute2
openstack baremetal node set compute2 --raid-interface no-raid
openstack baremetal node set compute2 --property root_device='{"size": "<= 2000", "rotational": false}'
openstack baremetal node maintenance unset compute2
openstack baremetal node provide compute2
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=OpenStack:Ironic&oldid=32844"