Adding Storpool Storage to OpenStack - Zed release
Jump to navigation
Jump to search
Get the Storpool containers
# storpool images downloaded define@kolla-deploy:~$ docker images | grep storpool cts.storpool.com/kolla/ubuntu-binary-nova-compute zed-ubuntu-jammy.20230309.2 f4fae03411f8 2 weeks ago 1.48GB cts.storpool.com/kolla/ubuntu-binary-cinder-volume zed-ubuntu-jammy.20230309.2 959e8af980ca 2 weeks ago 1.36GB
Check the naming used on Zed
# whats the current nova zed naming define@kolla-deploy:~$ docker images | grep nova | grep zed cts.storpool.com/kolla/ubuntu-binary-nova-compute zed-ubuntu-jammy.20230309.2 f4fae03411f8 2 weeks ago 1.48GB registry.api.internal:5000/kolla/nova-compute zed-ubuntu-jammy fc13cf204b3c 3 weeks ago 1.47GB registry.maas:5000/kolla/nova-compute zed-ubuntu-jammy fc13cf204b3c 3 weeks ago 1.47GB registry.api.internal:5000/kolla/nova-novncproxy zed-ubuntu-jammy 9d46649e580b 3 weeks ago 1.22GB registry.maas:5000/kolla/nova-novncproxy zed-ubuntu-jammy 9d46649e580b 3 weeks ago 1.22GB registry.api.internal:5000/kolla/nova-spicehtml5proxy zed-ubuntu-jammy 179707acb6ab 3 weeks ago 1.15GB registry.maas:5000/kolla/nova-spicehtml5proxy zed-ubuntu-jammy 179707acb6ab 3 weeks ago 1.15GB registry.maas:5000/kolla/nova-libvirt zed-ubuntu-jammy accca5769ec4 3 weeks ago 987MB registry.maas:5000/kolla/nova-serialproxy zed-ubuntu-jammy aebf88171aeb 3 weeks ago 1.11GB registry.maas:5000/kolla/nova-conductor zed-ubuntu-jammy 1274b18ec7f9 3 weeks ago 1.11GB registry.maas:5000/kolla/nova-compute-ironic zed-ubuntu-jammy b23d537996d3 3 weeks ago 1.12GB registry.maas:5000/kolla/nova-ssh zed-ubuntu-jammy 3b74d7d5b22f 3 weeks ago 1.11GB registry.maas:5000/kolla/nova-scheduler zed-ubuntu-jammy 80b82e6218a7 3 weeks ago 1.11GB registry.maas:5000/kolla/nova-api zed-ubuntu-jammy 1509aaad9a2f 3 weeks ago 1.11GB registry.maas:5000/kolla/nova-base zed-ubuntu-jammy 24460ee3b28a 3 weeks ago 1.11GB registry.maas:5000/nova-compute zed-ubuntu-jammy ed81b01df5db 7 weeks ago 1.47GB define@kolla-deploy:~$
Tag the images
# lets tag with a -storpool at end define@kolla-deploy:~$ docker tag cts.storpool.com/kolla/ubuntu-binary-nova-compute:zed-ubuntu-jammy.20230309.2 registry.maas:5000/kolla/nova-compute:zed-ubuntu-jammy-storpool
# lets check cinder naming define@kolla-deploy:~$ docker images | grep cinder | grep zed cts.storpool.com/kolla/ubuntu-binary-cinder-volume zed-ubuntu-jammy.20230309.2 959e8af980ca 2 weeks ago 1.36GB registry.maas:5000/kolla/cinder-volume zed-ubuntu-jammy 62fcdfbcfbc8 3 weeks ago 1.36GB registry.maas:5000/kolla/cinder-backup zed-ubuntu-jammy c5a4ea38f3d8 3 weeks ago 1.35GB registry.maas:5000/kolla/cinder-scheduler zed-ubuntu-jammy 667c90a72c25 3 weeks ago 1.32GB registry.maas:5000/kolla/cinder-api zed-ubuntu-jammy cf603f7615ec 3 weeks ago 1.32GB registry.maas:5000/kolla/cinder-base zed-ubuntu-jammy 89d7a6973593 3 weeks ago 1.32GB
# tag cinder; define@kolla-deploy:~$ docker tag cts.storpool.com/kolla/ubuntu-binary-cinder-volume:zed-ubuntu-jammy.20230309.2 registry.maas:5000/kolla/cinder-volume:zed-ubuntu-jammy-storpool define@kolla-deploy:~$ docker images | grep storpool cts.storpool.com/kolla/ubuntu-binary-nova-compute zed-ubuntu-jammy.20230309.2 f4fae03411f8 2 weeks ago 1.48GB registry.maas:5000/kolla/nova-compute zed-ubuntu-jammy-storpool f4fae03411f8 2 weeks ago 1.48GB cts.storpool.com/kolla/ubuntu-binary-cinder-volume zed-ubuntu-jammy.20230309.2 959e8af980ca 2 weeks ago 1.36GB registry.maas:5000/kolla/cinder-volume zed-ubuntu-jammy-storpool 959e8af980ca 2 weeks ago 1.36GB
Push the tagged containers
# push containers 1996 docker push registry.maas:5000/kolla/cinder-volume:zed-ubuntu-jammy-storpool 1998 docker push registry.maas:5000/kolla/nova-compute:zed-ubuntu-jammy-storpool
Storpool configuration files setup
# no storpool files exists so lets touch define@kolla-deploy:~$ ssh ubuntu@inf-233 sudo touch /etc/storpool.conf define@kolla-deploy:~$ ssh ubuntu@inf-234 sudo touch /etc/storpool.conf define@kolla-deploy:~$ ssh ubuntu@inf-235 sudo touch /etc/storpool.conf
# config file globals.yml enable_cinder: "yes" enable_cinder_backup: "yes" # storpool dp enable_cinder_backend_iscsi: "yes" enable_cinder_backend_lvm: "no" skip_cinder_backend_check: True cinder_volume_image: "registry.maas:5000/kolla/cinder-volume" cinder_volume_tag: "zed-ubuntu-jammy-storpool" cinder_volume_extra_volumes: - "/etc/storpool.conf:/etc/storpool.conf:ro" # end storpool
# add control to the iscsi investory (kolla-ansible-zed) define@kolla-deploy:~/kolla/config$ grep iscsi inventory/groups -A 5 [iscsid:children] compute storage ironic control
Deploy
# ok added the iqns – lets deploy kolla-ansible --configdir=$(pwd) -i inventory -t cinder,iscsid deploy
Get the iscsid setup
# get the iqns (kolla-ansible-zed) define@kolla-deploy:~/kolla/config$ pdsh -R ssh -lubuntu -w inf-23[3,4,5] sudo cat /var/lib/docker/volumes/iscsi_info/_data/initiatorname.iscsi inf-233: InitiatorName=iqn.2016-04.com.open-iscsi:99774b8f2d84 inf-234: InitiatorName=iqn.2016-04.com.open-iscsi:1e4c782a2284 inf-235: InitiatorName=iqn.2016-04.com.open-iscsi:842ef79751f
Setup the configuration cinder.conf
# cinder.conf [DEFAULT] enabled_backends=ceph-replicated, storpool #default_volume_type = ceph-fast [storpool] volume_driver = cinder.volume.drivers.storpool.StorPoolDriver volume_backend_name = storpool iscsi_cinder_volume = True iscsi_portal_group = "pg" #iscsi_export_to = " iqn.2016-04.com.open-iscsi:99774b8f2d841 iqn.2016-04.com.open-iscsi:1e4c782a2284 iqn.2016-04.com.open-iscsi:842ef79751f " image_upload_use_cinder_backend = True image_upload_use_internal_tenant = True storpool_template=nvme
reconfigure
# ok added the iqns – lets deploy kolla-ansible --configdir=$(pwd) -i inventory -t cinder,iscsid reconfigure
Check the container status
# check the rollout (kolla-ansible-zed) define@kolla-deploy:~/kolla/config$ pdsh -R ssh -lubuntu -w inf-23[3,4,5] "sudo docker ps | grep -e cinder -e iscsi" inf-234: 395216d08791 registry.maas:5000/kolla/iscsid:zed-ubuntu-jammy "dumb-init --single-…" 4 hours ago Up 3 hours iscsid inf-234: 82d85ac4bcd0 registry.maas:5000/kolla/cinder-volume:zed-ubuntu-jammy-storpool "dumb-init --single-…" 3 weeks ago Up About a minute (healthy) cinder_volume inf-234: 5eeaeda3c600 registry.maas:5000/kolla/cinder-backup:zed-ubuntu-jammy "dumb-init --single-…" 3 weeks ago Up About a minute (healthy) cinder_backup inf-234: 2415fef21f0a registry.maas:5000/kolla/cinder-scheduler:zed-ubuntu-jammy "dumb-init --single-…" 5 weeks ago Up About a minute (healthy) cinder_scheduler inf-234: c07f624be1c4 registry.maas:5000/kolla/cinder-api:zed-ubuntu-jammy "dumb-init --single-…" 5 weeks ago Up About a minute (healthy) cinder_api inf-235: 9ee5478c582c registry.maas:5000/kolla/iscsid:zed-ubuntu-jammy "dumb-init --single-…" 4 hours ago Up 3 hours iscsid inf-235: 9578d8a23f34 registry.maas:5000/kolla/cinder-volume:zed-ubuntu-jammy-storpool "dumb-init --single-…" 3 weeks ago Up About a minute (healthy) cinder_volume inf-235: 2d877f6cff8e registry.maas:5000/kolla/cinder-backup:zed-ubuntu-jammy "dumb-init --single-…" 3 weeks ago Up About a minute (healthy) cinder_backup inf-235: 8bd93484fa5a registry.maas:5000/kolla/cinder-scheduler:zed-ubuntu-jammy "dumb-init --single-…" 5 weeks ago Up About a minute (healthy) cinder_scheduler inf-235: b09e645d85b8 registry.maas:5000/kolla/cinder-api:zed-ubuntu-jammy "dumb-init --single-…" 5 weeks ago Up About a minute (healthy) cinder_api inf-233: c421245ae518 registry.maas:5000/kolla/iscsid:zed-ubuntu-jammy "dumb-init --single-…" 4 hours ago Up 3 hours iscsid inf-233: 71ee80e283cc registry.maas:5000/kolla/cinder-volume:zed-ubuntu-jammy-storpool "dumb-init --single-…" 3 weeks ago Up About a minute (healthy) cinder_volume inf-233: 0a29f3a47726 registry.maas:5000/kolla/cinder-backup:zed-ubuntu-jammy "dumb-init --single-…" 3 weeks ago Up About a minute (healthy) cinder_backup inf-233: 635873d31c5d registry.maas:5000/kolla/cinder-scheduler:zed-ubuntu-jammy "dumb-init --single-…" 5 weeks ago Up About a minute (healthy) cinder_scheduler inf-233: 465f48acac9e registry.maas:5000/kolla/cinder-api:zed-ubuntu-jammy "dumb-init --single-…" 5 weeks ago Up About a minute (healthy) cinder_api
Setup the storpool volumes
(kolla-ansible-zed) define@kolla-deploy:~/kolla/config$ openstack volume type create --property volume_backend_name=storpool --property storpool_template='nvme' storpool-nvme +-------------+----------------------------------------------------------+ | Field | Value | +-------------+----------------------------------------------------------+ | description | None | | id | 14bce409-37bf-4b34-9f45-80628b9c772d | | is_public | True | | name | storpool-nvme | | properties | storpool_template='nvme', volume_backend_name='storpool' | +-------------+----------------------------------------------------------+ (kolla-ansible-zed) define@kolla-deploy:~/kolla/config$ openstack volume type list +--------------------------------------+---------------+-----------+ | ID | Name | Is Public | +--------------------------------------+---------------+-----------+ | 14bce409-37bf-4b34-9f45-80628b9c772d | storpool-nvme | True | | 62970386-e08a-49c9-8c09-d83902fdd8a2 | __DEFAULT__ | True | +--------------------------------------+---------------+-----------
setup the nova compute side
Globals.yaml updated
# storpool dp enable_cinder_backend_iscsi: "yes" enable_cinder_backend_lvm: "no" skip_cinder_backend_check: True cinder_volume_image: "registry.maas:5000/kolla/cinder-volume" cinder_volume_tag: "zed-ubuntu-jammy-storpool" cinder_volume_extra_volumes: - "/etc/storpool.conf:/etc/storpool.conf:ro" nova_compute_image: "registry.maas:5000/kolla/nova-compute" nova_compute_tag: "zed-ubuntu-jammy-storpool" nova_compute_extra_volumes: - "/etc/storpool.conf:/etc/storpool.conf:ro" # end storpool
Nova compute docker options
root@gpu-227:~# cat /etc/docker/daemon.json
{
"bridge": "none",
"exec-opts": [
"native.cgroupdriver=systemd"
],
"ip-forward": false,
"iptables": false,
"log-opts": {
"max-file": "5",
"max-size": "50m"
}
Notes from storpool on setting up systems
# As per the changes, I did the following:
# 1. Install the systemd-container package on the host.
apt-get install -y systemd-container.
# 2. Add an extra systemd drop-in to make systemd-machined start at boot.
mkdir -p /etc/systemd/system/systemd-machined.service.d,
# 3. Then add a file called 10-install.conf with the following contents:
[Install]
WantedBy=multi-user.target
# 4. Issue systemctl daemon-reload followed by
systemctl enable --now systemd-machined.
# 5. In case the nova_libvirt container is already deployed, restart it with
docker restart nova_libvirt.
# update this was in the notes from storpool - should sort the above I think
Extra Ansbile compute host variables in INI format:
docker_custom_config="{'exec-opts': ['native.cgroupdriver=systemd']}"
default_container_dimensions="{'cpuset_cpus': '0,2-15'}"
Update the compute nodes with the new container for nova-compute
kolla-ansible --configdir=$(pwd) -i inventory --limit=gpu-227 -t nova deploy
Check storpool nova container is now running
root@gpu-227:~# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f46b27a8619c registry.maas:5000/kolla/nova-compute:zed-ubuntu-jammy-storpool "dumb-init --single-…" 38 seconds ago Up 37 seconds (healthy) nova_compute
Some compute node tuning and flavours
root@gpu-227:/var/log/kolla# grep huge /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS1,115200 amd_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=768"
sudo update-grub
root@gpu-227:~# dmesg | grep -i huge [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-69-generic root=UUID=8bec6a7e-c072-4e00-b47e-fe80e1a3fb9a ro i915.modeset=0 vga=normal swapaccount=1 systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller=1 video=vesafb:off nofb nomodeset console=ttyS1,115200 amd_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=768 crashkernel=2048M [ 1.312102] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-69-generic root=UUID=8bec6a7e-c072-4e00-b47e-fe80e1a3fb9a ro i915.modeset=0 vga=normal swapaccount=1 systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller=1 video=vesafb:off nofb nomodeset console=ttyS1,115200 amd_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=768 crashkernel=2048M [ 8.505475] HugeTLB registered 1.00 GiB page size, pre-allocated 768 pages [ 8.513007] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
update the new a6000 settings
Back to headnode
define@kolla-deploy:~/kolla/config/inventory$ pwd
/home/define/kolla/config/inventory
define@kolla-deploy:~/kolla/config$ cat inventory/group_vars/compute_gpu_a6000.yaml
---
dt_passthough_devs: [10de:2230,10de:1aef]
# AMD EPYC 7543 with ht (128 cores), avoiding 64,72,80,88,96,104,112,120
dt_shared_cpus_set: '64' #if this fails then steal at least one core from the dt_docker_cpuset and bootsrtrap and reconfigure again.
dt_dedicated_cpu_set: '0-7,65-71,8-15,73-79,16-23,81-87,24-31,89-95,32-39,97-103,40-47,105-111,48-55,113-119,56-63,121-127'
# the cores NOT in the shared and dedicated
dt_storpool_set: '104' #this is the core that storpool is using, invert output from cat /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus
dt_docker_cpuset: '72,80,88,96,112,120' # this should have the cores we excluded from dedicated, shared AND storpool sets
# old AC settings
# cpu core tuning settings
# AMD EPYC 7543 with ht (128 cores), avoiding 0,8,16,24,32,40,48,56
#dt_shared_cpus_set: '1-7,64-71,9-15,72-79,17-23,80-87,25-31,88-95,33-39,96-103,41-47,104-111,49-55,112-119,57-63,120-127'
#dt_shared_cpus_set: '0-7,65-71,8-15,73-79,16-23,81-87,24-31,89-95,32-39,97-103,40-47,105-111,48-55,113-119,56-63,121-127'
#dt_dedicated_cpu_set: ''
# the cores NOT in the shared and dedicated
#dt_storpool_set: '104' #this is the core that storpool is using, invert output from cat /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus
#dt_docker_cpuset: '64,72,80,88,96,112,120' # this should have the cores we excluded from dedicated, shared AND storpool sets
docker_custom_config:
exec-opts:
- 'native.cgroupdriver=systemd'
default_container_dimensions:
cpuset_cpus: "{{ dt_docker_cpuset }}"
nova_cell_compute_provider_config:
meta:
schema_version: '1.0'
providers:
- identification:
name: $COMPUTE_NODE
traits:
additional:
- 'CUSTOM_GENERIC_GPU'
- 'CUSTOM_PASSTHROUGH_A6000'
inventories:
additional:
- CUSTOM_GPU_A100_80GB_PCI:
total: 8
reserved: 0
min_unit: 1
max_unit: 8
step_size: 1
allocation_ratio: 1.0
reconfigure kolla nova
kolla-ansible --configdir=$(pwd) -i inventory --limit=gpu-227 -t nova reconfigure
Add numa tuned flavour
openstack flavor create g1.med.numa.1xa6000 \ --vcpu 12 \ --disk 40 \ --ram 98304 \ --property pci_passthrough:alias='a6000:1,NVaudio:1' \ --property hw:mem_page_size=1GB \ --property hw:cpu_policy=dedicated \ --property hw:numa_nodes=2 \ --property hw:cpu_sockets=2 \ --property hw:cpu_threads=2 \ --property hw:cpu_thread_policy=prefer \ --property hw:pci_numa_affinity_policy=preferred