View source for VScaler: Kolla Initial setup build and deployment (Newton)

== Pre-install Notes ==
You'll need the following systems: (can be all-in-one, but i've split) 
# kolla-build server (this is to build all the docker images) I've used the headnode of OpenHPC
# controller node(s) 
# compute node(s)
# network node(s)
# storage node(s)

=== XFS Issues ===
Problems with XFS - Either use EXT4 or format the XFS partition with the flags below:
* NOTE: Bug with XFS also - if building on an XFS file system: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/7.2_Release_Notes/technology-preview-file_systems.html
* create the XFS file system with: 
<syntaxhighlight>
mkfs -t xfs -n ftype=1 /PATH/TO/DEVICE
</syntaxhighlight>

If installing on kolla-build on a compute node, its missing loads of include files. (not a problem if using the headnode for buidling)  
<syntaxhighlight>
# NOTE: VNFS removes a lot of include files, ensure the following is commented out in:  /etc/warewulf/vnfs.conf
#hybridize += /usr/X11R6
#hybridize += /usr/lib/locale
#hybridize += /usr/lib64/locale
#hybridize += /usr/include
#hybridize += /usr/share/man
#hybridize += /usr/share/doc
#hybridize += /usr/share/locale
</syntaxhighlight>

=== Missing header files on OpenHPC compute nodes ===

If you want to apply to a pre-deployed node and not faff around 
<syntaxhighlight>
# for some weird reason, glibc-headers is missing loads of files. Have to force an uninstall and re-install to fix on a compute node
rpm -e glibc-headers --nodeps
yum -y install glibc-headers
rpm -e glibc-devel --nodeps
yum -y install glibc-devel
rpm -e gmp-devel --nodeps
yum -y install gmp-devel
yum -y install glibc-devel
rpm -e kernel-headers --nodeps
yum -y install kernel-headers
yum -y install openssl-devel

# (WARNING BREAKS YUM) and check version is the same
rpm -e python-libs --nodeps
wget http://mirror.centos.org/centos/7/os/x86_64/Packages/python-libs-2.7.5-48.el7.x86_64.rpm
rpm -ivh ./python-libs-2.7.5-48.el7.x86_64.rpm
# end wierdness! 
</syntaxhighlight>

== Setup the kolla-build system ==
<syntaxhighlight>
# remove openhpc and cobbler repo if they exist!
yum -y groupinstall 'Development Tools'
yum -y update
yum -y install epel-release
yum -y install python-pip python-devel
pip install -U pip
pip -V
# At this stage im at: 
#[root@node02 yum.repos.d]# pip -V 
#pip 9.0.1 from /usr/lib/python2.7/site-packages (python 2.7)
#[root@node02 yum.repos.d]# cat /etc/redhat-release 
#CentOS Linux release 7.3.1611 (Core) 
</syntaxhighlight>

== Setup Docker to act as a registry ==
<syntaxhighlight>
# Install docker 
curl -sSL https://get.docker.com/ | sh
usermod -aG docker root

# Setup the sysconfig file 
tee /etc/sysconfig/docker  <<-'EOF'
DOCKER_OPTS="--insecure-registry 10.10.10.1:4000"
EOF
 mkdir -p /etc/systemd/system/docker.service.d

# setup the service file
tee /etc/systemd/system/docker.service.d/kolla.conf <<-'EOF'
[Service]
MountFlags=shared
EnvironmentFile=-/etc/sysconfig/docker
#EnvironmentFile=-/etc/sysconfig/docker-storage
#EnvironmentFile=-/etc/sysconfig/docker-network
ExecStart=
ExecStart=/usr/bin/dockerd --insecure-registry 10.10.10.1:4000
#ExecStart=/usr/bin/dockerd $DOCKER_OPTS
EOF

# restart stuff 
systemctl daemon-reload
systemctl restart docker
systemctl status docker.service
systemctl enable docker.service
systemctl stop libvirtd.service
systemctl disable libvirtd.service

# last few packages 
pip install -U docker-py
pip install ansible==2.0.0

# start up the docker registry on the deployent node 
docker run -d -p 4000:5000 --restart=always -v /var/lib/docker/volumes/registry:/var/lib/registry --name registry registry:2 
</syntaxhighlight>

== Setup and Install Kolla (release 3.0.4 / Newton) ==
<syntaxhighlight>
git clone https://git.openstack.org/openstack/kolla
cd kolla/
git checkout stable/newton
cd ..
# Ref problem: http://stackoverflow.com/questions/42029545/pip-is-error-typeerror-call-takes-exactly-2-arguments-1-given
# vi /usr/lib/python2.7/site-packages/packaging/requirements.py
# Changed line 59:
# #MARKER_EXPR = originalTextFor(MARKER_EXPR())("marker")
# MARKER_EXPR = originalTextFor(MARKER_EXPR)("marker")
pip install -r kolla/requirements.txt -r kolla/test-requirements.txt
pip install kolla/
cd kolla
cp -rvf etc/kolla /etc/
yum install python-devel libffi-devel openssl-devel gcc -y
pip install -U python-openstackclient python-neutronclient
pip install tox
tox -e genconfig
</syntaxhighlight>

==  Additional Notes for Ocata ==
<syntaxhighlight>
git checkout stable/ocata
# then rest as above
# once kolla installed above, its only kolla-build. Need to add kolla-ansible as below
git clone https://git.openstack.org/openstack/kolla-ansible
git checkout stable/ocata
pip install kolla-ansible/

# copy conf files
cp /usr/share/kolla-ansible/etc_examples/kolla/globals.yml /etc/kolla/
cp /usr/share/kolla-ansible/etc_examples/kolla/passwords.yml /etc/kolla/
cp /root/kolla-ansible/ansible/inventory/all-in-one ~/kolla-deploy
cp /root/kolla-ansible/ansible/inventory/multinode ~/kolla-deploy

</syntaxhighlight>

=== Kolla installation files - Multinode === 

Create (copy) the multinode inventory file:
<syntaxhighlight>
#setup the installation / build file
mkdir ~/kolla-deploy
cd ~/kolla-deploy
cp /usr/share/kolla/ansible/inventory/multinode ~/kolla-deploy
</syntaxhighlight>

Edit the multinode inventory file. It should have something like this:
<syntaxhighlight>
[control]
# These hostname must be resolvable from your deployment host
controller01
controller02
controller03

# The above can also be specified as follows:
#controller[01:03]     ansible_user=kolla

# The network nodes are where your l3-agent and loadbalancers will run
# This can be the same as a host in the control group
[network]
controller01
controller02
controller03

[compute]
compute01
compute02
compute03
compute04
gpu01 api_interface=ens5

[monitoring]
controller01
controller02
controller03

# When compute nodes and control nodes use different interfaces,
# you can specify "api_interface" and other interfaces like below:
#compute01 neutron_external_interface=eth0 api_interface=em1 storage_interface=em1 tunnel_interface=em1

[storage]
storage01
storage02
storage03
storage04
.
.
.
</syntaxhighlight>

'''''NB''''' - GPU01 in our setup above has a different network interface name to the rest. The common interface name is defined in <code>globals.yml</code> and the exception is declared above in the <code>multinode</code> file.

=== Kolla installation files - /etc/kolla/passwords  === 
<syntaxhighlight>
kolla-genpwd
</syntaxhighlight>

=== Kolla installation files - /etc/kolla/globals.yml  === 
Make sure you define entries in '''<code>/etc/kolla/globals.yml</code>''' as below.

*This was for my 1GbE setup:
<syntaxhighlight>
[root@head01 ~]# egrep -v '(^#|^$)' /etc/kolla/globals.yml.1gbe 
---
kolla_internal_vip_address: "10.10.10.254"
docker_registry: "10.10.10.1:4000"
network_interface: "eth2"
neutron_external_interface: "eno2"
ceph_enable_cache: "yes"
ceph_cache_mode: "writeback"
ceph_pool_type: "erasure"
designate_backend: "bind9"
designate_ns_record: "sample.openstack.org"
tempest_image_id:
tempest_flavor_ref_id:
tempest_public_network_id:
tempest_floating_network_name:
</syntaxhighlight>

*This was Dave's original:
<syntaxhighlight>
[root@gpu01 kolla]# egrep -v '(^#|^$)' /etc/kolla/globals.yml
---
kolla_internal_vip_address: "10.10.10.254"
docker_registry: "10.10.10.1:4000"
network_interface: "eth2"
neutron_external_interface: "eno1"
</syntaxhighlight>

* And this was for my 10GbE setup:
<syntaxhighlight>
[root@head01 ~]# egrep -v '(^#|^$)' /etc/kolla/globals.yml
---
kolla_internal_vip_address: "192.168.50.254"
docker_registry: "10.10.10.1:4000"
network_interface: "enp2s0"
neutron_external_interface: "eno2"
ceph_enable_cache: "yes"
ceph_cache_mode: "writeback"
ceph_pool_type: "erasure"
designate_backend: "bind9"
designate_ns_record: "sample.openstack.org"
tempest_image_id:
tempest_flavor_ref_id:
tempest_public_network_id:
tempest_floating_network_name:
</syntaxhighlight>

=== Build some docker images ===
<syntaxhighlight>
kolla-build --base centos --type binary --registry 10.10.10.1:4000 --push
# some will fail because binary candidates not found - you can build from source using (for e.g. murano / mistral didnt build)
# kolla-build --base centos --type source --registry 10.10.10.1:4000 --push     
</syntaxhighlight>

check the docker image registry for all your images
<syntaxhighlight>
docker images -a 
docker ps -a
</syntaxhighlight>

=== Update Ansible and disable the standard libvirtd ===

ansible version 2.0.0.0 resulted in kolla-ansible bootstrap and precheck error, so
<syntaxhighlight>
pip install -U ansible
</syntaxhighlight>

Make sure libvirtd is stopped and disabled on compute nodes
<syntaxhighlight>
pdsh -w compute0[1-4],gpu01 'systemctl stop libvirtd && systemctl disable libvirtd'
</syntaxhighlight>

== Deploy and Test ==
<syntaxhighlight>
kolla-ansible bootstrap-servers -i all-in-one 
kolla-ansible prechecks -i ./multinode
kolla-ansible pull -i ./multinode # make sure you can pull all the images 
kolla-ansible -vvv deploy -i ./multinode
kolla-ansible post-deploy # this creates the admin-openrc.sh, in /etc/kolla, credentials
# sometime you neeed add or remove a computer node or controller node, you should modify /usr/share/kolla/ansible/inventory/multinode, then execute:
# kolla-ansible upgrade -i ./multinode
pip install python-openstackclient
source /etc/kolla/admin-openrc.sh
/usr/share/kolla/init-runonce
</syntaxhighlight>

== Configure CEPH ==

The disk setup on our storage nodes:
<syntaxhighlight>
[root@head01 ~]# ssh storage01 lsscsi
[0:0:0:0]    disk    ATA      TOSHIBA MG04ACA6 FS2B  /dev/sda 
[0:0:1:0]    disk    ATA      TOSHIBA MG04ACA6 FS2B  /dev/sdb 
[0:0:2:0]    disk    ATA      TOSHIBA MG04ACA6 FS2B  /dev/sdc 
[0:0:3:0]    disk    ATA      TOSHIBA MG04ACA6 FS2B  /dev/sdd 
[0:0:4:0]    disk    ATA      SDLFGC7R-960G-1H MC99  /dev/sde 
[0:0:5:0]    disk    ATA      TOSHIBA MG04ACA6 FS2B  /dev/sdf 
[0:0:7:0]    disk    ATA      TOSHIBA MG04ACA6 FS2B  /dev/sdg 
</syntaxhighlight>

And partition layout (the O/S is installed on '''<code>/dev/sda</code>'''):
<syntaxhighlight>
[root@head01 ~]# ssh storage01 df -h
Filesystem                Size  Used Avail Use% Mounted on
/dev/sda3                 2.0T  3.9G  1.9T   1% /
devtmpfs                   16G     0   16G   0% /dev
tmpfs                      16G     0   16G   0% /dev/shm
tmpfs                      16G   17M   16G   1% /run
tmpfs                      16G     0   16G   0% /sys/fs/cgroup
/dev/sda1                 463M  113M  322M  26% /boot
10.10.10.1:/home          909G   34G  830G   4% /home
10.10.10.1:/opt/ohpc/pub  909G   34G  830G   4% /opt/ohpc/pub
tmpfs                     3.2G     0  3.2G   0% /run/user/0
</syntaxhighlight>

=== Label each disk to be used as an OSD ===
In our setup, we're going to use '''<code>/dev/sd[b-d,f-g]</code>''' as OSDs.

<syntaxhighlight>
[root@head01 ~]# pdsh -w storage0[1-4] 'id=0; for i in "sdb" "sdc" "sdd" "sdf" "sdg"; do parted /dev/$i -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_FO${id} 1 -1;((id++)) ; done'
</syntaxhighlight>

Check that worked:
<syntaxhighlight>
[root@head01 ~]# pdsh -w storage0[1-4] 'id=0; for i in "sdb" "sdc" "sdd" "sdf" "sdg"; do parted /dev/$i print | grep FO;((id++)) ; done'
storage01:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO0
storage02:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO0
storage04:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO0
storage03:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO0
storage01:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO1
storage03:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO1
storage02:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO1
storage04:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO1
storage01:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO2
storage03:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO2
storage02:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO2
storage01:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO3
storage04:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO2
storage04:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO3
storage03:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO3
storage02:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO3
storage01:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO4
storage03:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO4
storage04:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO4
storage02:  1      1049kB  6001GB  6001GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO4
</syntaxhighlight>

== Example using Journal Drives as well == 
Single journal drive - split in to the number of OSD 
<syntaxhighlight>
[root@node00 centos7.4-mouza]# pdsh -w node[01-03] 'parted /dev/sdb -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_FO0_J 0% 25% mkpart KOLLA_CEPH_OSD_BOOTSTRAP_FO1_J 25% 50% mkpart KOLLA_CEPH_OSD_BOOTSTRAP_FO2_J 50% 75% mkpart KOLLA_CEPH_OSD_BOOTSTRAP_FO3_J 75% 100%;'
[root@node00 centos7.4-mouza]# pdsh -w node[01-03] 'id=0; for i in "sdc" "sdd" "sde" "sdf" ; do parted /dev/$i -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_FO${id} 1 -1;((id++)) ; done'
[root@node00 centos7.4-mouza]# pdsh -w node[01-03] 'id=0; for i in "sdb" "sdc" "sdd" "sde" "sdf"; do parted /dev/$i print | grep FO;((id++)) ; done'
node01:  1      1049kB  120GB  120GB               KOLLA_OSD_BOOTSTRAP_FO0_J
node01:  2      120GB   240GB  120GB               KOLLA_OSD_BOOTSTRAP_FO1_J
node01:  3      240GB   360GB  120GB               KOLLA_OSD_BOOTSTRAP_FO2_J
node01:  4      360GB   480GB  120GB               KOLLA_OSD_BOOTSTRAP_FO3_J
node01:  1      1049kB  1920GB  1920GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO0
node01:  1      1049kB  1920GB  1920GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO1
node01:  1      1049kB  1920GB  1920GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO2
node01:  1      1049kB  1920GB  1920GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO3
node02:  1      1049kB  120GB  120GB               KOLLA_OSD_BOOTSTRAP_FO0_J
node02:  2      120GB   240GB  120GB               KOLLA_OSD_BOOTSTRAP_FO1_J
node02:  3      240GB   360GB  120GB               KOLLA_OSD_BOOTSTRAP_FO2_J
node02:  4      360GB   480GB  120GB               KOLLA_OSD_BOOTSTRAP_FO3_J
node02:  1      1049kB  1920GB  1920GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO0
node02:  1      1049kB  1920GB  1920GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO1
node03:  1      1049kB  120GB  120GB               KOLLA_OSD_BOOTSTRAP_FO0_J
node03:  2      120GB   240GB  120GB               KOLLA_OSD_BOOTSTRAP_FO1_J
node03:  3      240GB   360GB  120GB               KOLLA_OSD_BOOTSTRAP_FO2_J
node03:  4      360GB   480GB  120GB               KOLLA_OSD_BOOTSTRAP_FO3_J
node02:  1      1049kB  1920GB  1920GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO2
node03:  1      1049kB  1920GB  1920GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO0
node02:  1      1049kB  1920GB  1920GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO3
node03:  1      1049kB  1920GB  1920GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO1
node03:  1      1049kB  1920GB  1920GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO2
node03:  1      1049kB  1920GB  1920GB               KOLLA_CEPH_OSD_BOOTSTRAP_FO3
</syntaxhighlight>

=== Label each SSD to be used as a cache drive ===
In our setup, we're going to use '''<code>/dev/sde</code>''' as a cache drive.

<syntaxhighlight>
[root@head01 ~]# pdsh -w storage0[1-4] 'parted /dev/sde -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_CACHE_BOOTSTRAP 1 -1'
</syntaxhighlight>

Check that worked:
<syntaxhighlight>
[root@head01 ~]# pdsh -w storage0[1-4] 'parted /dev/sde print | grep -i cache'
storage01:  1      1049kB  960GB  960GB               KOLLA_CEPH_OSD_CACHE_BOOTSTRAP
storage02:  1      1049kB  960GB  960GB               KOLLA_CEPH_OSD_CACHE_BOOTSTRAP
storage04:  1      1049kB  960GB  960GB               KOLLA_CEPH_OSD_CACHE_BOOTSTRAP
storage03:  1      1049kB  960GB  960GB               KOLLA_CEPH_OSD_CACHE_BOOTSTRAP
</syntaxhighlight>

=== Enable CEPH ===
Enable CEPH in '''<code>/etc/kolla/globals.yml</code>''':
<syntaxhighlight>
enable_ceph: "yes"
</syntaxhighlight>

Enable RadosGW in '''<code>/etc/kolla/globals.yml</code>''' also (this is optional):
<syntaxhighlight>
enable_ceph_rgw: "yes"
</syntaxhighlight>

Enable cache tier in '''<code>/etc/kolla/globals.yml</code>''':
<syntaxhighlight>
enable_ceph: "yes"
ceph_enable_cache: "yes"
# Valid options are [ forward, none, writeback ]
ceph_cache_mode: "writeback"
</syntaxhighlight>

To enable '''erasure coded''' pools add the following options to your '''<code>/etc/kolla/globals.yml</code>''' configuration file (this is optional (the default is 'replicated') but a cache tier is required for erasure coding):
<syntaxhighlight>
# A requirement for using the erasure-coded pools is you must setup a cache tier
# Valid options are [ erasure, replicated ]
ceph_pool_type: "erasure"
# Optionally, you can change the profile
#ceph_erasure_profile: "k=4 m=2 ruleset-failure-domain=host"
</syntaxhighlight>

== Check Ceph is working ok / Debug Ceph ==
If you are redeploying, please make sure you umount the OSD on the nodes and relabel partitions 
<syntaxhighlight>
# Labels before install:
KOLLA_CEPH_OSD_BOOTSTRAP_FO0_J
KOLLA_CEPH_OSD_BOOTSTRAP_FO0_J

#For e.g  labels after install: 
KOLLA_CEPH_DATA_2_J 
KOLLA_CEPH_DATA_2  

# Script to cleanup
# Redo Partitions with journal and osd on each SSD 
pdsh -w node[01-03] 'umount /dev/sdc2; umount /dev/sdd2; umount /dev/sde2 ; umount /dev/sdf2'
pdsh -w node[01-03] 'id=0; for i in "sdc" "sdd" "sde" "sdf" ; do parted -a optimal /dev/$i -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_FO${id}_J 1 5G mkpart KOLLA_CEPH_OSD_BOOTSTRAP_FO${id} 5G 100% ;((id++)) ; done'
pdsh -w node[01-03] 'id=0; for i in "sdc" "sdd" "sde" "sdf"; do parted /dev/$i print | grep FO;((id++)) ; done'
</syntaxhighlight>


<syntaxhighlight>
# from a controller:
[root@node03-enp175s0f0 ~]# docker exec ceph_mon ceph osd tree 
ID WEIGHT   TYPE NAME            UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-1 12.00000 root default                                           
-2  4.00000     host 10.10.20.12                                   
 0  1.00000         osd.0             up  1.00000          1.00000 
 4  1.00000         osd.4             up  1.00000          1.00000 
 8  1.00000         osd.8             up  1.00000          1.00000 
11  1.00000         osd.11            up  1.00000          1.00000 
-3  4.00000     host 10.10.20.13                                   
 2  1.00000         osd.2             up  1.00000          1.00000 
 5  1.00000         osd.5             up  1.00000          1.00000 
 6  1.00000         osd.6             up  1.00000          1.00000 
10  1.00000         osd.10            up  1.00000          1.00000 
-4  4.00000     host 10.10.20.11                                   
 1  1.00000         osd.1             up  1.00000          1.00000 
 3  1.00000         osd.3             up  1.00000          1.00000 
 7  1.00000         osd.7             up  1.00000          1.00000 
 9  1.00000         osd.9             up  1.00000          1.00000 

[root@node03-enp175s0f0 ~]# docker exec ceph_mon ceph status 
    cluster ee0c5a5d-2ba0-4af6-8d5c-21dc44bfff5c
     health HEALTH_OK
     monmap e2: 3 mons at {10.10.20.11=10.10.20.11:6789/0,10.10.20.12=10.10.20.12:6789/0,10.10.20.13=10.10.20.13:6789/0}
            election epoch 8, quorum 0,1,2 10.10.20.11,10.10.20.12,10.10.20.13
     osdmap e28: 12 osds: 12 up, 12 in
            flags sortbitwise,require_jewel_osds
      pgmap v46: 360 pgs, 8 pools, 1588 bytes data, 171 objects
            411 MB used, 21451 GB / 21451 GB avail
                 360 active+clean

# now check the glances images are being stored correctly
[root@node00 kolla-deploy]# openstack image list
+--------------------------------------+--------+--------+
| ID                                   | Name   | Status |
+--------------------------------------+--------+--------+
| 86896578-3d1a-4214-86c1-97b006304f2d | cirros | active |
+--------------------------------------+--------+--------+

[root@node00 kolla-deploy]# ssh node03  docker exec ceph_mon rbd ls images 
86896578-3d1a-4214-86c1-97b006304f2d
</syntaxhighlight>

== Sample Docker Images Output ==
<syntaxhighlight>
[root@mitaka-head ~]# docker images 
REPOSITORY                                                       TAG                 IMAGE ID            CREATED             SIZE
10.10.10.1:4000/kolla/centos-binary-magnum-api                   3.0.4               d78bd203cff2        3 days ago          736 MB
10.10.10.1:4000/kolla/centos-binary-magnum-conductor             3.0.4               b8f647207273        3 days ago          778 MB
10.10.10.1:4000/kolla/centos-binary-magnum-base                  3.0.4               06da81d5dcc0        3 days ago          713 MB
10.10.10.1:4000/kolla/centos-binary-openstack-base               3.0.4               d5f53a04be6f        3 days ago          563 MB
10.10.10.1:4000/kolla/centos-binary-cinder-api                   3.0.4               70f3a5e554f5        4 days ago          908 MB
10.10.10.1:4000/kolla/centos-binary-cinder-backup                3.0.4               aa042081cd42        4 days ago          896 MB
10.10.10.1:4000/kolla/centos-binary-cinder-volume                3.0.4               5ea20d978a44        4 days ago          869 MB
10.10.10.1:4000/kolla/centos-binary-cinder-rpcbind               3.0.4               2dcf7f6a52f0        4 days ago          896 MB
10.10.10.1:4000/kolla/centos-binary-nova-compute                 3.0.4               f36118d8d566        4 days ago          1.18 GB
10.10.10.1:4000/kolla/centos-binary-gnocchi-metricd              3.0.4               6bade29616fd        4 days ago          900 MB
10.10.10.1:4000/kolla/centos-binary-gnocchi-api                  3.0.4               16dbd9030f28        4 days ago          900 MB
10.10.10.1:4000/kolla/centos-binary-gnocchi-statsd               3.0.4               66a01dfe0086        4 days ago          900 MB
10.10.10.1:4000/kolla/centos-binary-cinder-scheduler             3.0.4               dca0e01ea4b2        4 days ago          867 MB
10.10.10.1:4000/kolla/centos-binary-trove-api                    3.0.4               cd2ea7c5daab        4 days ago          614 MB
10.10.10.1:4000/kolla/centos-binary-nova-libvirt                 3.0.4               2624b17fd29b        4 days ago          1.17 GB
10.10.10.1:4000/kolla/centos-binary-trove-conductor              3.0.4               c373b659c4c8        4 days ago          614 MB
10.10.10.1:4000/kolla/centos-binary-trove-guestagent             3.0.4               29de10abdab6        4 days ago          614 MB
10.10.10.1:4000/kolla/centos-binary-sahara-api                   3.0.4               610d007d006d        4 days ago          624 MB
10.10.10.1:4000/kolla/centos-binary-trove-taskmanager            3.0.4               313e89f059af        4 days ago          614 MB
10.10.10.1:4000/kolla/centos-binary-sahara-engine                3.0.4               e2ce45f721e0        4 days ago          624 MB
10.10.10.1:4000/kolla/centos-binary-nova-compute-ironic          3.0.4               110f5267fa67        4 days ago          1.09 GB
10.10.10.1:4000/kolla/centos-binary-designate-pool-manager       3.0.4               9443ac730c1a        4 days ago          650 MB
10.10.10.1:4000/kolla/centos-binary-designate-central            3.0.4               0fc8feaddace        4 days ago          643 MB
10.10.10.1:4000/kolla/centos-binary-designate-backend-bind9      3.0.4               3fa8aad9af41        4 days ago          650 MB
10.10.10.1:4000/kolla/centos-binary-designate-sink               3.0.4               51e17ddc0aa3        4 days ago          643 MB
10.10.10.1:4000/kolla/centos-binary-designate-api                3.0.4               af28ee2b73d7        4 days ago          643 MB
10.10.10.1:4000/kolla/centos-binary-ironic-conductor             3.0.4               ff90f3d51d4e        4 days ago          734 MB
10.10.10.1:4000/kolla/centos-binary-ironic-inspector             3.0.4               1415866fc44e        4 days ago          628 MB
10.10.10.1:4000/kolla/centos-binary-designate-mdns               3.0.4               9f412c0057fb        4 days ago          643 MB
10.10.10.1:4000/kolla/centos-binary-ironic-api                   3.0.4               5ea4be02bcf8        4 days ago          623 MB
10.10.10.1:4000/kolla/centos-binary-ironic-pxe                   3.0.4               d4485a2ec362        4 days ago          626 MB
10.10.10.1:4000/kolla/centos-binary-swift-proxy-server           3.0.4               e1572d4c0881        4 days ago          611 MB
10.10.10.1:4000/kolla/centos-binary-swift-container              3.0.4               9fc6f2c27987        4 days ago          611 MB
10.10.10.1:4000/kolla/centos-binary-swift-account                3.0.4               5ab93fed4cb1        4 days ago          611 MB
10.10.10.1:4000/kolla/centos-binary-swift-object-expirer         3.0.4               0dc90490613e        4 days ago          611 MB
10.10.10.1:4000/kolla/centos-binary-swift-rsyncd                 3.0.4               a503a9d894d5        4 days ago          592 MB
10.10.10.1:4000/kolla/centos-binary-swift-object                 3.0.4               9d36f1d4be69        4 days ago          611 MB
10.10.10.1:4000/kolla/centos-binary-neutron-openvswitch-agent    3.0.4               d9eea5cbaf66        4 days ago          669 MB
10.10.10.1:4000/kolla/centos-binary-neutron-metadata-agent       3.0.4               cf108ab59425        4 days ago          646 MB
10.10.10.1:4000/kolla/centos-binary-neutron-lbaas-agent          3.0.4               53e4646f80f6        4 days ago          671 MB
10.10.10.1:4000/kolla/centos-binary-neutron-dhcp-agent           3.0.4               478151798644        4 days ago          646 MB
10.10.10.1:4000/kolla/centos-binary-neutron-l3-agent             3.0.4               478151798644        4 days ago          646 MB
10.10.10.1:4000/kolla/centos-binary-neutron-linuxbridge-agent    3.0.4               222faa99b714        4 days ago          669 MB
10.10.10.1:4000/kolla/centos-binary-neutron-server               3.0.4               8f1525d66677        4 days ago          669 MB
10.10.10.1:4000/kolla/centos-binary-neutron-metering-agent       3.0.4               2cd99816449a        4 days ago          669 MB
10.10.10.1:4000/kolla/centos-binary-neutron-vpnaas-agent         3.0.4               f0544ff6f83f        4 days ago          677 MB
10.10.10.1:4000/kolla/centos-binary-nova-spicehtml5proxy         3.0.4               3a13a9952957        4 days ago          706 MB
10.10.10.1:4000/kolla/centos-binary-nova-novncproxy              3.0.4               aaefaa06bc23        4 days ago          666 MB
10.10.10.1:4000/kolla/centos-binary-nova-conductor               3.0.4               3ab8b021c390        4 days ago          665 MB
<none>                                                           <none>              41e68f24c58c        4 days ago          646 MB
10.10.10.1:4000/kolla/centos-binary-nova-ssh                     3.0.4               fad46d27b2f2        4 days ago          666 MB
10.10.10.1:4000/kolla/centos-binary-nova-scheduler               3.0.4               6125313f86cb        4 days ago          665 MB
10.10.10.1:4000/kolla/centos-binary-nova-consoleauth             3.0.4               00e3f0071755        4 days ago          665 MB
10.10.10.1:4000/kolla/centos-binary-nova-api                     3.0.4               f1fe496bcfe9        4 days ago          665 MB
10.10.10.1:4000/kolla/centos-binary-nova-network                 3.0.4               613ae54d2c96        4 days ago          643 MB
10.10.10.1:4000/kolla/centos-binary-nova-serialproxy             3.0.4               1a6b57cc8494        4 days ago          665 MB
10.10.10.1:4000/kolla/centos-binary-heat-api                     3.0.4               0f8b6f127351        4 days ago          618 MB
10.10.10.1:4000/kolla/centos-binary-heat-engine                  3.0.4               de851441fc42        4 days ago          618 MB
10.10.10.1:4000/kolla/centos-binary-heat-api-cfn                 3.0.4               cc82d8ac423a        4 days ago          618 MB
10.10.10.1:4000/kolla/centos-binary-manila-share                 3.0.4               8964a2d55a60        4 days ago          659 MB
10.10.10.1:4000/kolla/centos-binary-magnum-api                   <none>              e8df6d116e47        4 days ago          736 MB
10.10.10.1:4000/kolla/centos-binary-magnum-conductor             <none>              7fa1bac8fc26        4 days ago          778 MB
10.10.10.1:4000/kolla/centos-binary-murano-api                   3.0.4               af3c3b0e65e7        4 days ago          609 MB
10.10.10.1:4000/kolla/centos-binary-ceilometer-api               3.0.4               3650b46a3bd4        4 days ago          658 MB
10.10.10.1:4000/kolla/centos-binary-ceilometer-central           3.0.4               8049b28120c6        4 days ago          658 MB
10.10.10.1:4000/kolla/centos-binary-manila-scheduler             3.0.4               39394a79139b        4 days ago          614 MB
10.10.10.1:4000/kolla/centos-binary-manila-api                   3.0.4               ba7857422288        4 days ago          614 MB
10.10.10.1:4000/kolla/centos-binary-ceilometer-compute           3.0.4               a69956792ab9        4 days ago          658 MB
10.10.10.1:4000/kolla/centos-binary-murano-engine                3.0.4               8cd8126764fa        4 days ago          609 MB
10.10.10.1:4000/kolla/centos-binary-ceilometer-collector         3.0.4               cd82da011309        4 days ago          619 MB
10.10.10.1:4000/kolla/centos-binary-ceilometer-notification      3.0.4               c0e64f186be1        4 days ago          618 MB
10.10.10.1:4000/kolla/centos-binary-cinder-base                  3.0.4               eee63fa966a2        4 days ago          867 MB
10.10.10.1:4000/kolla/centos-binary-glance-api                   3.0.4               dd54b3d0a8dd        4 days ago          767 MB
10.10.10.1:4000/kolla/centos-binary-aodh-api                     3.0.4               ac14af310016        4 days ago          653 MB
10.10.10.1:4000/kolla/centos-binary-aodh-notifier                3.0.4               9971ea02b3ce        4 days ago          653 MB
10.10.10.1:4000/kolla/centos-binary-aodh-listener                3.0.4               86b95ea7357e        4 days ago          653 MB
10.10.10.1:4000/kolla/centos-binary-glance-registry              3.0.4               d5deb86446be        4 days ago          727 MB
10.10.10.1:4000/kolla/centos-binary-gnocchi-base                 3.0.4               d21049a6ee98        4 days ago          874 MB
10.10.10.1:4000/kolla/centos-binary-aodh-evaluator               3.0.4               d69194a554ea        4 days ago          653 MB
10.10.10.1:4000/kolla/centos-binary-aodh-expirer                 3.0.4               bcc23ba2efb5        4 days ago          653 MB
10.10.10.1:4000/kolla/centos-binary-mistral-api                  3.0.4               a0e6640e2a7a        4 days ago          721 MB
10.10.10.1:4000/kolla/centos-binary-mistral-executor             3.0.4               24d3b9511f3b        4 days ago          721 MB
10.10.10.1:4000/kolla/centos-binary-mistral-engine               3.0.4               7492b71c6427        4 days ago          721 MB
10.10.10.1:4000/kolla/centos-binary-keystone-fernet              3.0.4               74ab0570f7d3        4 days ago          655 MB
10.10.10.1:4000/kolla/centos-binary-keystone-ssh                 3.0.4               75dc526d7831        4 days ago          655 MB
10.10.10.1:4000/kolla/centos-binary-barbican-api                 3.0.4               df4ac8990165        4 days ago          617 MB
10.10.10.1:4000/kolla/centos-binary-keystone                     3.0.4               2432e09e913e        4 days ago          633 MB
10.10.10.1:4000/kolla/centos-binary-barbican-keystone-listener   3.0.4               2eacdcfa1542        4 days ago          615 MB
10.10.10.1:4000/kolla/centos-binary-barbican-worker              3.0.4               026000cbe4b9        4 days ago          615 MB
10.10.10.1:4000/kolla/centos-binary-trove-base                   3.0.4               5483308bac67        4 days ago          593 MB
10.10.10.1:4000/kolla/centos-binary-sahara-base                  3.0.4               34ec94aa1835        4 days ago          603 MB
10.10.10.1:4000/kolla/centos-binary-designate-base               3.0.4               ed346abbe9b9        4 days ago          621 MB
10.10.10.1:4000/kolla/centos-binary-ironic-base                  3.0.4               181c88398e5b        4 days ago          602 MB
10.10.10.1:4000/kolla/centos-binary-swift-base                   3.0.4               1adc3171914a        4 days ago          590 MB
10.10.10.1:4000/kolla/centos-binary-neutron-base                 3.0.4               1b289b46f068        4 days ago          646 MB
10.10.10.1:4000/kolla/centos-binary-nova-base                    3.0.4               2de65d31d0d5        4 days ago          643 MB
10.10.10.1:4000/kolla/centos-binary-zaqar                        3.0.4               056bdfbf8c45        4 days ago          624 MB
10.10.10.1:4000/kolla/centos-binary-heat-base                    3.0.4               088b1de82840        4 days ago          597 MB
10.10.10.1:4000/kolla/centos-binary-rally                        3.0.4               378e6d729a1c        4 days ago          617 MB
10.10.10.1:4000/kolla/centos-binary-collectd                     3.0.4               96c68198b1c1        4 days ago          586 MB
10.10.10.1:4000/kolla/centos-binary-horizon                      3.0.4               e66ecb72a90c        4 days ago          798 MB
10.10.10.1:4000/kolla/centos-binary-manila-base                  3.0.4               744aab43c0a6        4 days ago          614 MB
10.10.10.1:4000/kolla/centos-binary-magnum-base                  <none>              fe915ea749d0        4 days ago          713 MB
10.10.10.1:4000/kolla/centos-binary-murano-base                  3.0.4               79e425868dc9        4 days ago          588 MB
10.10.10.1:4000/kolla/centos-binary-ceilometer-base              3.0.4               6aa09e03f06e        4 days ago          597 MB
10.10.10.1:4000/kolla/centos-binary-tempest                      3.0.4               854131b7291c        4 days ago          642 MB
10.10.10.1:4000/kolla/centos-binary-glance-base                  3.0.4               a86429091688        4 days ago          727 MB
10.10.10.1:4000/kolla/centos-binary-aodh-base                    3.0.4               6d8952a93c01        4 days ago          631 MB
10.10.10.1:4000/kolla/centos-binary-mistral-base                 3.0.4               7312fbbf0859        4 days ago          698 MB
10.10.10.1:4000/kolla/centos-binary-keystone-base                3.0.4               87b5b322b9c7        4 days ago          633 MB
10.10.10.1:4000/kolla/centos-binary-dind                         3.0.4               f44486475b3e        4 days ago          563 MB
10.10.10.1:4000/kolla/centos-binary-barbican-base                3.0.4               3e5e7a8f6fba        4 days ago          594 MB
10.10.10.1:4000/kolla/centos-binary-kolla-toolbox                3.0.4               2cd463f3cbce        4 days ago          712 MB
<none>                                                           <none>              cc72267144fa        4 days ago          563 MB
10.10.10.1:4000/kolla/centos-binary-openstack-base               <none>              207b9fd4155c        4 days ago          563 MB
10.10.10.1:4000/kolla/centos-binary-ceph-mon                     3.0.4               0793b4e7ab13        4 days ago          680 MB
10.10.10.1:4000/kolla/centos-binary-mariadb                      3.0.4               b998b4aabcee        4 days ago          789 MB
10.10.10.1:4000/kolla/centos-binary-ceph-osd                     3.0.4               3d97cb4483cb        4 days ago          680 MB
10.10.10.1:4000/kolla/centos-binary-grafana                      3.0.4               264841c74653        4 days ago          536 MB
10.10.10.1:4000/kolla/centos-binary-ceph-rgw                     3.0.4               17fffb4aa582        4 days ago          680 MB
10.10.10.1:4000/kolla/centos-binary-ceph-base                    3.0.4               5b1dbb3bb9ff        4 days ago          680 MB
10.10.10.1:4000/kolla/centos-binary-openvswitch-db-server        3.0.4               9b46efbc7f66        4 days ago          410 MB
10.10.10.1:4000/kolla/centos-binary-iscsid                       3.0.4               b34845092c83        4 days ago          403 MB
10.10.10.1:4000/kolla/centos-binary-openvswitch-vswitchd         3.0.4               782e4c3db61f        4 days ago          410 MB
10.10.10.1:4000/kolla/centos-binary-memcached                    3.0.4               061fe09dfd0a        4 days ago          400 MB
10.10.10.1:4000/kolla/centos-binary-kibana                       3.0.4               78c3c2f08ae1        4 days ago          534 MB
10.10.10.1:4000/kolla/centos-binary-heka                         3.0.4               5be809a8f701        4 days ago          453 MB
10.10.10.1:4000/kolla/centos-binary-elasticsearch                3.0.4               20582ad45b37        4 days ago          607 MB
10.10.10.1:4000/kolla/centos-binary-dnsmasq                      3.0.4               5ede222f22fe        4 days ago          399 MB
10.10.10.1:4000/kolla/centos-binary-keepalived                   3.0.4               61ef2dd501bb        4 days ago          405 MB
10.10.10.1:4000/kolla/centos-binary-telegraf                     3.0.4               fb74d4f362fe        4 days ago          432 MB
10.10.10.1:4000/kolla/centos-binary-rabbitmq                     3.0.4               4949ffe59699        4 days ago          440 MB
10.10.10.1:4000/kolla/centos-binary-etcd                         3.0.4               26799eb72e7a        4 days ago          428 MB
10.10.10.1:4000/kolla/centos-binary-openvswitch-base             3.0.4               23f4c55b8dd6        4 days ago          410 MB
10.10.10.1:4000/kolla/centos-binary-haproxy                      3.0.4               8c4b31f52f01        4 days ago          420 MB
10.10.10.1:4000/kolla/centos-binary-influxdb                     3.0.4               0bfccba88174        4 days ago          454 MB
10.10.10.1:4000/kolla/centos-binary-mongodb                      3.0.4               a55285903cbf        4 days ago          571 MB
10.10.10.1:4000/kolla/centos-binary-cron                         3.0.4               1e7ec4979bd3        4 days ago          399 MB
10.10.10.1:4000/kolla/centos-binary-tgtd                         3.0.4               8110cd117d81        4 days ago          381 MB
10.10.10.1:4000/kolla/centos-binary-multipathd                   3.0.4               b50eb31c130c        4 days ago          409 MB
10.10.10.1:4000/kolla/centos-binary-base                         3.0.4               c33393731ca5        4 days ago          379 MB
centos                                                           7                   98d35105a391        2 weeks ago         192 MB
registry                                                         2                   047218491f8c        4 weeks ago         33.2 MB

</syntaxhighlight>

== Port Forwards ==
<syntaxhighlight>
firewall-cmd --zone=external --add-forward-port=port=443:proto=tcp:toport=443:toaddr=10.10.10.254 --permanent
firewall-cmd --zone=external --add-forward-port=port=80:proto=tcp:toport=80:toaddr=10.10.10.254 --permanent
firewall-cmd --zone=external --add-forward-port=port=6080:proto=tcp:toport=6080:toaddr=10.10.10.254 --permanent

firewall-cmd --zone=external --add-forward-port=port=5000:proto=tcp:toport=5000:toaddr=10.10.10.254 --permanent
firewall-cmd --zone=external --add-forward-port=port=35357:proto=tcp:toport=35357:toaddr=10.10.10.254 --permanent

firewall-cmd --zone=external --add-forward-port=port=8776:proto=tcp:toport=8776:toaddr=10.10.10.254 --permanent
firewall-cmd --zone=external --add-forward-port=port=8774:proto=tcp:toport=8774:toaddr=10.10.10.254 --permanent
firewall-cmd --zone=external --add-forward-port=port=8386:proto=tcp:toport=8386:toaddr=10.10.10.254 --permanent
firewall-cmd --zone=external --add-forward-port=port=9292:proto=tcp:toport=9292:toaddr=10.10.10.254 --permanent
firewall-cmd --zone=external --add-forward-port=port=9696:proto=tcp:toport=9696:toaddr=10.10.10.254 --permanent
firewall-cmd --zone=external --add-forward-port=port=6000:proto=tcp:toport=6000:toaddr=10.10.10.254 --permanent
firewall-cmd --zone=external --add-forward-port=port=6001:proto=tcp:toport=6001:toaddr=10.10.10.254 --permanent
firewall-cmd --zone=external --add-forward-port=port=6002:proto=tcp:toport=6002:toaddr=10.10.10.254 --permanent
firewall-cmd --zone=external --add-forward-port=port=8004:proto=tcp:toport=8004:toaddr=10.10.10.254 --permanent
firewall-cmd --zone=external --add-forward-port=port=8777:proto=tcp:toport=8777:toaddr=10.10.10.254 --permanent
firewall-cmd --zone=external --add-forward-port=port=8000:proto=tcp:toport=8000:toaddr=10.10.10.254 --permanent

firewall-cmd --reload
</syntaxhighlight>