VScaler: Thoubleshooting Kolla issues

Log location

The logs are on the nodes under: /var/lib/docker/volumes/kolla_logs/_data/

When a service fails you will find useful info in the koala logs of the container of that service. To check the logs of nova-conductor service for example we'll do:

[root@head01 ~]# ssh controller01
[root@controller01-enp2s0 ~]# tail /var/lib/docker/volumes/kolla_logs/_data/nova/nova-conductor.log

Interface ansible_<if> does not exist

If you see a message of this sort in the kolla-ansible output, it's most likely referring to a node that has an interface with a different name than the one specified in the "network_interface" variable in the /etc/kolla/globals.yaml file.

Solution

To get past this issue, just add api_interface=ens5 next to the node's name in the inventory file. I've had a time when I need to add tunnel_interface=ens5 as well. The error message will tell you if the tunnel or the api one is the problem. The inventory file should look like this:

...
gpu01-ens5 tunnel_interface=ens5 api_interface=ens5
...

Connection refused errors in nova-conductor logs

When I was doing a deploy the nova service wouldn't come up properly. After checking the nova-conductor logs on the controller node that reported the error, I saw a lot of errors like this:

ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.106:5672 is unreachable: [Errno 111] ECONNREFUSED

Solution

Disable SELINUX on the controller nodes and reboot them!

Debugging containers that dont start

When a container fails to start - you can recreate the error using the image name and then pass -a to docker start:

[root@controller01 ~]# docker ps -a
CONTAINER ID        IMAGE                                                     COMMAND             CREATED             STATUS                    PORTS               NAMES
4de70ccf5a4d        10.10.10.1:4000/kolla/centos-binary-glance-api:4.0.3      "kolla_start"       10 hours ago        Exited (1) 10 hours ago                       bootstrap_glance
10f4038a7d77        10.10.10.1:4000/kolla/centos-binary-keystone:4.0.3        "kolla_start"       10 hours ago        Up 10 hours                                   keystone
e96bf1cb3258        10.10.10.1:4000/kolla/centos-binary-rabbitmq:4.0.3        "kolla_start"       10 hours ago        Up 10 hours                                   rabbitmq
b0094b42cb75        10.10.10.1:4000/kolla/centos-binary-mariadb:4.0.3         "kolla_start"       10 hours ago        Up 10 hours                                   mariadb
d49e0b00bf84        10.10.10.1:4000/kolla/centos-binary-memcached:4.0.3       "kolla_start"       10 hours ago        Up 10 hours                                   memcached
1a1599296c59        10.10.10.1:4000/kolla/centos-binary-keepalived:4.0.3      "kolla_start"       10 hours ago        Up 10 hours                                   keepalived
accc84f93171        10.10.10.1:4000/kolla/centos-binary-haproxy:4.0.3         "kolla_start"       10 hours ago        Up 10 hours                                   haproxy
f25d30f403d2        10.10.10.1:4000/kolla/centos-binary-cron:4.0.3            "kolla_start"       10 hours ago        Up 10 hours                                   cron
0be143a36b6d        10.10.10.1:4000/kolla/centos-binary-kolla-toolbox:4.0.3   "kolla_start"       10 hours ago        Up 10 hours                                   kolla_toolbox
2f667f97a160        10.10.10.1:4000/kolla/centos-binary-fluentd:4.0.3         "kolla_start"       10 hours ago        Up 10 hours                                   fluentd
[root@controller01 ~]# docker start -a  bootstrap_glance
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting file /etc/glance/glance-api.conf
INFO:__main__:Coping file from /var/lib/kolla/config_files/glance-api.conf to /etc/glance/glance-api.conf
INFO:__main__:Setting file /etc/glance/glance-api.conf owner to glance:glance
INFO:__main__:Setting file /etc/glance/glance-api.conf permission to 0600
ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/ceph.* file is not found