VScaler: Thoubleshooting Kolla issues

Log location

The logs are on the nodes under: /var/lib/docker/volumes/kolla_logs/_data/

When a service fails you will find useful info in the koala logs of the container of that service. To check the logs of nova-conductor service for example we'll do:

[root@head01 ~]# ssh controller01
[root@controller01-enp2s0 ~]# tail /var/lib/docker/volumes/kolla_logs/_data/nova/nova-conductor.log

Interface ansible_<if> does not exist

If you see a message of this sort in the kolla-ansible output, it's most likely referring to a node that has an interface with a different name than the one specified in the "network_interface" variable in the /etc/kolla/globals.yaml file.

Solution

To get past this issue, just add api_interface=ens5 next to the node's name in the inventory file. I've had a time when I need to add tunnel_interface=ens5 as well. The error message will tell you if the tunnel or the api one is the problem. The inventory file should look like this:

...
gpu01-ens5 tunnel_interface=ens5 api_interface=ens5
...

Connection refused errors in nova-conductor logs

When I was doing a deploy the nova service wouldn't come up properly. After checking the nova-conductor logs on the controller node that reported the error, I saw a lot of errors like this:

ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.106:5672 is unreachable: [Errno 111] ECONNREFUSED

Solution

Disable SELINUX on the controller nodes and reboot them!