VScaler: Thoubleshooting Kolla issues - Revision history

Martin t: Added "Waiting for nova-compute service up timing out due to rabbitmq breaking"

2018-03-14T10:36:42Z

Added "Waiting for nova-compute service up timing out due to rabbitmq breaking"

Martin t: Moved bifrost related issues to http://wiki.bostonlabs.co.uk/w/index.php?title=VScaler:_Thoubleshooting_bifrost_issues

2017-12-08T15:02:53Z

Moved bifrost related issues to http://wiki.bostonlabs.co.uk/w/index.php?title=VScaler:_Thoubleshooting_bifrost_issues

Show changes

Martin t: bifrost issues and workarounds

2017-11-21T12:26:34Z

bifrost issues and workarounds

Martin t at 10:16, 21 November 2017

2017-11-21T10:16:22Z

Martin t at 17:11, 20 November 2017

2017-11-20T17:11:59Z

Martin t: Missing rabbitmq-server during bifrost container deploy

2017-11-20T12:57:49Z

Missing rabbitmq-server during bifrost container deploy

Martin t: bifrost-base source image build failure

2017-11-20T12:03:18Z

bifrost-base source image build failure

Martin t: /* "AttributeError: 'module' object has no attribute 'APIClient'" during kolla-build */

2017-11-20T09:49:44Z

"AttributeError: 'module' object has no attribute 'APIClient'" during kolla-build

Martin t: "AttributeError: 'module' object has no attribute 'APIClient'" during kolla-build workaround

2017-11-20T09:48:28Z

"AttributeError: 'module' object has no attribute 'APIClient'" during kolla-build workaround

David at 09:47, 24 August 2017

2017-08-24T09:47:39Z

@@ Line 105: / Line 105: @@
 <syntaxhighlight>
 pip install docker==2.4
 </syntaxhighlight>

← Older revision		Revision as of 12:26, 21 November 2017
Line 161:		Line 161:

	Follow the guide at [https://docs.openstack.org/project-install-guide/baremetal/draft/configure-pxe.html] to install and configure PXE/iPXE.		Follow the guide at [https://docs.openstack.org/project-install-guide/baremetal/draft/configure-pxe.html] to install and configure PXE/iPXE.
		+
		+	== nginx is missing during bifrost deploy==
		+
		+	Install nginx:
		+	<syntaxhighlight>
		+	yum install nginx
		+	</syntaxhighlight>
		+
		+	== SELinux related tasks failing due to SELinux being disabled in the bifrost deploy container ==
		+
		+	Symptoms:
		+	<syntaxhighlight>
		+	TASK [bifrost-ironic-install : Explicitly allow nginx and IPA port (TCP) on selinux] ***
		+	task path: /bifrost-base-source/bifrost-4.0.0/playbooks/roles/bifrost-ironic-install/tasks/bootstrap.yml:260
		+	fatal: [127.0.0.1]: FAILED! => {"failed": true, "msg": "The conditional check '(ansible_os_family == 'RedHat' or ansible_os_family == 'Suse') and ansible_selinux.status == 'enabled' and ansible_selinux.mode == \"enforcing\"' failed. The error was: error while evaluating conditional ((ansible_os_family == 'RedHat' or ansible_os_family == 'Suse') and ansible_selinux.status == 'enabled' and ansible_selinux.mode == \"enforcing\"): 'bool object' has no attribute 'status'\n\nThe error appears to have been in '/bifrost-base-source/bifrost-4.0.0/playbooks/roles/bifrost-ironic-install/tasks/bootstrap.yml': line 259, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n iptables -I INPUT -p tcp --dport 6385 -i {{ network_interface }} -j ACCEPT\n- block:\n ^ here\n"}
		+	</syntaxhighlight>
		+
		+	Solution:
		+	Edit <syntaxhighlight>vi /bifrost-base-source/bifrost-4.0.0/playbooks/roles/bifrost-ironic-install/tasks/bootstrap.yml</syntaxhighlight>
		+	Comment out or remove the following block:
		+	<syntaxhighlight>
		+	- block:
		+	- name: "Explicitly allow nginx and IPA port (TCP) on selinux"
		+	seport:
		+	ports: "{{ file_url_port }},6385"
		+	proto: tcp
		+	setype: http_port_t
		+	state: present
		+
		+	- name: "Add proper context on created data for http_boot"
		+	command: semanage fcontext -a -t httpd_sys_content_t "{{ http_boot_folder }}(/.*)?"
		+
		+	- name: Copy ironic policy file to temporary directory
		+	copy:
		+	src: ironic_policy.te
		+	dest: /tmp/ironic_policy.te
		+
		+	- name: Check ironic policy module
		+	command: checkmodule -M -m -o /tmp/ironic_policy.mod /tmp/ironic_policy.te
		+
		+	- name: Package ironic policy module
		+	command: semodule_package -m /tmp/ironic_policy.mod -o /tmp/ironic_policy.pp
		+
		+	- name: Include ironic policy module
		+	command: semodule -i /tmp/ironic_policy.pp
		+
		+	- name: Enable ironic policy module
		+	command: semodule -e ironic_policy
		+	when: (ansible_os_family == 'RedHat' or ansible_os_family == 'Suse') and
		+	ansible_selinux.status == 'enabled' and ansible_selinux.mode == "enforcing"
		+	</syntaxhighlight>
		+
		+	== Image-build failing during bitfrost deploy due to missing diskimage-builder ==
		+
		+	Symptoms:
		+	<syntaxhighlight>
		+	TASK [bifrost-create-dib-image : Initiate image build] *************************
		+	task path: /bifrost-base-source/bifrost-4.0.0/playbooks/roles/bifrost-create-dib-image/tasks/main.yml:121
		+	<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
		+	<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1511265018.75-262833849350496 `" && echo ansible-tmp-1511265018.75-262833849350496="` echo $HOME/.ansible/tmp/ansible-tmp-1511265018.75-262833849350496 `" ) && sleep 0'
		+	<127.0.0.1> PUT /tmp/tmprxrYqP TO /root/.ansible/tmp/ansible-tmp-1511265018.75-262833849350496/command
		+	<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1511265018.75-262833849350496/ /root/.ansible/tmp/ansible-tmp-1511265018.75-262833849350496/command && sleep 0'
		+	<127.0.0.1> EXEC /bin/sh -c 'LANG=C LC_MESSAGES=C no_proxy='"'"''"'"' http_proxy='"'"''"'"' https_proxy='"'"''"'"' LC_ALL=C DIB_INSTALLTYPE_simple_init=repo /var/lib/kolla/venv/bin/python /root/.ansible/tmp/ansible-tmp-1511265018.75-262833849350496/command; rm -rf "/root/.ansible/tmp/ansible-tmp-1511265018.75-262833849350496/" > /dev/null 2>&1 && sleep 0'
		+	fatal: [127.0.0.1]: FAILED! => {"changed": false, "cmd": "disk-image-create -o /httpboot/deployment_image.qcow2 -t qcow2 centos vm enable-serial-console simple-init", "failed": true, "invocation": {"module_args": {"_raw_params": "disk-image-create -o /httpboot/deployment_image.qcow2 -t qcow2 centos vm enable-serial-console simple-init ", "_uses_shell": false, "chdir": null, "creates": null, "executable": null, "removes": null, "warn": true}, "module_name": "command"}, "msg": "[Errno 2] No such file or directory", "rc": 2}
		+	</syntaxhighlight>
		+
		+	Solution:
		+	Install diskimage-builder
		+	<syntaxhighlight>
		+	yum install diskimage-builder
		+	</syntaxhighlight>
		+
		+	Note: Make sure <syntaxhighlight>dib_os_element: centos7</syntaxhighlight> is set in <syntaxhighlight>/etc/bifrost/dib.yml</syntaxhighlight> instead of <syntaxhighlight>dib_os_element: centos</syntaxhighlight>
		+	Otherwise the following failure may occur:
		+	<syntaxhighlight>
		+	TASK [bifrost-create-dib-image : Initiate image build] *************************
		+	task path: /bifrost-base-source/bifrost-4.0.0/playbooks/roles/bifrost-create-dib-image/tasks/main.yml:121
		+	<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
		+	<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1511265599.4-237699719654061 `" && echo ansible-tmp-1511265599.4-237699719654061="` echo $HOME/.ansible/tmp/ansible-tmp-1511265599.4-237699719654061 `" ) && sleep 0'
		+	<127.0.0.1> PUT /tmp/tmpeWn782 TO /root/.ansible/tmp/ansible-tmp-1511265599.4-237699719654061/command
		+	<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1511265599.4-237699719654061/ /root/.ansible/tmp/ansible-tmp-1511265599.4-237699719654061/command && sleep 0'
		+	<127.0.0.1> EXEC /bin/sh -c 'LANG=C LC_MESSAGES=C no_proxy='"'"''"'"' http_proxy='"'"''"'"' https_proxy='"'"''"'"' LC_ALL=C DIB_INSTALLTYPE_simple_init=repo /var/lib/kolla/venv/bin/python /root/.ansible/tmp/ansible-tmp-1511265599.4-237699719654061/command; rm -rf "/root/.ansible/tmp/ansible-tmp-1511265599.4-237699719654061/" > /dev/null 2>&1 && sleep 0'
		+	fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": ["disk-image-create", "-o", "/httpboot/deployment_image.qcow2", "-t", "qcow2", "centos", "vm", "enable-serial-console", "simple-init"], "delta": "0:00:00.740333", "end": "2017-11-21 12:00:00.311265", "failed": true, "invocation": {"module_args": {"_raw_params": "disk-image-create -o /httpboot/deployment_image.qcow2 -t qcow2 centos vm enable-serial-console simple-init ", "_uses_shell": false, "chdir": null, "creates": null, "executable": null, "removes": null, "warn": true}, "module_name": "command"}, "rc": 1, "start": "2017-11-21 11:59:59.570932", "stderr": "Traceback (most recent call last):\n File \"/usr/bin/element-info\", line 10, in <module>\n sys.exit(main())\n File \"/usr/lib/python2.7/site-packages/diskimage_builder/element_dependencies.py\", line 337, in main\n elements = _get_elements(args.elements)\n File \"/usr/lib/python2.7/site-packages/diskimage_builder/element_dependencies.py\", line 248, in _get_elements\n return _expand_element_dependencies(elements, all_elements)\n File \"/usr/lib/python2.7/site-packages/diskimage_builder/element_dependencies.py\", line 148, in _expand_element_dependencies\n raise MissingElementException(\"Element '%s' not found\" % element)\ndiskimage_builder.element_dependencies.MissingElementException: Element 'centos' not found", "stdout": "diskimage-builder version 2.8.0\nBuilding elements: base centos vm enable-serial-console simple-init", "stdout_lines": ["diskimage-builder version 2.8.0", "Building elements: base centos vm enable-serial-console simple-init"], "warnings": []}
		+	</syntaxhighlight>

← Older revision		Revision as of 10:16, 21 November 2017
Line 157:		Line 157:
	yum install openstack-ironic-common openstack-ironic-api openstack-ironic-conductor		yum install openstack-ironic-common openstack-ironic-api openstack-ironic-conductor
	</syntaxhighlight>		</syntaxhighlight>
		+
		+	== PXE/iPXE (pxelinux.0/undionly.kpxe) missing during bifrost deploy ==
		+
		+	Follow the guide at [https://docs.openstack.org/project-install-guide/baremetal/draft/configure-pxe.html] to install and configure PXE/iPXE.

@@ Line 146: / Line 146: @@
 docker exec -it bifrost_deploy bash
 yum install rabbitmq-server
 </syntaxhighlight>

@@ Line 127: / Line 127: @@
 + RUN bash -c 'sed -e "s/\-\-force\-reinstall //g" -i /bifrost/playbooks/roles/bifrost-{ironic,keystone}-install/tasks/install.yml' \
      && bash -c './scripts/env-setup.sh && source ./env-vars && \
 </syntaxhighlight>

@@ Line 63: / Line 63: @@
 == "AttributeError: 'module' object has no attribute 'APIClient'" during kolla-build ==
 <syntaxhighlight>
@@ Line 79: / Line 80: @@
 </syntaxhighlight>
-Worked around by removing docker-py and installing docker==2.4.
+Work around by removing docker-py and installing docker==2.4.
 <syntaxhighlight>
@@ Line 100: / Line 101: @@
 </syntaxhighlight>
-Downgrade docker:
+Downgrade docker to work around:
 <syntaxhighlight>
 pip install docker==2.4
 </syntaxhighlight>

@@ Line 60: / Line 60: @@
 INFO:__main__:Setting file /etc/glance/glance-api.conf permission to 0600
 ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/ceph.* file is not found
 </syntaxhighlight>

← Older revision		Revision as of 09:47, 24 August 2017
Line 33:		Line 33:

	Disable SELINUX on the controller nodes and reboot them!		Disable SELINUX on the controller nodes and reboot them!
		+
		+	== Debugging containers that dont start ==
		+	When a container fails to start - you can recreate the error using the image name and then pass -a to docker start:
		+
		+	<syntaxhighlight>
		+	[root@controller01 ~]# docker ps -a
		+	CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
		+	4de70ccf5a4d 10.10.10.1:4000/kolla/centos-binary-glance-api:4.0.3 "kolla_start" 10 hours ago Exited (1) 10 hours ago bootstrap_glance
		+	10f4038a7d77 10.10.10.1:4000/kolla/centos-binary-keystone:4.0.3 "kolla_start" 10 hours ago Up 10 hours keystone
		+	e96bf1cb3258 10.10.10.1:4000/kolla/centos-binary-rabbitmq:4.0.3 "kolla_start" 10 hours ago Up 10 hours rabbitmq
		+	b0094b42cb75 10.10.10.1:4000/kolla/centos-binary-mariadb:4.0.3 "kolla_start" 10 hours ago Up 10 hours mariadb
		+	d49e0b00bf84 10.10.10.1:4000/kolla/centos-binary-memcached:4.0.3 "kolla_start" 10 hours ago Up 10 hours memcached
		+	1a1599296c59 10.10.10.1:4000/kolla/centos-binary-keepalived:4.0.3 "kolla_start" 10 hours ago Up 10 hours keepalived
		+	accc84f93171 10.10.10.1:4000/kolla/centos-binary-haproxy:4.0.3 "kolla_start" 10 hours ago Up 10 hours haproxy
		+	f25d30f403d2 10.10.10.1:4000/kolla/centos-binary-cron:4.0.3 "kolla_start" 10 hours ago Up 10 hours cron
		+	0be143a36b6d 10.10.10.1:4000/kolla/centos-binary-kolla-toolbox:4.0.3 "kolla_start" 10 hours ago Up 10 hours kolla_toolbox
		+	2f667f97a160 10.10.10.1:4000/kolla/centos-binary-fluentd:4.0.3 "kolla_start" 10 hours ago Up 10 hours fluentd
		+	[root@controller01 ~]# docker start -a bootstrap_glance
		+	INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
		+	INFO:__main__:Validating config file
		+	INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
		+	INFO:__main__:Copying service configuration files
		+	INFO:__main__:Deleting file /etc/glance/glance-api.conf
		+	INFO:__main__:Coping file from /var/lib/kolla/config_files/glance-api.conf to /etc/glance/glance-api.conf
		+	INFO:__main__:Setting file /etc/glance/glance-api.conf owner to glance:glance
		+	INFO:__main__:Setting file /etc/glance/glance-api.conf permission to 0600
		+	ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/ceph.* file is not found
		+	</syntaxhighlight>