http://wiki.define-technology.com/mediawiki-1.35.0/index.php?action=history&feed=atom&title=VScaler%3A_Debugging_kolla_ceph_OSD_issues 2026-05-05T00:03:04Z Revision history for this page on the wiki MediaWiki 1.35.0 http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=VScaler:_Debugging_kolla_ceph_OSD_issues&diff=25703&oldid=prev 2018-02-19T11:28:38Z

<p>Added a link to VScaler: Adding and removing kolla ceph OSDs</p> <table class="diff diff-contentalign-left diff-editfont-monospace" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 11:28, 19 February 2018</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l59" >Line 59:</td> <td colspan="2" class="diff-lineno">Line 59:</td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>A lof of the errors above occured. Looks like we have a faulty disk. depending on the issue, wipe and readd to ceph or replace and readd.  </div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>A lof of the errors above occured. Looks like we have a faulty disk. depending on the issue, wipe and readd to ceph or replace and readd.  </div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr> <tr><td class='diff-marker'>−</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del class="diffchange diffchange-inline">MArtin please add link here to replacing an OSD in </del>kolla<del class="diffchange diffchange-inline">. </del></div></td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline">* [[VScaler: Adding and removing </ins>kolla <ins class="diffchange diffchange-inline">ceph OSDs]]</ins></div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Sidenote: we can also run smart tests on the disk</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Sidenote: we can also run smart tests on the disk</div></td></tr> </table>

Martin t http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=VScaler:_Debugging_kolla_ceph_OSD_issues&diff=25684&oldid=prev 2018-02-19T11:10:45Z

<p>Created page with &quot;== Check the OSD status == &lt;syntaxhighlight&gt; [root@node02-enp175s0f0 ~]# docker exec ceph_mon ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 16.0000...&quot;</p> <p><b>New page</b></p><div>== Check the OSD status ==<br /> &lt;syntaxhighlight&gt;<br /> [root@node02-enp175s0f0 ~]# docker exec ceph_mon ceph osd tree<br /> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY <br /> -1 16.00000 root default <br /> -2 4.00000 host 10.10.20.12 <br /> 5 1.00000 osd.5 down 0 1.00000 <br /> 8 1.00000 osd.8 up 1.00000 1.00000 <br /> 13 1.00000 osd.13 up 1.00000 1.00000 <br /> 16 1.00000 osd.16 up 1.00000 1.00000 <br /> -3 4.00000 host 10.10.20.13 <br /> 2 1.00000 osd.2 up 1.00000 1.00000 <br /> 4 1.00000 osd.4 up 1.00000 1.00000 <br /> 9 1.00000 osd.9 up 1.00000 1.00000 <br /> 12 1.00000 osd.12 up 1.00000 1.00000 <br /> -4 4.00000 host 10.10.20.11 <br /> 1 1.00000 osd.1 up 1.00000 1.00000 <br /> 6 1.00000 osd.6 up 1.00000 1.00000 <br /> 10 1.00000 osd.10 up 1.00000 1.00000 <br /> 14 1.00000 osd.14 up 1.00000 1.00000 <br /> -5 4.00000 host 10.10.20.10 <br /> 3 1.00000 osd.3 up 1.00000 1.00000 <br /> 7 1.00000 osd.7 up 1.00000 1.00000 <br /> 11 1.00000 osd.11 up 1.00000 1.00000 <br /> 15 1.00000 osd.15 up 1.00000 1.00000 <br /> &lt;/syntaxhighlight&gt;<br /> <br /> So we can see OSD 5 is down above. Lets look at docker to see whats happening (we see the container is stuck restarting - with an unexpected error (fairly useless error!) <br /> &lt;syntaxhighlight&gt;<br /> [root@node02-enp175s0f0 ~]# docker ps | grep -i osd_5 <br /> 5bce4c98e95a registry.vscaler.com:5000/kolla/centos-binary-ceph-osd:4.0.3 &quot;kolla_start&quot; 9 weeks ago Restarting (134) 13 hours ago ceph_osd_5<br /> <br /> [root@node02-enp175s0f0 ~]# docker logs ceph_osd_5 | grep -i fail | tail -n 3<br /> os/filestore/FileStore.cc: 2920: FAILED assert(0 == &quot;unexpected error&quot;)<br /> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x561f203c7ad5]<br /> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x267) [0x561f203c7cb7]<br /> &lt;/syntaxhighlight&gt;<br /> <br /> Ok lets drill down some more. What physical device is backing OSD 5<br /> &lt;syntaxhighlight&gt;<br /> [root@node02-enp175s0f0 ~]# docker inspect ceph_osd_5 | grep -i &#039;/var/lib/ceph/osd&#039; <br /> &quot;/var/lib/ceph/osd/1d29b3f9-e8c6-406d-b881-9eb6ca878d28:/var/lib/ceph/osd/ceph-5:rw&quot;,<br /> &quot;Source&quot;: &quot;/var/lib/ceph/osd/1d29b3f9-e8c6-406d-b881-9eb6ca878d28&quot;,<br /> &quot;Destination&quot;: &quot;/var/lib/ceph/osd/ceph-5&quot;,<br /> &quot;/var/lib/ceph/osd/ceph-5&quot;: {},<br /> [root@node02-enp175s0f0 ~]# df | grep /var/lib/ceph/osd/1d29b3f9-e8c6-406d-b881-9eb6ca878d28<br /> /dev/sde2 1869578324 50644684 1818933640 3% /var/lib/ceph/osd/1d29b3f9-e8c6-406d-b881-9eb6ca878d28<br /> &lt;/syntaxhighlight&gt;<br /> <br /> So know we know OSD5 is sde on the host system. Lets check dmesg for IO errors. <br /> &lt;syntaxhighlight&gt;<br /> [849146.277876] sd 10:0:0:0: [sde] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE<br /> [849146.285800] sd 10:0:0:0: [sde] Sense Key : Medium Error [current] [descriptor] <br /> [849146.293209] sd 10:0:0:0: [sde] Add. Sense: Unrecovered read error - auto reallocate failed<br /> [849146.301580] sd 10:0:0:0: [sde] CDB: Read(10) 28 00 70 88 68 b0 00 00 08 00<br /> [849146.308550] blk_update_request: I/O error, dev sde, sector 1887987888<br /> &lt;/syntaxhighlight&gt;<br /> <br /> A lof of the errors above occured. Looks like we have a faulty disk. depending on the issue, wipe and readd to ceph or replace and readd. <br /> <br /> MArtin please add link here to replacing an OSD in kolla. <br /> <br /> Sidenote: we can also run smart tests on the disk<br /> &lt;syntaxhighlight&gt;<br /> smartctl -t short -a /dev/sde<br /> smartctl -t long -a /dev/sde<br /> &lt;/syntaxhighlight&gt;</div>

David