Working with CRUSH for SSD and HDD pools
Jump to navigation
Jump to search
== Crush Rules ==
In this scenario we have a few HCI nodes running the standard volumes for OpenStack. There's two additional servers, 1x HDD and 1x SSD server. We are going to create a crush rule to isolate these systems and drives in seperate pools in Ceph. At this point it is assumed that you have a ceph cluster spun up and configured and all OSDs are added and online.
```bash
# this command could also be: $ ceph osd crush tree
[root@ukr-dpy1 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 1489.06531 root default
-7 43.65807 host ukr-hci1-n1-mlnx
7 hdd 14.55269 osd.7 up 1.00000 1.00000
15 hdd 14.55269 osd.15 up 1.00000 1.00000
23 hdd 14.55269 osd.23 up 1.00000 1.00000
-9 43.65807 host ukr-hci1-n2-mlnx
6 hdd 14.55269 osd.6 up 1.00000 1.00000
14 hdd 14.55269 osd.14 up 1.00000 1.00000
22 hdd 14.55269 osd.22 up 1.00000 1.00000
-11 43.65807 host ukr-hci1-n3-mlnx
5 hdd 14.55269 osd.5 up 1.00000 1.00000
11 hdd 14.55269 osd.11 up 1.00000 1.00000
21 hdd 14.55269 osd.21 up 1.00000 1.00000
-17 43.65807 host ukr-hci1-n4-mlnx
2 hdd 14.55269 osd.2 up 1.00000 1.00000
13 hdd 14.55269 osd.13 up 1.00000 1.00000
19 hdd 14.55269 osd.19 up 1.00000 1.00000
-13 43.65807 host ukr-hci2-n1-mlnx
0 hdd 14.55269 osd.0 up 1.00000 1.00000
12 hdd 14.55269 osd.12 up 1.00000 1.00000
17 hdd 14.55269 osd.17 up 1.00000 1.00000
-15 43.65807 host ukr-hci2-n2-mlnx
1 hdd 14.55269 osd.1 up 1.00000 1.00000
10 hdd 14.55269 osd.10 up 1.00000 1.00000
16 hdd 14.55269 osd.16 up 1.00000 1.00000
-3 43.65807 host ukr-hci2-n3-mlnx
3 hdd 14.55269 osd.3 up 1.00000 1.00000
8 hdd 14.55269 osd.8 up 1.00000 1.00000
18 hdd 14.55269 osd.18 up 1.00000 1.00000
-5 43.65807 host ukr-hci2-n4-mlnx
4 hdd 14.55269 osd.4 up 1.00000 1.00000
9 hdd 14.55269 osd.9 up 1.00000 1.00000
20 hdd 14.55269 osd.20 up 1.00000 1.00000
-31 902.26672 host ukr-hdd1
58 hdd 14.55269 osd.58 up 1.00000 1.00000
<snip for space>
115 hdd 14.55269 osd.115 up 1.00000 1.00000
116 hdd 14.55269 osd.116 up 1.00000 1.00000
117 hdd 14.55269 osd.117 up 1.00000 1.00000
118 hdd 14.55269 osd.118 up 1.00000 1.00000
119 hdd 14.55269 osd.119 up 1.00000 1.00000
-28 237.53412 host ukr-ssd1-mlnx
24 ssd 6.98630 osd.24 up 1.00000 1.00000
53 ssd 6.98630 osd.53 up 1.00000 1.00000
<snip for space>
54 ssd 6.98630 osd.54 up 1.00000 1.00000
55 ssd 6.98630 osd.55 up 1.00000 1.00000
56 ssd 6.98630 osd.56 up 1.00000 1.00000
57 ssd 6.98630 osd.57 up 1.00000 1.00000
```
Confirm the device class that Ceph is aware of
```bash
[root@ukr-dpy1 ~]# ceph osd crush class ls
[
"hdd",
"ssd"
]
```
Lets confirm the crush rule(s) already in place
```bash
[root@ukr-dpy1 ~]# ceph osd crush rule list
replicated_rule
[root@ukr-dpy1 ~]# ceph osd crush rule dump replicated_rule
{
"rule_id": 0,
"rule_name": "replicated_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
```
Lets try out crushtool to see the crush map from a running cluster and convert it into ascii text
```bash
[root@ukr-dpy1 ~]# ceph osd getcrushmap -o /tmp/crush.bin
201
[root@ukr-dpy1 ~]# crushtool -d /tmp/crush.bin -o /tmp/crush.ascii
[root@ukr-dpy1 ~]# cat /tmp/crush.ascii
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54
# devices
device 0 osd.0 class hdd
device 1 osd.1 class hdd
device 2 osd.2 class hdd
device 3 osd.3 class hdd
device 4 osd.4 class hdd
device 5 osd.5 class hdd
device 6 osd.6 class hdd
device 7 osd.7 class hdd
device 8 osd.8 class hdd
device 9 osd.9 class hdd
device 10 osd.10 class hdd
device 11 osd.11 class hdd
device 12 osd.12 class hdd
device 13 osd.13 class hdd
device 14 osd.14 class hdd
device 15 osd.15 class hdd
device 16 osd.16 class hdd
device 17 osd.17 class hdd
device 18 osd.18 class hdd
device 19 osd.19 class hdd
device 20 osd.20 class hdd
device 21 osd.21 class hdd
device 22 osd.22 class hdd
device 23 osd.23 class hdd
device 24 osd.24 class ssd
device 25 osd.25 class ssd
device 26 osd.26 class ssd
device 27 osd.27 class ssd
device 28 osd.28 class ssd
device 29 osd.29 class ssd
device 30 osd.30 class ssd
device 31 osd.31 class ssd
device 32 osd.32 class ssd
device 33 osd.33 class ssd
device 34 osd.34 class ssd
device 35 osd.35 class ssd
device 36 osd.36 class ssd
device 37 osd.37 class ssd
device 38 osd.38 class ssd
device 39 osd.39 class ssd
device 40 osd.40 class ssd
device 41 osd.41 class ssd
device 42 osd.42 class ssd
device 43 osd.43 class ssd
device 44 osd.44 class ssd
device 45 osd.45 class ssd
device 46 osd.46 class ssd
device 47 osd.47 class ssd
device 48 osd.48 class ssd
device 49 osd.49 class ssd
device 50 osd.50 class ssd
device 51 osd.51 class ssd
device 52 osd.52 class ssd
device 53 osd.53 class ssd
device 54 osd.54 class ssd
device 55 osd.55 class ssd
device 56 osd.56 class ssd
device 57 osd.57 class ssd
device 58 osd.58 class hdd
device 59 osd.59 class hdd
device 60 osd.60 class hdd
device 61 osd.61 class hdd
device 62 osd.62 class hdd
device 63 osd.63 class hdd
device 64 osd.64 class hdd
device 65 osd.65 class hdd
device 66 osd.66 class hdd
device 67 osd.67 class hdd
device 68 osd.68 class hdd
device 69 osd.69 class hdd
device 70 osd.70 class hdd
device 71 osd.71 class hdd
device 72 osd.72 class hdd
device 73 osd.73 class hdd
device 74 osd.74 class hdd
device 75 osd.75 class hdd
device 76 osd.76 class hdd
device 77 osd.77 class hdd
device 78 osd.78 class hdd
device 79 osd.79 class hdd
device 80 osd.80 class hdd
device 81 osd.81 class hdd
device 82 osd.82 class hdd
device 83 osd.83 class hdd
device 84 osd.84 class hdd
device 85 osd.85 class hdd
device 86 osd.86 class hdd
device 87 osd.87 class hdd
device 88 osd.88 class hdd
device 89 osd.89 class hdd
device 90 osd.90 class hdd
device 91 osd.91 class hdd
device 92 osd.92 class hdd
device 93 osd.93 class hdd
device 94 osd.94 class hdd
device 95 osd.95 class hdd
device 96 osd.96 class hdd
device 97 osd.97 class hdd
device 98 osd.98 class hdd
device 99 osd.99 class hdd
device 100 osd.100 class hdd
device 101 osd.101 class hdd
device 102 osd.102 class hdd
device 103 osd.103 class hdd
device 104 osd.104 class hdd
device 105 osd.105 class hdd
device 106 osd.106 class hdd
device 107 osd.107 class hdd
device 108 osd.108 class hdd
device 109 osd.109 class hdd
device 110 osd.110 class hdd
device 111 osd.111 class hdd
device 112 osd.112 class hdd
device 113 osd.113 class hdd
device 114 osd.114 class hdd
device 115 osd.115 class hdd
device 116 osd.116 class hdd
device 117 osd.117 class hdd
device 118 osd.118 class hdd
device 119 osd.119 class hdd
# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 zone
type 10 region
type 11 root
# buckets
host ukr-hci2-n3-mlnx {
id -3 # do not change unnecessarily
id -4 class hdd # do not change unnecessarily
id -19 class ssd # do not change unnecessarily
# weight 43.658
alg straw2
hash 0 # rjenkins1
item osd.3 weight 14.553
item osd.8 weight 14.553
item osd.18 weight 14.553
}
host ukr-hci2-n4-mlnx {
id -5 # do not change unnecessarily
id -6 class hdd # do not change unnecessarily
id -20 class ssd # do not change unnecessarily
# weight 43.658
alg straw2
hash 0 # rjenkins1
item osd.4 weight 14.553
item osd.9 weight 14.553
item osd.20 weight 14.553
}
host ukr-hci1-n1-mlnx {
id -7 # do not change unnecessarily
id -8 class hdd # do not change unnecessarily
id -21 class ssd # do not change unnecessarily
# weight 43.658
alg straw2
hash 0 # rjenkins1
item osd.7 weight 14.553
item osd.15 weight 14.553
item osd.23 weight 14.553
}
host ukr-hci1-n2-mlnx {
id -9 # do not change unnecessarily
id -10 class hdd # do not change unnecessarily
id -22 class ssd # do not change unnecessarily
# weight 43.658
alg straw2
hash 0 # rjenkins1
item osd.6 weight 14.553
item osd.14 weight 14.553
item osd.22 weight 14.553
}
host ukr-hci1-n3-mlnx {
id -11 # do not change unnecessarily
id -12 class hdd # do not change unnecessarily
id -23 class ssd # do not change unnecessarily
# weight 43.658
alg straw2
hash 0 # rjenkins1
item osd.5 weight 14.553
item osd.11 weight 14.553
item osd.21 weight 14.553
}
host ukr-hci2-n1-mlnx {
id -13 # do not change unnecessarily
id -14 class hdd # do not change unnecessarily
id -24 class ssd # do not change unnecessarily
# weight 43.658
alg straw2
hash 0 # rjenkins1
item osd.0 weight 14.553
item osd.12 weight 14.553
item osd.17 weight 14.553
}
host ukr-hci2-n2-mlnx {
id -15 # do not change unnecessarily
id -16 class hdd # do not change unnecessarily
id -25 class ssd # do not change unnecessarily
# weight 43.658
alg straw2
hash 0 # rjenkins1
item osd.1 weight 14.553
item osd.10 weight 14.553
item osd.16 weight 14.553
}
host ukr-hci1-n4-mlnx {
id -17 # do not change unnecessarily
id -18 class hdd # do not change unnecessarily
id -26 class ssd # do not change unnecessarily
# weight 43.658
alg straw2
hash 0 # rjenkins1
item osd.2 weight 14.553
item osd.13 weight 14.553
item osd.19 weight 14.553
}
host ukr-ssd1-mlnx {
id -28 # do not change unnecessarily
id -29 class hdd # do not change unnecessarily
id -30 class ssd # do not change unnecessarily
# weight 237.534
alg straw2
hash 0 # rjenkins1
item osd.24 weight 6.986
item osd.25 weight 6.986
item osd.26 weight 6.986
item osd.27 weight 6.986
item osd.28 weight 6.986
item osd.29 weight 6.986
item osd.30 weight 6.986
item osd.31 weight 6.986
item osd.32 weight 6.986
item osd.33 weight 6.986
item osd.34 weight 6.986
item osd.35 weight 6.986
item osd.36 weight 6.986
item osd.37 weight 6.986
item osd.38 weight 6.986
item osd.39 weight 6.986
item osd.40 weight 6.986
item osd.41 weight 6.986
item osd.42 weight 6.986
item osd.43 weight 6.986
item osd.44 weight 6.986
item osd.45 weight 6.986
item osd.46 weight 6.986
item osd.47 weight 6.986
item osd.48 weight 6.986
item osd.49 weight 6.986
item osd.50 weight 6.986
item osd.51 weight 6.986
item osd.52 weight 6.986
item osd.53 weight 6.986
item osd.54 weight 6.986
item osd.55 weight 6.986
item osd.56 weight 6.986
item osd.57 weight 6.986
}
host ukr-hdd1 {
id -31 # do not change unnecessarily
id -32 class hdd # do not change unnecessarily
id -33 class ssd # do not change unnecessarily
# weight 902.267
alg straw2
hash 0 # rjenkins1
item osd.58 weight 14.553
item osd.59 weight 14.553
item osd.60 weight 14.553
item osd.61 weight 14.553
item osd.62 weight 14.553
item osd.63 weight 14.553
item osd.64 weight 14.553
item osd.65 weight 14.553
item osd.66 weight 14.553
item osd.67 weight 14.553
item osd.68 weight 14.553
item osd.69 weight 14.553
item osd.70 weight 14.553
item osd.71 weight 14.553
item osd.72 weight 14.553
item osd.73 weight 14.553
item osd.74 weight 14.553
item osd.75 weight 14.553
item osd.76 weight 14.553
item osd.77 weight 14.553
item osd.78 weight 14.553
item osd.79 weight 14.553
item osd.80 weight 14.553
item osd.81 weight 14.553
item osd.82 weight 14.553
item osd.83 weight 14.553
item osd.84 weight 14.553
item osd.85 weight 14.553
item osd.86 weight 14.553
item osd.87 weight 14.553
item osd.88 weight 14.553
item osd.89 weight 14.553
item osd.90 weight 14.553
item osd.91 weight 14.553
item osd.92 weight 14.553
item osd.93 weight 14.553
item osd.94 weight 14.553
item osd.95 weight 14.553
item osd.96 weight 14.553
item osd.97 weight 14.553
item osd.98 weight 14.553
item osd.99 weight 14.553
item osd.100 weight 14.553
item osd.101 weight 14.553
item osd.102 weight 14.553
item osd.103 weight 14.553
item osd.104 weight 14.553
item osd.105 weight 14.553
item osd.106 weight 14.553
item osd.107 weight 14.553
item osd.108 weight 14.553
item osd.109 weight 14.553
item osd.110 weight 14.553
item osd.111 weight 14.553
item osd.112 weight 14.553
item osd.113 weight 14.553
item osd.114 weight 14.553
item osd.115 weight 14.553
item osd.116 weight 14.553
item osd.117 weight 14.553
item osd.118 weight 14.553
item osd.119 weight 14.553
}
root default {
id -1 # do not change unnecessarily
id -2 class hdd # do not change unnecessarily
id -27 class ssd # do not change unnecessarily
# weight 1489.065
alg straw2
hash 0 # rjenkins1
item ukr-hci2-n3-mlnx weight 43.658
item ukr-hci2-n4-mlnx weight 43.658
item ukr-hci1-n1-mlnx weight 43.658
item ukr-hci1-n2-mlnx weight 43.658
item ukr-hci1-n3-mlnx weight 43.658
item ukr-hci2-n1-mlnx weight 43.658
item ukr-hci2-n2-mlnx weight 43.658
item ukr-hci1-n4-mlnx weight 43.658
item ukr-ssd1-mlnx weight 237.534
item ukr-hdd1 weight 902.267
}
# rules
rule replicated_rule {
id 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
# end crush map
[root@ukr-dpy1 ~]#
```
Lets investigate what needs to be run to create a new crush replicated rule.
```bash
# The command expected the following arguments
ceph osd crush rule create-replicated <rulesetname> default <failure-domain> <class>
# note: failure domains can be, from crush map
# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root
# class is as above, hdd/ssd
```
Lets create the SSD crush rule
```bash
[root@ukr-dpy1 ~]# ceph osd crush rule create-replicated highspeedpool default osd ssd
# On a 3 node environment use "host" as the failure domain, however with a single host ssd configuration we'd need to set replication factor to 1 for ceph to be healthy
[root@ukr-dpy1 ~]# ceph osd crush rule create-replicated highspeedpool default host ssd
# if you make a mistake use: ceph osd crush rule rm
```
Confirm the rule has been created and lets take a look at the dump
```bash
[root@ukr-dpy1 ~]# ceph osd crush rule list
replicated_rule
highspeedpool
[root@ukr-dpy1 ~]# ceph osd crush rule dump highspeedpool
{
"rule_id": 1,
"rule_name": "highspeedpool",
"ruleset": 1,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -27,
"item_name": "default~ssd"
},
{
"op": "choose_firstn",
"num": 0,
"type": "osd"
},
{
"op": "emit"
}
]
}
```
Create pools with the new ruleset
```bash
[root@ukr-dpy1 ~]# ceph osd pool create ssdpool 64 64 highspeedpool
pool 'ssdpool' created
```
Check the details of the pool (in this instance we can see that ssdpool is pool 11)
```bash
[root@ukr-dpy1 ~]# ceph osd pool ls detail | grep ssd
pool 12 'ssdpool' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 2026 lfor 0/2026/2024 flags hashpspool stripe_width 0
# if i hadnt set the failure domain to osd, it would have been host. With host failure and a single node, 3x replication would not have worked and we would have had to set something like below.
[root@ukr-dpy1 ~]# ceph osd pool ls detail | grep ssd
pool 11 'ssdpool' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 256 pgp_num 256 pg_num_target 32 pgp_num_target 32 autoscale_mode on last_change 1304 flags hashpspool stripe_width 0
# in this scenario i also had to set pool replication size to 1 as i only have a single host
[root@ukr-dpy1 ~]# ceph osd pool set ssdpool size 1
set pool 11 size to 1
```
Check ceph status for cluster to become healthy
```bash
[root@ukr-dpy1 ~]# ceph status
cluster:
id: f7d119bf-94c5-4f9b-bf0e-f6cc0e1d4df8
health: HEALTH_WARN
1 pools have many more objects per pg than average
services:
mon: 3 daemons, quorum ukr-hci1-n1-mlnx,ukr-hci1-n2-mlnx,ukr-hci1-n3-mlnx (age 3d)
mgr: ukr-hci1-n1-mlnx(active, since 4d), standbys: ukr-hci1-n4-mlnx, ukr-hci1-n3-mlnx, ukr-hci1-n2-mlnx
mds: cephfs:3 {0=ukr-hci1-n2-mlnx=up:active,1=ukr-hci1-n4-mlnx=up:active,2=ukr-hci1-n1-mlnx=up:active} 1 up:standby
osd: 120 osds: 120 up (since 3d), 120 in (since 3d)
rgw: 3 daemons active (ukr-hci1-n1-mlnx.rgw0, ukr-hci1-n2-mlnx.rgw0, ukr-hci1-n3-mlnx.rgw0)
task status:
data:
pools: 11 pools, 329 pgs
objects: 20.91k objects, 25 GiB
usage: 145 GiB used, 1.5 PiB / 1.5 PiB avail
pgs: 329 active+clean
```
Ok so lets reduce the pg_num down to 32 / pgp_num
```bash
[root@ukr-dpy1 ~]# ceph osd pool set ssdpool pg_num 32
set pool 12 pg_num to 32
[root@ukr-dpy1 ~]# ceph osd pool set ssdpool pgp_num 32
set pool 12 pgp_num to 32
[root@ukr-dpy1 ~]# ceph status
cluster:
id: f7d119bf-94c5-4f9b-bf0e-f6cc0e1d4df8
health: HEALTH_OK
services:
mon: 3 daemons, quorum ukr-hci1-n1-mlnx,ukr-hci1-n2-mlnx,ukr-hci1-n3-mlnx (age 3d)
mgr: ukr-hci1-n1-mlnx(active, since 4d), standbys: ukr-hci1-n4-mlnx, ukr-hci1-n3-mlnx, ukr-hci1-n2-mlnx
mds: cephfs:3 {0=ukr-hci1-n2-mlnx=up:active,1=ukr-hci1-n4-mlnx=up:active,2=ukr-hci1-n1-mlnx=up:active} 1 up:standby
osd: 120 osds: 120 up (since 3d), 120 in (since 3d)
rgw: 3 daemons active (ukr-hci1-n1-mlnx.rgw0, ukr-hci1-n2-mlnx.rgw0, ukr-hci1-n3-mlnx.rgw0)
task status:
data:
pools: 11 pools, 326 pgs
objects: 20.91k objects, 25 GiB
usage: 145 GiB used, 1.5 PiB / 1.5 PiB avail
pgs: 326 active+clean
[root@ukr-dpy1 ~]#
```
Allow the cinder user to access to ssdpool
```bash
[root@ukr-dpy1 ~]# ceph auth caps client.cinder osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images, profile rbd pool=ssdpool' mon 'profile rbd'
updated caps for client.cinder
```
Setup the cinder.conf with the additional ssdpool details
```bash
[root@ukr-dpy1 kolla]# cat config/cinder/cinder-volume.conf
[DEFAULT]
enabled_backends=rbd-1,rbd-ssd
[rbd-1]
rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=cinder
backend_host=rbd:volumes
rbd_pool=volumes
volume_backend_name=rbd-1
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_secret_uuid = fa5bb534-cea6-4219-8214-7f5d711e23c2
[rbd-ssd]
rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=cinder
backend_host=rbd:ssdpool
rbd_pool=ssdpool
volume_backend_name=rbd-ssd
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_secret_uuid = fa5bb534-cea6-4219-8214-7f5d711e23c2
```
Reconfigure OpenStack
```bash
root@kolla-deploy:/kolla# kolla-ansible -i /etc/kolla/multinode-train reconfigure
```
Health warning in ceph
```bash
[root@ukr-dpy1 ~]# ceph status
cluster:
id: f7d119bf-94c5-4f9b-bf0e-f6cc0e1d4df8
health: HEALTH_WARN
1 pool(s) do not have an application enabled
<snip>
[root@ukr-dpy1 ~]# ceph osd pool application enable ssdpool rbd
enabled application 'rbd' on pool 'ssdpool'
# ceph health should be good again
```
Setup the new ssdpool as a volume type in openstack, create a volume and attach to a server
```bash
openstack volume type list
openstack volume type create --property volume_backend_name=ssdpool ssdpool
openstack volume create --type ssdpool --size 10 ssdvol1
openstack server add volume demo1 ssdvol1
```
Benchmark the pools to confirm SSDs are being used when writing to the ssdpool
```bash
e]# logout
[root@ukr-dpy1 ~]# rados bench -p ssdpool 50 write
# then on the ssd1 node
# yum install sysstat
# on ssd node
iostat 1
```
Lets edit the original crushmap/rule to only all IO on HDDs
```bash
ceph osd crush rule create-replicated highcapacitypool default host hdd
# set the pools to use this hdd pool
ceph osd pool set device_health_metrics crush_rule highcapacitypool
ceph osd pool set images crush_rule highcapacitypool
ceph osd pool set volumes crush_rule highcapacitypool
ceph osd pool set backups crush_rule highcapacitypool
ceph osd pool set manilla_data crush_rule highcapacitypool
ceph osd pool set manila_data crush_rule highcapacitypool
ceph osd pool set manila_metadata crush_rule highspeedpool
ceph osd pool set .rgw.root crush_rule highcapacitypool
ceph osd pool set default.rgw.log crush_rule highcapacitypool
ceph osd pool set default.rgw.control crush_rule highcapacitypool
ceph osd pool set default.rgw.meta crush_rule highcapacitypool
[root@ukr-dpy1 kolla]# ceph status
cluster:
id: f7d119bf-94c5-4f9b-bf0e-f6cc0e1d4df8
health: HEALTH_WARN
Degraded data redundancy: 30493/66636 objects degraded (45.761%), 28 pgs degraded, 25 pgs undersized
services:
mon: 3 daemons, quorum ukr-hci1-n1-mlnx,ukr-hci1-n2-mlnx,ukr-hci1-n3-mlnx (age 3d)
mgr: ukr-hci1-n1-mlnx(active, since 4d), standbys: ukr-hci1-n4-mlnx, ukr-hci1-n3-mlnx, ukr-hci1-n2-mlnx
mds: cephfs:3 {0=ukr-hci1-n2-mlnx=up:active,1=ukr-hci1-n4-mlnx=up:active,2=ukr-hci1-n1-mlnx=up:active} 1 up:standby
osd: 120 osds: 120 up (since 4d), 120 in (since 4d); 57 remapped pgs
rgw: 3 daemons active (ukr-hci1-n1-mlnx.rgw0, ukr-hci1-n2-mlnx.rgw0, ukr-hci1-n3-mlnx.rgw0)
task status:
data:
pools: 11 pools, 297 pgs
objects: 22.21k objects, 30 GiB
usage: 166 GiB used, 1.5 PiB / 1.5 PiB avail
pgs: 30493/66636 objects degraded (45.761%)
11822/66636 objects misplaced (17.741%)
225 active+clean
31 active+remapped+backfill_wait
20 active+recovery_wait+undersized+degraded+remapped
8 active+recovery_wait+degraded
7 active+recovery_wait
5 active+recovering+undersized+remapped
1 active+recovery_wait+undersized+remapped
io:
client: 8.2 KiB/s rd, 9 op/s rd, 0 op/s wr
recovery: 8.4 MiB/s, 33 objects/s
```