Ceph运维常用命令

集群管理

启动和停止各组件

查看当前节点的systemd unit服务：

systemctl list-units 'ceph*' --type=service
UNIT                           LOAD   ACTIVE SUB     DESCRIPTION
ceph-mgr@node1.service         loaded active running Ceph cluster manager daemon
ceph-mon@node1.service         loaded active running Ceph cluster monitor daemon
ceph-osd@0.service             loaded active running Ceph object storage daemon osd.0
ceph-radosgw@rgw.node1.service loaded active running Ceph rados gateway

查看某个服务是否开机启动：

systemctl is-enabled ceph-mon@node1.service
enabled

确认了相关的systemd unit后就可以使用systemd在Ceph集群各个节点上启动或停止各个Ceph组件。

查看集群状态

查看集群的概要状态：

ceph health
HEALTH_OK


ceph -s
    cluster 83d9e421-46bf-4d64-af15-af0e2c381b88
     health HEALTH_WARN
            clock skew detected on mon.node2
            Monitor clock skew detected
     monmap e2: 3 mons at {node1=192.168.61.41:6789/0,node2=192.168.61.42:6789/0,node3=192.168.61.43:6789/0}
            election epoch 114, quorum 0,1,2 node1,node2,node3
        mgr active: node2 standbys: node1, node3
     osdmap e138: 3 osds: 3 up, 3 in
            flags sortbitwise,require_jewel_osds,require_kraken_osds
      pgmap v9909: 152 pgs, 12 pools, 2686 bytes data, 214 objects
            133 MB used, 284 GB / 284 GB avail
                 152 active+clean

查看集群空间使用

ceph df
GLOBAL:
    SIZE     AVAIL     RAW USED     %RAW USED
    284G      284G         133M          0.05
POOLS:
    NAME                          ID     USED     %USED     MAX AVAIL     OBJECTS
    rbd                           0         0         0        97185M           0
    .rgw.root                     1      1681         0        97185M           4
    default.rgw.control           2         0         0        97185M           8
    default.rgw.data.root         3       603         0        97185M           2
    default.rgw.gc                4         0         0        97185M          32
    default.rgw.lc                5         0         0        97185M          32
    default.rgw.log               6         0         0        97185M         128
    default.rgw.users.uid         7       350         0        97185M           2
    default.rgw.users.email       8         8         0        97185M           1
    default.rgw.users.keys        9         8         0        97185M           1
    default.rgw.buckets.index     10        0         0        97185M           1
    default.rgw.buckets.data      11       36         0        97185M           3

MON节点命令

查看MON节点状态：

ceph quorum_status

{
  "election_epoch": 114,
  "quorum": [
    0,
    1,
    2
  ],
  "quorum_names": [
    "node1",
    "node2",
    "node3"
  ],
  "quorum_leader_name": "node1",
  "monmap": {
    "epoch": 2,
    "fsid": "83d9e421-46bf-4d64-af15-af0e2c381b88",
    "modified": "2017-04-06 19:52:00.882973",
    "created": "2017-04-06 19:51:47.569731",
    "features": {
      "persistent": [
        "kraken"
      ],
      "optional": []
    },
    "mons": [
      {
        "rank": 0,
        "name": "node1",
        "addr": "192.168.61.41:6789/0",
        "public_addr": "192.168.61.41:6789/0"
      },
      {
        "rank": 1,
        "name": "node2",
        "addr": "192.168.61.42:6789/0",
        "public_addr": "192.168.61.42:6789/0"
      },
      {
        "rank": 2,
        "name": "node3",
        "addr": "192.168.61.43:6789/0",
        "public_addr": "192.168.61.43:6789/0"
      }
    ]
  }
}

MON使用Paxos算法进行选举，上面的输出中 election_epoch 表示一共进行的投票轮次数量， quorum 和 quorum_names 表示参与投票者的编号和名称， quorum_leader_name 表示当前的leader名称，rank表示每个MON节点的权重，权重越小在选举时月容易得到支持。

OSD节点命令

查看OSD节点状态

ceph osd stat
     osdmap e138: 3 osds: 3 up, 3 in
            flags sortbitwise,require_jewel_osds,require_kraken_osds

使用 ceph osd dump 可查看osd节点的详细信息。

查看OSD节点的分布

ceph osd tree
ID WEIGHT  TYPE NAME      UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.27809 root default
-2 0.09270     host node1
 0 0.09270         osd.0       up  1.00000          1.00000
-3 0.09270     host node2
 1 0.09270         osd.1       up  1.00000          1.00000
-4 0.09270     host node3
 2 0.09270         osd.2       up  1.00000          1.00000

参考

• MONITORING A CLUSTER