内容简介：生成环境中，Ceph集群的部署是个非常专业的事情，需要懂Ceph的人来规划和部署，通常部署方式有两种：1、ceph-deploy2、ceph-ansible
生成环境中，Ceph集群的部署是个非常专业的事情，需要懂Ceph的人来规划和部署，通常部署方式有两种：
1、ceph-deploy
2、ceph-ansible
无论上面的哪一种方式，都需要比较多的专业知识，这明显不适合规模化的需求或者测试环境的需求。而Rook则基于Kubernetes，有效的解决了Ceph集群的部署难题。
Rook
Rook简介
项目地址： https://github.com/rook/rook
网站： https://rook.io/
Rook与Kubernetes结合的架构图如下：
从上面可以看出，Rook是基于 Kubernetes 之上，提供一键部署存储系统的编排系统。
英文描述：Cloud-Native Storage Orchestrator
当前支持的存储系统有：
- Ceph
- NFS
- Minio
- CockroachDB
其中只有Ceph系统的支持处于beta版本，其他的都还是alpha版本，后续主要介绍通过Rook来部署Ceph集群。
Rook组件
Rook的主要组件有两个，功能如下：
-
Rook Operator
-
Rook与Kubernetes交互的组件
-
整个Rook集群只有一个
-
-
Rook Agent
- 与Rook Operator交互，执行命令
- 每个Kubernetes的Node上都会启动一个
- 不同的存储系统，启动的Agent是不同的
Rook & Ceph框架
使用Rook部署Ceph集群的架构图如下：
从上面可以看出，通过Rook部署完Ceph集群后，就可以提供 Volume Claim 给Kubernetes集群里的App使用了。
部署的Ceph系统可以提供下面三种 Volume Claim 服务：
- Volume：不需要额外组件；
- FileSystem：部署MDS；
- Object：部署RGW；
当前也就Ceph RBD支持的比较完善
部署Rook系统
Github上下载Rook最新release 0.9.0版本： https://github.com/rook/rook/tree/v0.9.0
要通过Rook部署Ceph集群，首先要部署Rook的两个组件：
1、Rook operator
2、Rook agent
如上面的Rook架构所示，Rook operator会启动一个pod，而Rook agent则会在kubernetes所有的nodes上都启动一个。
部署Rook的组件比较简单，如下命令：
# cd rook-master/cluster/examples/kubernetes/ceph/ # kubectl create -f operator.yaml namespace "rook-ceph-system" created customresourcedefinition "clusters.ceph.rook.io" created customresourcedefinition "filesystems.ceph.rook.io" created customresourcedefinition "objectstores.ceph.rook.io" created customresourcedefinition "pools.ceph.rook.io" created customresourcedefinition "volumes.rook.io" created clusterrole "rook-ceph-cluster-mgmt" created role "rook-ceph-system" created clusterrole "rook-ceph-global" created serviceaccount "rook-ceph-system" created rolebinding "rook-ceph-system" created clusterrolebinding "rook-ceph-global" created deployment "rook-ceph-operator” created
如上所示，它会创建如下资源：
- namespace：rook-ceph-system，之后的所有rook相关的pod都会创建在该namespace下面
- CRD：创建五个CRDs，
.ceph.rook.io - role & clusterrole：用户资源控制
- serviceaccount：ServiceAccount资源，给Rook创建的Pod使用
- deployment：rook-ceph-operator，部署rook ceph相关的组件
部署rook-ceph-operator过程中，会触发以 DaemonSet 的方式在集群部署 Agent 和 Discover pods。
所以部署成功后，可以看到如下pods：
# kubectl -n rook-ceph-systemm get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE rook-ceph-agent-bz4nk 1/1 Running 0 1d 172.20.4.27 ke-dev1-worker3 rook-ceph-agent-f8vcf 1/1 Running 0 1d 172.20.4.19 ke-dev1-master3 rook-ceph-agent-fhxq2 1/1 Running 0 1d 172.20.4.25 ke-dev1-worker1 rook-ceph-agent-nzhrp 1/1 Running 0 1d 172.20.6.175 ke-dev1-worker4 rook-ceph-operator-5dc97f5c79-vq7xs 1/1 Running 0 1d 192.168.32.174 ke-dev1-worker1 rook-discover-hcxhj 1/1 Running 0 1d 192.168.32.130 ke-dev1-worker1 rook-discover-j4q9m 1/1 Running 0 1d 192.168.217.172 ke-dev1-worker4 rook-discover-ldrzv 1/1 Running 0 1d 192.168.2.85 ke-dev1-worker3 rook-discover-wcwxx 1/1 Running 0 1d 192.168.53.255 ke-dev1-master3
上述部署在第一次会比较慢，这是因为所有节点都要拉取 rook/ceph:master 镜像，有500多MB大小。
可以通过手动拉取一份，然后在各个节点上load的方式：
# docker pull rook/ceph:master # docker save rook/ceph:master > rook-ceph-master.image # docker load -i rook-ceph-master.image
部署Ceph cluster
创建Ceph集群
当检查到Rook部署中的所有pods都为running状态后，就可以部署Ceph集群了，命令如下：
# cd rook-master/cluster/examples/kubernetes/ceph/ # kubectl create -f cluster.yaml namespace "rook-ceph" created serviceaccount "rook-ceph-cluster" created role "rook-ceph-cluster" created rolebinding "rook-ceph-cluster-mgmt" created rolebinding "rook-ceph-cluster" created cluster "rook-ceph” created
如上所示，它会创建如下资源：
- namespace：roo-ceph，之后的所有Ceph集群相关的pod都会创建在该namespace下
- serviceaccount：ServiceAccount资源，给Ceph集群的Pod使用
- role & rolebinding：用户资源控制
- cluster：rook-ceph，创建的Ceph集群
Ceph集群部署成功后，可以查看到的pods如下：
# kubectl -n rook-ceph get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE rook-ceph-mgr-a-959d64b9d-vwqsw 1/1 Running 0 2m 192.168.32.139 ke-dev1-worker1 rook-ceph-mon-a-6d9447cbd9-xgnjd 1/1 Running 0 15m 192.168.53.240 ke-dev1-master3 rook-ceph-mon-d-84648b4585-fvkbw 1/1 Running 0 13m 192.168.32.181 ke-dev1-worker1 rook-ceph-mon-f-55994d4b94-w6dds 1/1 Running 0 12m 192.168.2.79 ke-dev1-worker3 rook-ceph-osd-0-54c8ddbc5b-wzght 1/1 Running 0 1m 192.168.32.179 ke-dev1-worker1 rook-ceph-osd-1-844896d6c-q7rm8 1/1 Running 0 1m 192.168.2.119 ke-dev1-worker3 rook-ceph-osd-2-67d5d8f754-qfxx6 1/1 Running 0 1m 192.168.53.231 ke-dev1-master3 rook-ceph-osd-3-5bd9b98dd4-qhzk5 1/1 Running 0 1m 192.168.217.174 ke-dev1-worker4
可以看出部署的Ceph集群有：
- Ceph Monitors：默认启动三个ceph-mon，可以在cluster.yaml里配置
- Ceph Mgr：默认启动一个，可以在cluster.yaml里配置
- Ceph OSDs：根据cluster.yaml里的配置启动，默认在所有的可用节点上启动
上述Ceph组件对应kubernetes的kind是deployment：
# kubectl -n rook-ceph get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE rook-ceph-mgr-a 1 1 1 1 22h rook-ceph-mon-a 1 1 1 1 22h rook-ceph-mon-b 1 1 1 1 22h rook-ceph-mon-c 1 1 1 1 22h rook-ceph-osd-0 1 1 1 1 22h rook-ceph-osd-1 1 1 1 1 22h rook-ceph-osd-2 1 1 1 1 22h rook-ceph-tools 1 1 1 1 22h
Ceph集群配置
monitor pod的 ceph.conf ：
[root@rook-ceph-mon-a-6d9447cbd9-xgnjd ceph]# cat ceph.conf [global] fsid = 165c5334-9e0d-47e7-aa42-8c8006fb924f run dir = /var/lib/rook/mon-a mon initial members = f d a mon host = 10.96.111.234:6790,10.96.160.0:6790,10.96.237.241:6790 log file = /dev/stderr mon cluster log file = /dev/stderr public addr = 10.96.237.241 cluster addr = 192.168.53.240 mon keyvaluedb = rocksdb mon_allow_pool_delete = true mon_max_pg_per_osd = 1000 debug default = 0 debug rados = 0 debug mon = 0 debug osd = 0 debug bluestore = 0 debug filestore = 0 debug journal = 0 debug leveldb = 0 filestore_omap_backend = rocksdb osd pg bits = 11 osd pgp bits = 11 osd pool default size = 1 osd pool default min size = 1 osd pool default pg num = 100 osd pool default pgp num = 100 rbd_default_features = 3 fatal signal handlers = false [mon.a] keyring = /var/lib/rook/mon-a/keyring public bind addr = 192.168.53.240:6790
osd pod的 ceph.conf ：
[root@rook-ceph-osd-2-67d5d8f754-qfxx6 ceph]# cat ceph.conf [global] run dir = /var/lib/rook/osd2 mon initial members = f d a mon host = 10.96.111.234:6790,10.96.160.0:6790,10.96.237.241:6790 log file = /dev/stderr mon cluster log file = /dev/stderr public addr = 192.168.53.231 cluster addr = 192.168.53.231 mon keyvaluedb = rocksdb mon_allow_pool_delete = true mon_max_pg_per_osd = 1000 debug default = 0 debug rados = 0 debug mon = 0 debug osd = 0 debug bluestore = 0 debug filestore = 0 debug journal = 0 debug leveldb = 0 filestore_omap_backend = rocksdb osd pg bits = 11 osd pgp bits = 11 osd pool default size = 1 osd pool default min size = 1 osd pool default pg num = 100 osd pool default pgp num = 100 osd max object name len = 256 osd max object namespace len = 64 osd objectstore = filestore crush location = root=default host=ke-dev1-master3 rbd_default_features = 3 fatal signal handlers = false [osd.2] keyring = /var/lib/rook/osd2/keyring osd journal size = 1024
mgr pod的 ceph.conf ：
[root@rook-ceph-mgr-a-959d64b9d-xld9r /]# cat /etc/ceph/ceph.conf [global] run dir = /var/lib/rook/mgr-a mon initial members = b c a mon host = 10.96.125.224:6790,10.96.201.208:6790,10.96.59.167:6790 log file = /dev/stderr mon cluster log file = /dev/stderr public addr = 192.168.2.94 cluster addr = 192.168.2.94 mon keyvaluedb = rocksdb mon_allow_pool_delete = true mon_max_pg_per_osd = 1000 debug default = 0 debug rados = 0 debug mon = 0 debug osd = 0 debug bluestore = 0 debug filestore = 0 debug journal = 0 debug leveldb = 0 filestore_omap_backend = rocksdb osd pg bits = 11 osd pgp bits = 11 osd pool default size = 1 osd pool default min size = 1 osd pool default pg num = 100 osd pool default pgp num = 100 rbd_default_features = 3 fatal signal handlers = false [mgr.a] keyring = /var/lib/rook/mgr-a/keyring mgr data = /var/lib/rook/mgr-a
Ceph集群的配置生成是代码中指定的，代码： pkg/daemon/ceph/config/config.go
[ceph默认配置]
// CreateDefaultCephConfig creates a default ceph config file.
func CreateDefaultCephConfig(context *clusterd.Context, cluster *ClusterInfo, runDir string) *CephConfig {
// extract a list of just the monitor names, which will populate the "mon initial members"
// global config field
monMembers := make([]string, len(cluster.Monitors))
monHosts := make([]string, len(cluster.Monitors))
i := 0
for _, monitor := range cluster.Monitors {
monMembers[i] = monitor.Name
monHosts[i] = monitor.Endpoint
i++
}
cephLogLevel := logLevelToCephLogLevel(context.LogLevel)
return &CephConfig{
GlobalConfig: &GlobalConfig{
FSID: cluster.FSID,
RunDir: runDir,
MonMembers: strings.Join(monMembers, " "),
MonHost: strings.Join(monHosts, ","),
LogFile: "/dev/stderr",
MonClusterLogFile: "/dev/stderr",
PublicAddr: context.NetworkInfo.PublicAddr,
PublicNetwork: context.NetworkInfo.PublicNetwork,
ClusterAddr: context.NetworkInfo.ClusterAddr,
ClusterNetwork: context.NetworkInfo.ClusterNetwork,
MonKeyValueDb: "rocksdb",
MonAllowPoolDelete: true,
MaxPgsPerOsd: 1000,
DebugLogDefaultLevel: cephLogLevel,
DebugLogRadosLevel: cephLogLevel,
DebugLogMonLevel: cephLogLevel,
DebugLogOSDLevel: cephLogLevel,
DebugLogBluestoreLevel: cephLogLevel,
DebugLogFilestoreLevel: cephLogLevel,
DebugLogJournalLevel: cephLogLevel,
DebugLogLevelDBLevel: cephLogLevel,
FileStoreOmapBackend: "rocksdb",
OsdPgBits: 11,
OsdPgpBits: 11,
OsdPoolDefaultSize: 1,
OsdPoolDefaultMinSize: 1,
OsdPoolDefaultPgNum: 100,
OsdPoolDefaultPgpNum: 100,
RbdDefaultFeatures: 3,
FatalSignalHandlers: "false",
},
}
}
删除Ceph集群
删除已创建的Ceph集群，可执行下面命令：
# kubectl delete -f cluster.yaml namespace "rook-ceph" deleted serviceaccount "rook-ceph-cluster" deleted role "rook-ceph-cluster" deleted rolebinding "rook-ceph-cluster-mgmt" deleted rolebinding "rook-ceph-cluster" deleted cluster "rook-cephn” deleted
删除Ceph集群后，在之前部署Ceph组件节点的/var/lib/rook/目录，会遗留下Ceph集群的配置信息。
若之后再部署新的Ceph集群，先把之前Ceph集群的这些信息删除，不然启动monitor会失败；
# cat clean-rook-dir.sh
hosts=(
ke-dev1-master3
ke-dev1-worker1
ke-dev1-worker3
ke-dev1-worker4
)
for host in ${hosts[@]} ; do
ssh $host "rm -rf /var/lib/rook/*"
done
部署Ceph集群的log
在部署Ceph集群中，可能出错或者不符合预期，这时候就需要查看对应的log，rook是通过 rook-ceph-operator pod来执行部署Ceph集群的，可以查看它的log：
# kubectl -n rook-ceph-system get pods -o wide | grep operrator
rook-ceph-operator-5dc97f5c79-vq7xs 1/1 Running 0 18h 192.168.32.174 ke-dev1-worker1
# kubectl -n rook-ceph-system log rook-ceph-operator-5dc97f5c79-vq7xs | less
2018-11-27 06:52:13.795165 I | op-cluster: starting cluster in namespace rook-ceph
2018-11-27 06:52:13.818565 I | op-k8sutil: waiting for job rook-ceph-detect-version to complete...
2018-11-27 06:52:28.885610 I | op-cluster: detected ceph version mimic for image ceph/ceph:v13
2018-11-27 06:52:34.921566 I | op-mon: start running mons
...
2018-11-27 06:53:54.772352 I | op-mon: Ceph monitors formed quorum
2018-11-27 06:53:54.777563 I | op-cluster: creating initial crushmap
2018-11-27 06:53:54.777593 I | cephclient: setting crush tunables to firefly
...
2018-11-27 06:53:56.610181 I | op-cluster: created initial crushmap
2018-11-27 06:53:56.641097 I | op-mgr: start running mgr
...
2018-11-27 06:54:04.742093 I | op-mgr: mgr metrics service started
2018-11-27 06:54:04.742125 I | op-osd: start running osds in namespace rook-ceph
...
2018-11-27 06:54:05.438535 I | exec: noscrub is set
...
2018-11-27 06:54:06.502961 I | exec: nodeep-scrub is set
2018-11-27 06:54:06.520914 I | op-osd: 3 of the 3 storage nodes are valid
2018-11-27 06:54:06.520949 I | op-osd: checking if orchestration is still in progress
2018-11-27 06:54:06.529539 I | op-osd: start provisioning the osds on nodes, if needed
2018-11-27 06:54:06.722825 I | op-osd: avail devices for node ke-dev1-worker1: [{Name:vda FullPath: Config:map[]} {Name:vdb FullPath: Config:map[]} {Name:vdc FullPath: Config:map[]} {Name:vdd FullPath: Config:map[]} {Name:vde FullPath: Config:map[]}]
...
2018-11-27 06:55:31.377544 I | op-osd: completed running osds in namespace rook-ceph
...
2018-11-27 06:55:32.831393 I | exec: noscrub is unset
...
2018-11-27 06:55:33.932523 I | exec: nodeep-scrub is unset
2018-11-27 06:55:33.932698 I | op-cluster: Done creating rook instance in namespace rook-ceph
2018-11-27 06:55:33.957988 I | op-pool: start watching pool resources in namespace rook-ceph
...
2018-11-27 06:55:33.975970 I | op-cluster: added finalizer to cluster rook-ceph
Ceph组件的log
从上面章节中各个Ceph组件的 ceph.conf 文件中看出，配置的log输出为 /dev/stderr ，所以我们可以通过获取Ceph组件所在pod的log来查看Ceph组件的log：
# kubectl -n rook-ceph log rook-ceph-mon-a-7c47978fbb-hbbjv
Ceph toolbox部署
默认启动的Ceph集群，是开启Ceph认证的，这样你登陆Ceph组件所在的Pod里，是没法去获取集群状态，执行CLI命令的，这时需要部署Ceph toolbox，命令如下：
# kubectl create -f toolbox.yaml deployment "rook-ceph-tools" created
部署成功后，pod如下：
# kubectl -n rook-ceph get pods -o wide | grep ceph-tools rook-ceph-tools-79954fdf9d-n9qn5 1/1 Running 0 6m 172.20.4.27 ke-dev1-worker3
然后可以登陆该pod后，执行Ceph CLI命令：
# kubectl -n rook-ceph exec -it rook-ceph-tools-79954fdf9d-n9qn5 bash
...
[root@ke-dev1-worker3 /]# ceph status
cluster:
id: 165c5334-9e0d-47e7-aa42-8c8006fb924f
health: HEALTH_OK
services:
mon: 3 daemons, quorum f,d,a
mgr: a(active)
osd: 4 osds: 4 up, 4 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 57 GiB used, 21 GiB / 78 GiB avail
pgs:
[root@ke-dev1-worker1 /]# cd /etc/ceph/
[root@ke-dev1-worker1 ceph]# ls
ceph.conf keyring rbdmap
[root@ke-dev1-worker1 ceph]# cat ceph.conf
[global]
mon_host = 10.96.59.167:6790,10.96.125.224:6790,10.96.201.208:6790
[client.admin]
keyring = /etc/ceph/keyring
[root@ke-dev1-worker1 ceph]# cat keyring
[client.admin]
key = AQAL9PxbMXKpDBAAGO8g8GaD6vYC8iD9TaCd1Q==
[root@ke-dev1-worker1 ceph]# cat rbdmap
# RbdDevice Parameters
#poolname/imagename id=client,keyring=/etc/ceph/ceph.client.keyring
当不需要执行Ceph CLI命令时，可以通过kubectl命令很方便的删除：
# kubectl create -f toolbox.yaml deployment "rook-ceph-tools" deleted
RBD服务
在kubernetes集群里，要提供rbd块设备服务，需要有如下步骤：
- 创建rbd-provisioner pod
- 创建rbd对应的storageclass
- 创建pvc使用rbd对应的storageclass
- 创建pod使用rbd pvc
通过rook创建Ceph Cluster之后，rook自身提供了rbd-provisioner服务，所以我们不需要再部署其provisioner。
代码：pkg/operator/ceph/provisioner/provisioner.go
创建pool和StorageClass
# cat storageclass.yaml
apiVersion: ceph.rook.io/v1beta1
kind: Pool
metadata:
name: replicapool
namespace: rook-ceph
spec:
replicated:
size: 1
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-ceph-block
provisioner: ceph.rook.io/block
parameters:
pool: replicapool
# Specify the namespace of the rook cluster from which to create volumes.
# If not specified, it will use `rook` as the default namespace of the cluster.
# This is also the namespace where the cluster will be
clusterNamespace: rook-ceph
# Specify the filesystem type of the volume. If not specified, it will use `ext4`.
fstype: xfs
# kubectl create -f storageclass.yaml
pool "replicapool" created
storageclass "rook-ceph-block" created
# kubectl get sc
NAME PROVISIONER AGE
...
rook-ceph-block ceph.rook.io/block 1h
# kubectl get sc rook-ceph-block -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
creationTimestamp: 2018-11-27T08:22:16Z
name: rook-ceph-block
resourceVersion: "248878533"
selfLink: /apis/storage.k8s.io/v1/storageclasses/rook-ceph-block
uid: 8ce4471c-f21d-11e8-bb3e-fa163e65e579
parameters:
clusterNamespace: rook-ceph
fstype: xfs
pool: replicapool
provisioner: ceph.rook.io/block
reclaimPolicy: Delete
创建pvc
创建一个pvc，使用刚才配置的rbd storageclass。
# vim tst-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: myclaim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
storageClassName: rook-ceph-block
# kubectl create -f tst-pvc.yaml
persistentvolumeclaim "myclaim” created
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
myclaim Bound pvc-f85e52ac-f21f-11e8-bb3e-fa163e65e579 8Gi RWO rook-ceph-block 6s
在Ceph集群端检查：
[root@ke-dev1-worker1 ~]# rbd info -p replicapool pvc-f85e52ac-f21f-11e8-bb3e-fa163e65e579
rbd image 'pvc-f85e52ac-f21f-11e8-bb3e-fa163e65e579':
size 8 GiB in 2048 objects
order 22 (4 MiB objects)
id: 133e6b8b4567
block_name_prefix: rbd_data.133e6b8b4567
format: 2
features: layering
op_features:
flags:
create_timestamp: Tue Nov 27 08:39:37 2018
创建pod
创建一个pod，配置使用刚才创建的rbd pvc。
# vim tst-pvc-pod.yaml
kind: Pod
apiVersion: v1
metadata:
name: test-pod-rbd
namespace: rook-ceph
labels:
test-rbd: "true"
spec:
containers:
- name: test-pod
image: busybox:latest
command:
- "/bin/sh"
args:
- "-c"
- "trap exit TERM; while true; do sleep 1; done"
volumeMounts:
- name: data
mountPath: "/mnt"
volumes:
- name: data
persistentVolumeClaim:
claimName: myclaim
# kubectl create -f tst-pvc-pod.yaml
pod "test-pod-rbd” created
登陆pod检查rbd设备：
# kubectl get pods -o wide ... test-pod-rbd 1/1 Running 0 18s 192.168.32.191 ke-dev1-worker1 # kubectl exec -it test-pod-rbd sh / # mount | grep rbd /dev/rbd0 on /mnt type xfs (rw,relatime,attr2,inode64,sunit=8192,swidth=8192,noquota)
CephFS服务
与RBD服务类似，要想在kubernetes里使用CephFS，也需要一个cephfs-provisioner服务，在Rook代码里默认还不支持这个，所以需要单独部署。
参考： https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/cephfs
创建CephFS
可以通过下面的命令快速创建一个CephFS，根据自己的需要配置 filesystem.yaml ：
# cat filesystem.yaml
apiVersion: ceph.rook.io/v1beta1
kind: Filesystem
metadata:
name: cephfs
namespace: rook-ceph
spec:
# The metadata pool spec
metadataPool:
replicated:
# Increase the replication size if you have more than one osd
size: 2
# The list of data pool specs
dataPools:
- failureDomain: osd
replicated:
size: 2
# The metadata service (mds) configuration
metadataServer:
# The number of active MDS instances
activeCount: 2
# Whether each active MDS instance will have an active standby with a warm metadata cache for faster failover.
# If false, standbys will be available, but will not have a warm cache.
activeStandby: true
# The affinity rules to apply to the mds deployment
placement:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: role
# operator: In
# values:
# - mds-node
# tolerations:
# - key: mds-node
# operator: Exists
# podAffinity:
# podAntiAffinity:
resources:
# The requests and limits set here, allow the filesystem MDS Pod(s) to use half of one CPU core and 1 gigabyte of memory
# limits:
# cpu: "500m"
# memory: "1024Mi"
# requests:
# cpu: "500m"
# memory: "1024Mi"
执行命令：
# kubectl create -f filesystem.yaml filesystem "myfs” created # kubectl -n rook-ceph get pods -o wide | grep mds rook-ceph-mds-cephfs-a-57db46bfc4-2j9tq 1/1 Running 0 21h 192.168.2.65 ke-dev1-worker3 rook-ceph-mds-cephfs-b-6887d9649b-kwkxg 1/1 Running 0 21h 192.168.32.137 ke-dev1-worker1 rook-ceph-mds-cephfs-c-8bb84fcf4-lgfjl 1/1 Running 0 21h 192.168.2.84 ke-dev1-worker3 rook-ceph-mds-cephfs-d-78cf67cfcb-p9q9k 1/1 Running 0 21h 192.168.32.175 ke-dev1-worker1
我们配置 activeCount : 2，activeStandby: true，所以这里创建了4个MDS Daemons
mds pod的ceph.conf：
[root@rook-ceph-mds-cephfs-a-57db46bfc4-2j9tq /]# cat /etc/ceph/ceph.conf [global] run dir = /var/lib/rook/mds-cephfs-a mon initial members = a b c mon host = 10.96.59.167:6790,10.96.125.224:6790,10.96.201.208:6790 log file = /dev/stderr mon cluster log file = /dev/stderr public addr = 192.168.2.65 cluster addr = 192.168.2.65 mon keyvaluedb = rocksdb mon_allow_pool_delete = true mon_max_pg_per_osd = 1000 debug default = 0 debug rados = 0 debug mon = 0 debug osd = 0 debug bluestore = 0 debug filestore = 0 debug journal = 0 debug leveldb = 0 filestore_omap_backend = rocksdb osd pg bits = 11 osd pgp bits = 11 osd pool default size = 1 osd pool default min size = 1 osd pool default pg num = 100 osd pool default pgp num = 100 rbd_default_features = 3 fatal signal handlers = false [mds.cephfs-a] keyring = /var/lib/rook/mds-cephfs-a/keyring mds_standby_for_fscid = 1 mds_standby_replay = true
检查CephFS状态：
[root@ke-dev1-worker1 /]# ceph fs ls
name: cephfs, metadata pool: cephfs-metadata, data pools: [cephfs-data0 ]
[root@ke-dev1-worker1 /]# ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
298 GiB 235 GiB 63 GiB 21.13
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
...
cephfs-metadata 2 62 KiB 0 110 GiB 41
cephfs-data0 3 4 B 0 110 GiB 2
[root@ke-dev1-worker1 /]# ceph fs status
cephfs - 2 clients
======
+------+----------------+----------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+----------------+----------+---------------+-------+-------+
| 0 | active | cephfs-d | Reqs: 0 /s | 18 | 21 |
| 1 | active | cephfs-a | Reqs: 0 /s | 11 | 14 |
| 0-s | standby-replay | cephfs-b | Evts: 0 /s | 0 | 0 |
| 1-s | standby-replay | cephfs-c | Evts: 0 /s | 0 | 0 |
+------+----------------+----------+---------------+-------+-------+
+-----------------+----------+-------+-------+
| Pool | type | used | avail |
+-----------------+----------+-------+-------+
| cephfs-metadata | metadata | 61.6k | 110G |
| cephfs-data0 | data | 4 | 110G |
+-----------------+----------+-------+-------+
+-------------+
| Standby MDS |
+-------------+
+-------------+
创建StorageClass
获取Ceph client.admin的keyring：
# ceph auth get-key client.admin AQAL9PxbMXKpDBAAGO8g8GaD6vYC8iD9TaCd1Q==
创建kubernetes的secret：
# cat secret AQAL9PxbMXKpDBAAGO8g8GaD6vYC8iD9TaCd1Q== # kubectl create secret generic rook-ceph-secret-admin --from-file=secret --namespace=kube-system secret "rook-ceph-secret-admin" created # kubectl -n kube-system get secret | grep rook rook-ceph-secret-admin Opaque 1 18s # kubectl -n kube-system get secret rook-ceph-secret-admin -o yaml apiVersion: v1 data: secret: QVFBTDlQeGJNWEtwREJBQUdPOGc4R2FENnZZQzhpRDlUYUNkMVE9PQo= kind: Secret metadata: creationTimestamp: 2018-11-27T10:41:57Z name: rook-ceph-secret-admin namespace: kube-system resourceVersion: "248907628" selfLink: /api/v1/namespaces/kube-system/secrets/rook-ceph-secret-admin uid: 0ffcf324-f231-11e8-bb3e-fa163e65e579 type: Opaque
基于上面创建的adminsecret，创建对应的Storageclass；
# cat storageclass.yaml kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: rook-ceph-fs provisioner: ceph.com/cephfs parameters: monitors: 10.96.59.167:6790,10.96.125.224:6790,10.96.201.208:6790 adminId: admin adminSecretName: rook-ceph-secret-admin adminSecretNamespace: kube-system # kubectl create -f storageclass.yaml storageclass "rook-ceph-fs” created # kubectl get sc NAME PROVISIONER AGE ... rook-ceph-block ceph.rook.io/block 2h rook-ceph-fs ceph.com/cephfs 4s
创建pvc
测试pvc的yaml如下：
# cat tst-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mycephfsclaim
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 8Gi
storageClassName: rook-ceph-fs
创建测试的pvc，并检查其状态：
# kubectl create -f tst-pvc.yaml persistentvolumeclaim "mycephfsclaim" created # kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ... mycephfsclaim Bound pvc-1ded8c22-f232-11e8-bb3e-fa163e65e579 8Gi RWX rook-ceph-fs 4s
创建pod
创建测试pod的yaml如下：
# cat tst-pvc-pod.yaml
kind: Pod
apiVersion: v1
metadata:
name: test-pod-cephfs
labels:
test-rbd: "true"
spec:
containers:
- name: test-pod
image: busybox:latest
command:
- "/bin/sh"
args:
- "-c"
- "trap exit TERM; while true; do sleep 1; done"
volumeMounts:
- name: data
mountPath: "/mnt"
volumes:
- name: data
persistentVolumeClaim:
claimName: mycephfsclaim
创建测试pod挂载使用上一步创建的pvc：
# kubectl create -f tst-pvc-pod.yaml
pod "test-pod-cephfs” created
# kubectl get pod
NAME READY STATUS RESTARTS AGE
...
test-pod-cephfs 1/1 Running 0 10s
# kubectl exec -it test-pod-cephfs sh
/ # df -h
Filesystem Size Used Available Use% Mounted on
/dev/mapper/docker-253:48-4456452-9c5c32993ec7add2b6f89fe3f79c95860a2524312c1809d2b5a0fd49462cb828
20.0G 34.2M 19.9G 0% /
tmpfs 15.7G 0 15.7G 0% /dev
tmpfs 15.7G 0 15.7G 0% /sys/fs/cgroup
10.96.59.167:6790,10.96.125.224:6790,10.96.201.208:6790:/volumes/kubernetes/kubernetes/kubernetes-dynamic-pvc-1dfe985a-f232-11e8-a11d-fa163e05e95a
298.3G 63.0G 235.3G 21% /mnt
...
/ # cd /mnt/
/mnt # echo hello > file
在ceph tool的pod里安装ceph-fuse，然后mount cephfs后检查：
[root@ke-dev1-worker1 ~]# yum install -y ceph-fuse [root@ke-dev1-worker1 ~]# ceph-fuse /mnt/ ceph-fuse[4369]: starting ceph client2018-11-27 10:49:50.913 7fb9af66ecc0 -1 init, newargv = 0x55a4c80752c0 newargc=7 ceph-fuse[4369]: starting fuse [root@ke-dev1-worker1 ~]# df Filesystem 1K-blocks Used Available Use% Mounted on ... ceph-fuse 115531776 0 115531776 0% /mnt [root@ke-dev1-worker1 ~]# cd /mnt/volumes/kubernetes/kubernetes/kubernetes-dynamic-pvc-1dfe985a-f232-11e8-a11d-fa163e05e95a [root@ke-dev1-worker1 kubernetes-dynamic-pvc-1dfe985a-f232-11e8-a11d-fa163e05e95a]# cat file hello
总结
本文主要介绍了如何通过Rook在Kubernetes上部署Ceph集群，然后如何来提供Kubernetes里的PV服务：包括RBD和CephFS两种。
整个操作过程还是比较顺利的，对于熟悉Kubernetes的同学来说，你完全不需要理解Ceph系统，不需要理解Ceph的部署步骤，就可以很方便的来通过Ceph系统提供Kubernetes的PV存储服务。
但真正遇到问题的时候，还是需要Rook系统的log 和 Ceph Pod的log来一起定位，这对于后期维护Ceph集群的同学来说，你需要额外的Kubernetes知识，熟悉Kubernetes的基本概念和操作才行。
以上就是本文的全部内容，希望本文的内容对大家的学习或者工作能带来一定的帮助，也希望大家多多支持 码农网
猜你喜欢:
- 使用Docker容器化部署实践之Django应用部署(一）
- syncd的使用和部署
- Eureka使用及集群部署
- 如何使用 Docker 部署容器
- 使用Docker部署RabbitMQ集群
- 开始使用 Docker 线上部署
本站部分资源来源于网络，本站转载出于传递更多信息之目的，版权归原作者或者来源机构所有，如转载稿涉及版权问题，请联系我们。
UML和模式应用
拉曼 / 李洋、郑䶮 / 机械工业出版社 / 2006-5 / 66.00元
《UML和模式应用(原书第3版)》英文版面世以来，广受业界专家和读者的好评，历经3个版本的锤炼，吸收了大量OOA，D的精华思想和现代实践方法。全书叙述清晰、用词精炼、构思巧妙，将面向对象分析设计的概念、过程、方法、原则和个人的实践建议娓娓道来，以实例为证，将软件的分析和设计的过程叙述得如逻辑推理一般，于细节处见真知。 《UML和模式应用(原书第3版)》是一本经典的面向对象分析设计技术的入门书......一起来看看 《UML和模式应用》 这本书的介绍吧!
