环境说明
-
使用
Kolla-Ansible请参考《使用Kolla-Ansible在CentOS7单节点上部署OpenStackPike》； -
部署
Ceph服务请参考《Ceph学习笔记1-Mimic版本多节点部署》。
配置Ceph
-
以
osdev用户登录：
$ ssh osdev@osdev01 $ cd /opt/ceph/deploy/
创建Pool
创建镜像Pool
-
用于保存
Glance镜像：
$ ceph osd pool create images 32 32 pool 'images' created
创建卷Pool
-
用于保存
Cinder的卷：
$ ceph osd pool create volumes 32 32 pool 'volumes' created
-
用于保存
Cinder的卷备份：
$ ceph osd pool create backups 32 32 pool 'backups' created
创建虚拟机Pool
- 用于保存虚拟机系统卷：
$ ceph osd pool create vms 32 32 pool 'vms' created
查看Pool
$ ceph osd lspools 1 .rgw.root 2 default.rgw.control 3 default.rgw.meta 4 default.rgw.log 6 rbd 8 images 9 volumes 10 backups 11 vms
创建用户
查看用户
- 查看所有用户：
$ ceph auth list installed auth entries: mds.osdev01 key: AQCabn5b18tHExAAkZ6Aq3IQ4/aqYEBBey5O3Q== caps: [mds] allow caps: [mon] allow profile mds caps: [osd] allow rwx mds.osdev02 key: AQCbbn5bcq4yJRAAUfhoqPNfyp2m/ORu/7vHBA== caps: [mds] allow caps: [mon] allow profile mds caps: [osd] allow rwx mds.osdev03 key: AQCcbn5bTAIdORAApGu9NJvC3AmS+L3EWXLMdw== caps: [mds] allow caps: [mon] allow profile mds caps: [osd] allow rwx osd.0 key: AQCyJH5bG2ZBHRAAsDaLHcoOxv/mLCHwITA7JQ== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * osd.1 key: AQDTJH5bjvQ8HxAA4cyLttvZwiqFq1srFoSXWg== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * osd.2 key: AQD9JH5bbPi6IRAA7DbwaCh5JBaa6RfWPoe9VQ== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * client.admin key: AQA1In5bZkxwGBAA9bBLE5/NKstK1CRMzfGgKQ== caps: [mds] allow * caps: [mgr] allow * caps: [mon] allow * caps: [osd] allow * client.bootstrap-mds key: AQA1In5boIRwGBAAgj5OccvTGYkuB+btlgL0BQ== caps: [mon] allow profile bootstrap-mds client.bootstrap-mgr key: AQA1In5bS6pwGBAA379v3LXJrdURLmA1gnTaLQ== caps: [mon] allow profile bootstrap-mgr client.bootstrap-osd key: AQA1In5bnMpwGBAAXohUfa4rGS0Rd2weMl4dPg== caps: [mon] allow profile bootstrap-osd client.bootstrap-rbd key: AQA1In5buelwGBAANQSalrSzH3yslSc4rYPu1g== caps: [mon] allow profile bootstrap-rbd client.bootstrap-rgw key: AQA1In5b0ghxGBAAIGK3WmBSkKZMnSEfvnEQow== caps: [mon] allow profile bootstrap-rgw client.rgw.osdev01 key: AQDZbn5b6aChEBAAzRuX4UWlxyws+aX1i+D26Q== caps: [mon] allow rw caps: [osd] allow rwx client.rgw.osdev02 key: AQDabn5bypCDJBAAt18L5ppG5lEg6NkGQLYs5w== caps: [mon] allow rw caps: [osd] allow rwx client.rgw.osdev03 key: AQDbbn5bbEVNNBAArX+/AKQu9q3hCRn/05Ya3A== caps: [mon] allow rw caps: [osd] allow rwx mgr.osdev01 key: AQDPIn5beqPTORAAEzcX3fMCCclLR2RiPyvugw== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow * mgr.osdev02 key: AQDRIn5bLRVqDxAA/yWXO8pX6fQynJNyCcoNww== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow * mgr.osdev03 key: AQDSIn5bGyrhHxAAvtAEOveovRxmdDlF45i2Cg== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow *
- 查看指定用户：
$ ceph auth get client.admin exported keyring for client.admin [client.admin] key = AQA1In5bZkxwGBAA9bBLE5/NKstK1CRMzfGgKQ== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *"
创建Glance用户
-
创建
glance用户，并授予images存储池访问权限：
$ ceph auth get-or-create client.glance [client.glance] key = AQBQq4NboVHdGxAAlfK2WJkiZMolluATpvOviQ== $ ceph auth caps client.glance mon 'allow r' osd 'allow rwx pool=images' updated caps for client.glance
-
查看并保存
glance用户的KeyRing文件：
$ ceph auth get client.glance exported keyring for client.glance [client.glance] key = AQBQq4NboVHdGxAAlfK2WJkiZMolluATpvOviQ== caps mon = "allow r" caps osd = "allow rwx pool=images" $ ceph auth get client.glance -o /opt/ceph/deploy/ceph.client.glance.keyring exported keyring for client.glance
创建Cinder用户
-
创建
cinder-volume用户，并授予volumes存储池访问权限：
$ ceph auth get-or-create client.cinder-volume [client.cinder-volume] key = AQBKt4NbqROVIxAACnH+pVv141+wOpgWj14RjA== $ ceph auth caps client.cinder-volume mon 'allow r' osd 'allow rwx pool=volumes' updated caps for client.cinder-volume
-
查看并保存
cinder-volume用户的KeyRing文件：
$ ceph auth get client.cinder-volume exported keyring for client.cinder-volume [client.cinder-volume] key = AQBKt4NbqROVIxAACnH+pVv141+wOpgWj14RjA== caps mon = "allow r" caps osd = "allow rwx pool=volumes" $ ceph auth get client.cinder-volume -o /opt/ceph/deploy/ceph.client.cinder-volume.keyring exported keyring for client.cinder-volume
-
创建
cinder-backup用户，并授予volumes和backups存储池访问权限：
$ ceph auth get-or-create client.cinder-backup [client.cinder-backup] key = AQBit4NbN0rvLRAAYoa4SBM0qvwY8kPo5Md0og== $ ceph auth caps client.cinder-backup mon 'allow r' osd 'allow rwx pool=volumes, allow rwx pool=backups' updated caps for client.cinder-backup
-
查看并保存
cinder-backup用户的KeyRing文件：
$ ceph auth get client.cinder-backup exported keyring for client.cinder-backup [client.cinder-backup] key = AQBit4NbN0rvLRAAYoa4SBM0qvwY8kPo5Md0og== caps mon = "allow r" caps osd = "allow rwx pool=volumes, allow rwx pool=backups" $ ceph auth get client.cinder-backup -o /opt/ceph/deploy/ceph.client.cinder-backup.keyring exported keyring for client.cinder-backup
创建Nova用户
-
创建
nova用户，并授予vms存储池的访问权限：
$ ceph auth get-or-create client.nova [client.nova] key = AQD7tINb4A58GRAA7CsAM9EAwFwtIpTdQFGO7A== $ ceph auth caps client.nova mon 'allow r' osd 'allow rwx pool=vms' updated caps for client.nova
-
查看并保存
nova用户的KeyRing文件：
$ ceph auth get client.nova exported keyring for client.nova [client.nova] key = AQD7tINb4A58GRAA7CsAM9EAwFwtIpTdQFGO7A== caps mon = "allow r" caps osd = "allow rwx pool=vms" $ ceph auth get client.nova -o /opt/ceph/deploy/ceph.client.nova.keyring exported keyring for client.nova
配置Kolla-Ansible
-
以
root用户身份登录osdev01部署节点，并设置好环境变量：
$ ssh root@osdev01
$ export KOLLA_ROOT=/opt/kolla
$ cd ${KOLLA_ROOT}/myconfig
全局配置
-
编辑
globals.yml，禁止部署Ceph：
enable_ceph: "no"
-
开启
Cinder服务，并开启Glance、Cinder和Nova的后端Ceph功能：
enable_cinder: "yes" glance_backend_ceph: "yes" cinder_backend_ceph: "yes" nova_backend_ceph: "yes"
配置Glance
-
配置
Glance使用glance用户使用Ceph的images存储池：
$ mkdir -pv config/glance mkdir: 已创建目录 "config/glance" $ vi config/glance/glance-api.conf [glance_store] stores = rbd default_store = rbd rbd_store_pool = images rbd_store_user = glance rbd_store_ceph_conf = /etc/ceph/ceph.conf
-
新增
Glance的Ceph客户端配置和glance用户的KeyRing文件：
$ vi config/glance/ceph.conf [global] fsid = 383237bd-becf-49d5-9bd6-deb0bc35ab2a mon_initial_members = osdev01, osdev02, osdev03 mon_host = 172.29.101.166,172.29.101.167,172.29.101.168 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx $ cp -v /opt/ceph/deploy/ceph.client.glance.keyring config/glance/ceph.client.glance.keyring "/opt/ceph/deploy/ceph.client.glance.keyring" -> "config/glance/ceph.client.glance.keyring"
配置Cinder
-
配置
Cinder卷服务使用Ceph的cinder-volume用户使用volumes存储池，Cinder卷备份服务使用Ceph的cinder-backup用户使用backups存储池：
$ mkdir -pv config/cinder/
mkdir: 已创建目录 "config/cinder/"
$ vi config/cinder/cinder-volume.conf
[DEFAULT]
enabled_backends=rbd-1
[rbd-1]
rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=cinder-volume
backend_host=rbd:volumes
rbd_pool=volumes
volume_backend_name=rbd-1
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_secret_uuid = {{ cinder_rbd_secret_uuid }}
$ vi config/cinder/cinder-backup.conf
[DEFAULT]
backup_ceph_conf=/etc/ceph/ceph.conf
backup_ceph_user=cinder-backup
backup_ceph_chunk_size = 134217728
backup_ceph_pool=backups
backup_driver = cinder.backup.drivers.ceph
backup_ceph_stripe_unit = 0
backup_ceph_stripe_count = 0
restore_discard_excess_bytes = true
-
新增
Cinder的卷服务和卷备份服务的Ceph客户端配置和KeyRing文件：
$ cp config/glance/ceph.conf config/cinder/ceph.conf $ mkdir -pv config/cinder/cinder-backup/ config/cinder/cinder-volume/ mkdir: 已创建目录 "config/cinder/cinder-backup/" mkdir: 已创建目录 "config/cinder/cinder-volume/" $ cp -v /opt/ceph/deploy/ceph.client.cinder-volume.keyring config/cinder/cinder-backup/ceph.client.cinder-volume.keyring "/opt/ceph/deploy/ceph.client.cinder-volume.keyring" -> "config/cinder/cinder-backup/ceph.client.cinder-volume.keyring" $ cp -v /opt/ceph/deploy/ceph.client.cinder-backup.keyring config/cinder/cinder-backup/ceph.client.cinder-backup.keyring "/opt/ceph/deploy/ceph.client.cinder-backup.keyring" -> "config/cinder/cinder-backup/ceph.client.cinder-backup.keyring" $ cp -v /opt/ceph/deploy/ceph.client.cinder-volume.keyring config/cinder/cinder-volume/ceph.client.cinder-volume.keyring "/opt/ceph/deploy/ceph.client.cinder-volume.keyring" -> "config/cinder/cinder-volume/ceph.client.cinder.keyring"
配置Nova
-
配置
Nova使用Ceph的nova用户使用vms存储池：
$ vi config/nova/nova-compute.conf [libvirt] images_rbd_pool=vms images_type=rbd images_rbd_ceph_conf=/etc/ceph/ceph.conf rbd_user=nova
-
新增Nova的
Ceph客户端配置和nova用户的KeyRing文件：
$ cp -v config/glance/ceph.conf config/nova/ceph.conf "config/glance/ceph.conf" -> "config/nova/ceph.conf" $ cp -v /opt/ceph/deploy/ceph.client.nova.keyring config/nova/ceph.client.nova.keyring "/opt/ceph/deploy/ceph.client.nova.keyring" -> "config/nova/ceph.client.nova.keyring" $ cp -v /opt/ceph/deploy/ceph.client.cinder-volume.keyring config/nova/ceph.client.cinder.keyring "/opt/ceph/deploy/ceph.client.cinder-volume.keyring" -> "config/nova/ceph.client.cinder.keyring"
部署测试
开始部署
-
编辑部署脚本
osdev.sh：
#!/bin/bash
set -uexv
usage()
{
echo -e "usage : \n$0 <action>"
echo -e " \$1 action"
}
if [ $# -lt 1 ]; then
usage
exit 1
fi
${KOLLA_ROOT}/kolla-ansible/tools/kolla-ansible --configdir ${KOLLA_ROOT}/myconfig --passwords ${KOLLA_ROOT}/myconfig/passwords.yml --inventory ${KOLLA_ROOT}/myconfig/mynodes.conf $1
- 增加可执行权限：
$ chmod a+x osdev.sh
-
部署
OpenStack集群：
$ ./osdev.sh bootstrap-servers $ ./osdev.sh prechecks $ ./osdev.sh pull $ ./osdev.sh deploy $ ./osdev.sh post-deploy # ./osdev.sh "destroy --yes-i-really-really-mean-it"
- 查看部署的服务概况：
$ openstack service list +----------------------------------+-------------+----------------+ | ID | Name | Type | +----------------------------------+-------------+----------------+ | 304c9c5073f14f4a97ca1c3cf5e1b49e | neutron | network | | 46de4440a5cf4a5697fa94b2d0424ba9 | heat | orchestration | | 60b46b491ce7403aaec0c064384dde49 | heat-cfn | cloudformation | | 7726ab5d41c5450d954f073f1a9aff28 | cinderv2 | volumev2 | | 7a4bd5fc12904cc7b5c3810412f98c57 | gnocchi | metric | | 7ae6f98018fb4d509e862e45ebf10145 | glance | image | | a0ec333149284c09ac0e157753205fd6 | nova | compute | | b15e90c382864723945b15c37d3317a6 | placement | placement | | b5eaa49c50d64316b583eb1c0c4f9ce2 | cinderv3 | volumev3 | | c6474640f5d9424da0ec51c70c1e6e01 | nova_legacy | compute_legacy | | db27eb8524be4db3be12b9dd0dab16b8 | keystone | identity | | edf5c8b894a74a69b65bb49d8e014fff | cinder | volume | +----------------------------------+-------------+----------------+ $ openstack volume service list +------------------+-------------------+------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+-------------------+------+---------+-------+----------------------------+ | cinder-scheduler | osdev02 | nova | enabled | up | 2018-08-27T11:33:27.000000 | | cinder-volume | rbd:volumes@rbd-1 | nova | enabled | up | 2018-08-27T11:33:18.000000 | | cinder-backup | osdev02 | nova | enabled | up | 2018-08-27T11:33:17.000000 | +------------------+-------------------+------+---------+-------+----------------------------+
初始化环境
-
查看初始的
RBD存储池情况，全部是空的：
$ rbd -p images ls $ rbd -p volumes ls $ rbd -p vms ls
-
设置环境变量，并初始化
OpenStack环境：
$ . ${KOLLA_ROOT}/myconfig/admin-openrc.sh
$ ${KOLLA_ROOT}/myconfig/init-runonce
- 查看新增的镜像信息：
$ openstack image list
+--------------------------------------+--------+--------+
| ID | Name | Status |
+--------------------------------------+--------+--------+
| 293b25bb-30be-4839-b4e2-1dba3c43a56a | cirros | active |
+--------------------------------------+--------+--------+
$ openstack image show 293b25bb-30be-4839-b4e2-1dba3c43a56a
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum | 443b7623e27ecf03dc9e01ee93f67afe |
| container_format | bare |
| created_at | 2018-08-27T11:25:29Z |
| disk_format | qcow2 |
| file | /v2/images/293b25bb-30be-4839-b4e2-1dba3c43a56a/file |
| id | 293b25bb-30be-4839-b4e2-1dba3c43a56a |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros |
| owner | 68ada1726a864e2081a56be0a2dca3a0 |
| properties | locations='[{u'url': u'rbd://383237bd-becf-49d5-9bd6-deb0bc35ab2a/images/293b25bb-30be-4839-b4e2-1dba3c43a56a/snap', u'metadata': {}}]', os_type='linux' |
| protected | False |
| schema | /v2/schemas/image |
| size | 12716032 |
| status | active |
| tags | |
| updated_at | 2018-08-27T11:25:30Z |
| virtual_size | None |
| visibility | public |
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
-
查看
RBD存储池的变化，可见镜像被存储在images存储池中，并且有一个快照：
$ rbd -p images ls
293b25bb-30be-4839-b4e2-1dba3c43a56a
$ rbd -p volumes ls
$ rbd -p vms ls
$ rbd -p images info 293b25bb-30be-4839-b4e2-1dba3c43a56a
rbd image '293b25bb-30be-4839-b4e2-1dba3c43a56a':
size 12 MiB in 2 objects
order 23 (8 MiB objects)
id: 178f4008d95
block_name_prefix: rbd_data.178f4008d95
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Mon Aug 27 19:25:29 2018
$ rbd -p images snap list 293b25bb-30be-4839-b4e2-1dba3c43a56a
SNAPID NAME SIZE TIMESTAMP
6 snap 12 MiB Mon Aug 27 19:25:30 2018
创建虚拟机
- 创建一个虚拟机：
$ openstack server create --image cirros --flavor m1.tiny --key-name mykey --nic net-id=9aa15b3e-7084-450f-b0a4-7c905e6bb7c0 demo1 +-------------------------------------+-----------------------------------------------+ | Field | Value | +-------------------------------------+-----------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | None | | OS-EXT-SRV-ATTR:hypervisor_hostname | None | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-STS:power_state | NOSTATE | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | None | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | | | adminPass | 65cVBJ7S6yaD | | config_drive | | | created | 2018-08-27T11:29:03Z | | flavor | m1.tiny (1) | | hostId | | | id | 309f1364-4d58-413d-a865-dfc37ff04308 | | image | cirros (293b25bb-30be-4839-b4e2-1dba3c43a56a) | | key_name | mykey | | name | demo1 | | progress | 0 | | project_id | 68ada1726a864e2081a56be0a2dca3a0 | | properties | | | security_groups | name='default' | | status | BUILD | | updated | 2018-08-27T11:29:03Z | | user_id | c7111728fbbd4fd79bdd2b60e7d7cb42 | | volumes_attached | | +-------------------------------------+-----------------------------------------------+ $ openstack server show 309f1364-4d58-413d-a865-dfc37ff04308 +-------------------------------------+----------------------------------------------------------+ | Field | Value | +-------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | osdev03 | | OS-EXT-SRV-ATTR:hypervisor_hostname | osdev03 | | OS-EXT-SRV-ATTR:instance_name | instance-00000001 | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2018-08-27T11:29:16.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | demo-net=10.0.0.11 | | config_drive | | | created | 2018-08-27T11:29:03Z | | flavor | m1.tiny (1) | | hostId | 4e345dd9f770f63f80d3eafe97c20d97746e890b2971a8398e26db86 | | id | 309f1364-4d58-413d-a865-dfc37ff04308 | | image | cirros (293b25bb-30be-4839-b4e2-1dba3c43a56a) | | key_name | mykey | | name | demo1 | | progress | 0 | | project_id | 68ada1726a864e2081a56be0a2dca3a0 | | properties | | | security_groups | name='default' | | status | ACTIVE | | updated | 2018-08-27T11:29:16Z | | user_id | c7111728fbbd4fd79bdd2b60e7d7cb42 | | volumes_attached | | +-------------------------------------+----------------------------------------------------------+
-
可见虚拟机在
vms存储池中创建了一个卷：
$ rbd -p images ls 293b25bb-30be-4839-b4e2-1dba3c43a56a $ rbd -p volumes ls $ rbd -p backups ls $ rbd -p vms ls 309f1364-4d58-413d-a865-dfc37ff04308_disk
-
登录虚拟机所在节点，可以看到虚拟机的系统卷使用的是在
vms中创建的这个卷，从进程参数可以看出qemu直接使用的是Ceph的librbd库访问的RBD块设备：
$ ssh osdev@osdev03
$ sudo docker exec -it nova_libvirt virsh list
Id Name State
----------------------------------------------------
1 instance-00000001 running
$ sudo docker exec -it nova_libvirt virsh dumpxml 1
...
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<auth username='nova'>
<secret type='ceph' uuid='2ea5db42-c8f1-4601-927c-3c64426907aa'/>
</auth>
<source protocol='rbd' name='vms/309f1364-4d58-413d-a865-dfc37ff04308_disk'>
<host name='172.29.101.166' port='6789'/>
<host name='172.29.101.167' port='6789'/>
<host name='172.29.101.168' port='6789'/>
</source>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
...
$ ps -aux | grep qemu
42436 2678909 4.6 0.0 1341144 171404 ? Sl 19:29 0:08 /usr/libexec/qemu-kvm -name guest=instance-00000001,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-instance-00000001/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client-IBRS,ss=on,hypervisor=on,tsc_adjust=on,avx512f=on,avx512dq=on,clflushopt=on,clwb=on,avx512cd=on,avx512bw=on,avx512vl=on,pku=on,stibp=on,pdpe1gb=on -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 309f1364-4d58-413d-a865-dfc37ff04308 -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=17.0.2,serial=74bf926c-70b7-03df-b211-d21d6016081a,uuid=309f1364-4d58-413d-a865-dfc37ff04308,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-instance-00000001/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -object secret,id=virtio-disk0-secret0,data=zNy84nlNYigA4vjbuOxcGQa1/hh8w28i/WoJbO1Xsl4=,keyid=masterKey0,iv=OhX+FApyFyq2XLWq0ff/Ew==,format=base64 -drive file=rbd:vms/309f1364-4d58-413d-a865-dfc37ff04308_disk:id=nova:auth_supported=cephx\;none:mon_host=172.29.101.166\:6789\;172.29.101.167\:6789\;172.29.101.168\:6789,file.password-secret=virtio-disk0-secret0,format=raw,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=79,id=hostnet0,vhost=on,vhostfd=80 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:04:e8:e9,bus=pci.0,addr=0x3 -chardev pty,id=charserial0,logfile=/var/lib/nova/instances/309f1364-4d58-413d-a865-dfc37ff04308/console.log,logappend=off -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 172.29.101.168:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
$ ldd /usr/libexec/qemu-kvm | grep -e ceph -e rbd
librbd.so.1 => /lib64/librbd.so.1 (0x00007fde38815000)
libceph-common.so.0 => /usr/lib64/ceph/libceph-common.so.0 (0x00007fde28247000)
创建卷
- 创建一个卷：
$ openstack volume create --size 1 volume1 +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2018-08-27T11:33:52.000000 | | description | None | | encrypted | False | | id | 3ccca300-bee3-4b5a-b89b-32e6b8b806d9 | | migration_status | None | | multiattach | False | | name | volume1 | | properties | | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | type | None | | updated_at | None | | user_id | c7111728fbbd4fd79bdd2b60e7d7cb42 | +---------------------+--------------------------------------+
-
查看存储池状态，可以看到新建的卷被放在
volumes存储池：
$ rbd -p images ls 293b25bb-30be-4839-b4e2-1dba3c43a56a $ rbd -p volumes ls volume-3ccca300-bee3-4b5a-b89b-32e6b8b806d9 $ rbd -p backups ls $ rbd -p vms ls 309f1364-4d58-413d-a865-dfc37ff04308_disk
创建备份
-
创建一个卷备份，可以看到是创建在
backups存储池中：
$ openstack volume backup create 3ccca300-bee3-4b5a-b89b-32e6b8b806d9 +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | f2321578-88d5-4337-b93c-798855b817ce | | name | None | +-------+--------------------------------------+ $ openstack volume backup list +--------------------------------------+------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+------+-------------+-----------+------+ | f2321578-88d5-4337-b93c-798855b817ce | None | None | available | 1 | +--------------------------------------+------+-------------+-----------+------+ $ openstack volume backup show f2321578-88d5-4337-b93c-798855b817ce +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | availability_zone | nova | | container | backups | | created_at | 2018-08-27T11:39:40.000000 | | data_timestamp | 2018-08-27T11:39:40.000000 | | description | None | | fail_reason | None | | has_dependent_backups | False | | id | f2321578-88d5-4337-b93c-798855b817ce | | is_incremental | False | | name | None | | object_count | 0 | | size | 1 | | snapshot_id | None | | status | available | | updated_at | 2018-08-27T11:39:46.000000 | | volume_id | 3ccca300-bee3-4b5a-b89b-32e6b8b806d9 | +-----------------------+--------------------------------------+ $ rbd -p backups ls volume-3ccca300-bee3-4b5a-b89b-32e6b8b806d9.backup.base
-
在此创建一个备份，发现
backups存储池并无变化，仅仅是在原有的备份卷中增加一个快照：
$ volume backup create 3ccca300-bee3-4b5a-b89b-32e6b8b806d9
+-------+--------------------------------------+
| Field | Value |
+-------+--------------------------------------+
| id | 07132063-9bdb-4391-addd-a791dae2cfea |
| name | None |
+-------+--------------------------------------+
$ rbd -p backups ls
volume-3ccca300-bee3-4b5a-b89b-32e6b8b806d9.backup.base
$ rbd -p backups snap list volume-3ccca300-bee3-4b5a-b89b-32e6b8b806d9.backup.base
SNAPID NAME SIZE TIMESTAMP
4 backup.f2321578-88d5-4337-b93c-798855b817ce.snap.1535369984.08 1 GiB Mon Aug 27 19:39:46 2018
5 backup.07132063-9bdb-4391-addd-a791dae2cfea.snap.1535370126.76 1 GiB Mon Aug 27 19:42:08 2018
连接卷
- 把新增的卷链接到之前创建的虚拟机中：
$ openstack server add volume demo1 volume1
$ openstack volume show volume1
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'server_id': u'309f1364-4d58-413d-a865-dfc37ff04308', u'attachment_id': u'fb4d9ec0-8a33-4ed0-8845-09e6f17aac81', u'attached_at': u'2018-08-27T11:44:51.000000', u'host_name': u'osdev03', u'volume_id': u'3ccca300-bee3-4b5a-b89b-32e6b8b806d9', u'device': u'/dev/vdb', u'id': u'3ccca300-bee3-4b5a-b89b-32e6b8b806d9'}] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2018-08-27T11:33:52.000000 |
| description | None |
| encrypted | False |
| id | 3ccca300-bee3-4b5a-b89b-32e6b8b806d9 |
| migration_status | None |
| multiattach | False |
| name | volume1 |
| os-vol-host-attr:host | rbd:volumes@rbd-1#rbd-1 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 68ada1726a864e2081a56be0a2dca3a0 |
| properties | attached_mode='rw' |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| type | None |
| updated_at | 2018-08-27T11:44:52.000000 |
| user_id | c7111728fbbd4fd79bdd2b60e7d7cb42 |
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
到虚拟机所在节点查看其
libvirt上参数的变化，发现新增了一个RBD磁盘：
$ sudo docker exec -it nova_libvirt virsh dumpxml 1
...
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<auth username='nova'>
<secret type='ceph' uuid='2ea5db42-c8f1-4601-927c-3c64426907aa'/>
</auth>
<source protocol='rbd' name='vms/309f1364-4d58-413d-a865-dfc37ff04308_disk'>
<host name='172.29.101.166' port='6789'/>
<host name='172.29.101.167' port='6789'/>
<host name='172.29.101.168' port='6789'/>
</source>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none' discard='unmap'/>
<auth username='cinder-volume'>
<secret type='ceph' uuid='3fa55f7c-b556-4095-9253-b908d5408ec8'/>
</auth>
<source protocol='rbd' name='volumes/volume-3ccca300-bee3-4b5a-b89b-32e6b8b806d9'>
<host name='172.29.101.166' port='6789'/>
<host name='172.29.101.167' port='6789'/>
<host name='172.29.101.168' port='6789'/>
</source>
<target dev='vdb' bus='virtio'/>
<serial>3ccca300-bee3-4b5a-b89b-32e6b8b806d9</serial>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>
...
-
为虚拟机创建一个浮动
IP，使用SSH登陆进去：
$ openstack console url show demo1 +-------+-------------------------------------------------------------------------------------+ | Field | Value | +-------+-------------------------------------------------------------------------------------+ | type | novnc | | url | http://172.29.101.167:6080/vnc_auto.html?token=9f835216-1c53-41ae-849a-44a85429a334 | +-------+-------------------------------------------------------------------------------------+ $ openstack floating ip create public1 +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | created_at | 2018-08-27T11:49:02Z | | description | | | fixed_ip_address | None | | floating_ip_address | 192.168.162.52 | | floating_network_id | ff69b3ff-c2c4-4474-a7ba-952fa99df919 | | id | 2aa86075-9c62-49f5-84ac-e7b6353c9591 | | name | 192.168.162.52 | | port_id | None | | project_id | 68ada1726a864e2081a56be0a2dca3a0 | | qos_policy_id | None | | revision_number | 0 | | router_id | None | | status | DOWN | | subnet_id | None | | tags | [] | | updated_at | 2018-08-27T11:49:02Z | +---------------------+--------------------------------------+ $ openstack server add floating ip demo1 192.168.162.52 $ openstack server list +--------------------------------------+-------+--------+------------------------------------+--------+---------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------+--------+------------------------------------+--------+---------+ | 309f1364-4d58-413d-a865-dfc37ff04308 | demo1 | ACTIVE | demo-net=10.0.0.11, 192.168.162.52 | cirros | m1.tiny | +--------------------------------------+-------+--------+------------------------------------+--------+---------+ $ ssh root@osdev02 $ ip netns qrouter-65759e60-6e20-41cc-a79c-fc492232b127 (id: 1) qdhcp-9aa15b3e-7084-450f-b0a4-7c905e6bb7c0 (id: 0) $ ip netns exec qrouter-65759e60-6e20-41cc-a79c-fc492232b127 ping 192.168.162.50 $ ip netns exec qdhcp-9aa15b3e-7084-450f-b0a4-7c905e6bb7c0 ping 10.0.0.9 （用户名"cirros"，密码"gocubsgo"） $ ip netns exec qrouter-65759e60-6e20-41cc-a79c-fc492232b127 ssh cirros@192.168.162.52 $ ip netns exec qdhcp-9aa15b3e-7084-450f-b0a4-7c905e6bb7c0 ssh cirros@10.0.0.11 $ sudo passwd root Changing password for root New password: Bad password: too weak Retype password: Password for root changed by root $ su - Password:
- 创建分区并写入测试文件，最后卸载分区：
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 1G 0 disk |-vda1 253:1 0 1015M 0 part / `-vda15 253:15 0 8M 0 part vdb 253:16 0 1G 0 disk # mkfs.ext4 /dev/vdb mke2fs 1.42.12 (29-Aug-2014) Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: ede8d366-bfbc-4b9a-9d3f-306104f410d7 Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done # mount /dev/vdb /mnt # df -h Filesystem Size Used Available Use% Mounted on /dev 240.1M 0 240.1M 0% /dev /dev/vda1 978.9M 23.9M 914.1M 3% / tmpfs 244.2M 0 244.2M 0% /dev/shm tmpfs 244.2M 92.0K 244.1M 0% /run /dev/vdb 975.9M 1.3M 907.4M 0% /mnt # echo "hello openstack, volume test." > /mnt/ceph_rbd_test # umount /mnt # df -h Filesystem Size Used Available Use% Mounted on /dev 240.1M 0 240.1M 0% /dev /dev/vda1 978.9M 23.9M 914.1M 3% / tmpfs 244.2M 0 244.2M 0% /dev/shm tmpfs 244.2M 92.0K 244.1M 0% /run
断开卷
- 断开卷，同时查看虚拟机内部变化：
$ openstack server remove volume demo1 volume1 # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 1G 0 disk |-vda1 253:1 0 1015M 0 part / `-vda15 253:15 0 8M 0 part
-
在宿主机映射和挂载
RBD卷，并查看之前虚拟机内部创建的文件，完全相同：
$ rbd showmapped id pool image snap device 0 rbd rbd_test - /dev/rbd0 $ rbd feature disable volumes/volume-3ccca300-bee3-4b5a-b89b-32e6b8b806d9 object-map fast-diff deep-flatten $ rbd map volumes/volume-3ccca300-bee3-4b5a-b89b-32e6b8b806d9 /dev/rbd1 $ mkdir /mnt/volume1 $ mount /dev/rbd1 /mnt/volume1/ $ cat /mnt/volume1/ ceph_rbd_test lost+found/ $ cat /mnt/volume1/ceph_rbd_test hello openstack, volume test.
参考文档
以上就是本文的全部内容，希望本文的内容对大家的学习或者工作能带来一定的帮助，也希望大家多多支持 码农网
猜你喜欢:
- MySQL 提升笔记（二）：存储引擎盘点
- Python笔记（五）：异常处理和数据存储
- Oracle-OCP学习笔记:存储结构
- Mysql学习笔记：存储引擎MyISAM和InnoDB的区别
- golang分布式存储 读书笔记(2)——流操作之PutStream封装
- golang分布式存储 读书笔记(3)——数据冗余之RS码
本站部分资源来源于网络，本站转载出于传递更多信息之目的，版权归原作者或者来源机构所有，如转载稿涉及版权问题，请联系我们。
