Openstack运维常见问题记录

栏目: 数据库 · 发布时间: 7年前

内容简介:分析:感觉是node的信息数据库不同步nova show 出错的vm,包cell错误问题解决。
  • 首先确定故障的资源ID,并判断故障发生的组件
  • 查看对应组件的日志,根据故障资源ID进行搜索,找到相应的ERROR日志
  • 如果ERROR日志又将问题指向了其他组件,则根据ERROR日志中的资源ID、时间、req-id等信息,其他组件继续查找问题,直到发现问题根源。
  • 如果没有在日志中发现任何ERROR,有可能是网络不通,导致请求无法到达API,此时需要检查和API的连通性(如果使用VIP的话,需要分别检查和VIP的联通性和实IP的连通性)。
  • 如果API中能找到相应请求,但是conductor/scheduler/compute没有找到相应日志,则有可能是MQ发生故障。
  • 如果组件长时间没有任何日志刷新,有可能是组件进程挂掉或者处于僵死状态,可以尝试重启服务,或先打开Debug再重启服务。(对线上环境,重启服务操作需要首先报备,经甲方同意后再操作!)

创建vm报错

[root@EXTENV-194-18-2-16 nova]# cat nova-compute.log | grep 620cd801-8849-481a-80e0-2980b6c8dba6
2018-08-23 15:23:36.136 3558 INFO nova.compute.resource_tracker [req-f76d5408-00f8-4a67-854e-ad3da2098811 - - - - -] Instance 620cd801-8849-481a-80e0-2980b6c8dba6 has allocations against this compute host but is not found in the database.

分析:感觉是node的信息数据库不同步

nova show 出错的vm,包cell错误

####每次增加一个计算节点在控制节点需要执行:
# su -s /bin/sh -c "nova-manage cell_v2 discover_hosts --verbose" nova

问题解决。

neutron服务是好的,命令行创建网络查看网络正常,就是dashboard不能查看网络相关的页面

dashboard 网络页面报错 Invalid service catalog service: network

分析:应该是Keystone没有正常配置。导致没有找到相关的Catalog信息。

[root@EXTENV-194-18-2-11 ~]# openstack catalog list
+-----------+-----------+-----------------------------------------------+
| Name      | Type      | Endpoints                                     |
+-----------+-----------+-----------------------------------------------+
| placement | placement | RegionOne                                     |
|           |           |   internal: http://nova-ha-vip:8778           |
|           |           | RegionOne                                     |
|           |           |   admin: http://nova-ha-vip:8778              |
|           |           | RegionOne                                     |
|           |           |   public: http://nova-ha-vip:8778             |
|           |           |                                               |
| keystone  | identity  | RegionOne                                     |
|           |           |   public: http://keystone-ha-vip:5000/v3/     |
|           |           | RegionOne                                     |
|           |           |   internal: http://keystone-ha-vip:35357/v3/  |
|           |           | RegionOne                                     |
|           |           |   admin: http://keystone-ha-vip:35357/v3/     |
|           |           |                                               |
| glance    | image     | RegionOne                                     |
|           |           |   admin: http://glance-ha-vip:9292            |
|           |           | RegionOne                                     |
|           |           |   internal: http://glance-ha-vip:9292         |
|           |           | RegionOne                                     |
|           |           |   public: http://glance-ha-vip:9292           |
|           |           |                                               |
| nova      | compute   | RegionOne                                     |
|           |           |   public: http://nova-ha-vip:8774/v2.1        |
|           |           | RegionOne                                     |
|           |           |   admin: http://nova-ha-vip:8774/v2.1         |
|           |           | RegionOne                                     |
|           |           |   internal: http://nova-ha-vip:8774/v2.1      |
|           |           |                                               |
| neutron   | network   |                                               |
| neutron   | network   | RegionOne                                     |
|           |           |   public: http://neutron-server-ha-vip:9696   |
|           |           | RegionOne                                     |
|           |           |   admin: http://neutron-server-ha-vip:9696    |
|           |           | RegionOne                                     |
|           |           |   internal: http://neutron-server-ha-vip:9696 |
|           |           |                                               |
+-----------+-----------+-----------------------------------------------+

所以把第一条没有url的数据删掉就好了,但是发现只有openstack catalog list,没有openstack catalog delete命令,后来查看keystone的配置文件keystone.conf发现如下配置

见[catalog]

从配置文件和一些资料看出catalog是从 mysql 里面读取的数据,然后从mysql的keystone库中的service表里找到了脏数据,然后知道了用openstack service delete去删除‘脏数据’,问题就解决了。

MariaDB [keystone]> select * from service;
+----------------------------------+-----------+---------+-------------------------------------------------------------+
| id                               | type      | enabled | extra                                                       |
+----------------------------------+-----------+---------+-------------------------------------------------------------+
| 520f6bf8564240be9678c4ef25305cad | placement |       1 | {"description": "OpenStack Placement", "name": "placement"} |
| 960580852a594c078e68fe3683e35db5 | identity  |       1 | {"name": "keystone"}                                        |
| 98ed18fcd8104732919bb5869a5a6dc2 | image     |       1 | {"description": "OpenStack Image", "name": "glance"}        |
| abef1b9469d94d3ab9f27c8ed72a5a48 | compute   |       1 | {"description": "OpenStack Compute", "name": "nova"}        |
| e37085e8fb2a49c0921c2d24f5e4f9b5 | network   |       1 | {"description": "OpenStack Networking", "name": "neutron"}  |
| f1b661407ce04f79bc24605fa59bb74c | network   |       1 | {"description": "OpenStack Networking", "name": "neutron"}  |
+----------------------------------+-----------+---------+-------------------------------------------------------------+
6 rows in set (0.00 sec)

MariaDB [keystone]> select * from endpoint;
+----------------------------------+--------------------+-----------+----------------------------------+-----------------------------------+-------+---------+-----------+
| id                               | legacy_endpoint_id | interface | service_id                       | url                               | extra | enabled | region_id |
+----------------------------------+--------------------+-----------+----------------------------------+-----------------------------------+-------+---------+-----------+
| 142cb619cd2242828b0c9394d5baaea1 | NULL               | public    | f1b661407ce04f79bc24605fa59bb74c | http://neutron-server-ha-vip:9696 | {}    |       1 | RegionOne |
| 2252d3ef840b4c5aa1184ebe8d6094f1 | NULL               | public    | abef1b9469d94d3ab9f27c8ed72a5a48 | http://nova-ha-vip:8774/v2.1      | {}    |       1 | RegionOne |
| 476654c6e7dd4d22b290de451e3afda0 | NULL               | admin     | abef1b9469d94d3ab9f27c8ed72a5a48 | http://nova-ha-vip:8774/v2.1      | {}    |       1 | RegionOne |
| 562a5d5443af4dfab6760204d0adf3bf | NULL               | internal  | 520f6bf8564240be9678c4ef25305cad | http://nova-ha-vip:8778           | {}    |       1 | RegionOne |
| 58bd5f09811a4ebcb62a4b51fb7ae444 | NULL               | admin     | f1b661407ce04f79bc24605fa59bb74c | http://neutron-server-ha-vip:9696 | {}    |       1 | RegionOne |
| 600811f8ccaf42669d4d83b897af3933 | NULL               | admin     | 520f6bf8564240be9678c4ef25305cad | http://nova-ha-vip:8778           | {}    |       1 | RegionOne |
| 80683f619efb41dcbb6796ea04f16159 | NULL               | internal  | f1b661407ce04f79bc24605fa59bb74c | http://neutron-server-ha-vip:9696 | {}    |       1 | RegionOne |
| 8e0a684607294a729f87d7d8b1a639ca | NULL               | public    | 520f6bf8564240be9678c4ef25305cad | http://nova-ha-vip:8778           | {}    |       1 | RegionOne |
| 9ef0f18d891e45608ffc41985dc6afa6 | NULL               | public    | 960580852a594c078e68fe3683e35db5 | http://keystone-ha-vip:5000/v3/   | {}    |       1 | RegionOne |
| a0b10cb04a5b4ca3859aaf2ea4ca2a3b | NULL               | admin     | 98ed18fcd8104732919bb5869a5a6dc2 | http://glance-ha-vip:9292         | {}    |       1 | RegionOne |
| c53979becccc44f1813e9f50a619af7e | NULL               | internal  | 960580852a594c078e68fe3683e35db5 | http://keystone-ha-vip:35357/v3/  | {}    |       1 | RegionOne |
| dadbb8dc218245bbba8c9a34237413ec | NULL               | internal  | 98ed18fcd8104732919bb5869a5a6dc2 | http://glance-ha-vip:9292         | {}    |       1 | RegionOne |
| f4034b8c086a451caed52ac51a761fb0 | NULL               | public    | 98ed18fcd8104732919bb5869a5a6dc2 | http://glance-ha-vip:9292         | {}    |       1 | RegionOne |
| fc150884825544baaf4912f14e76f51a | NULL               | internal  | abef1b9469d94d3ab9f27c8ed72a5a48 | http://nova-ha-vip:8774/v2.1      | {}    |       1 | RegionOne |
| fc7132052063438895674fd7b840db68 | NULL               | admin     | 960580852a594c078e68fe3683e35db5 | http://keystone-ha-vip:35357/v3/  | {}    |       1 | RegionOne |
+----------------------------------+--------------------+-----------+----------------------------------+-----------------------------------+-------+---------+-----------+
15 rows in set (0.00 sec)

[root@EXTENV-194-18-2-11 ~]#  openstack service list
+----------------------------------+-----------+-----------+
| ID                               | Name      | Type      |
+----------------------------------+-----------+-----------+
| 520f6bf8564240be9678c4ef25305cad | placement | placement |
| 960580852a594c078e68fe3683e35db5 | keystone  | identity  |
| 98ed18fcd8104732919bb5869a5a6dc2 | glance    | image     |
| abef1b9469d94d3ab9f27c8ed72a5a48 | nova      | compute   |
| e37085e8fb2a49c0921c2d24f5e4f9b5 | neutron   | network   |
| f1b661407ce04f79bc24605fa59bb74c | neutron   | network   |
+----------------------------------+-----------+-----------+
[root@EXTENV-194-18-2-11 ~]#  openstack service delete e37085e8fb2a49c0921c2d24f5e4f9b5
[root@EXTENV-194-18-2-11 ~]# systemctl restart httpd.service memcached.service

查看中文命名的镜像的时候,报错

[root@NFJD-TESTVM-CORE-API-1 ~]# glance image-list
'ascii' codec can't encode character u'\u5982' in position 1242: ordinal not in range(128)

分析:镜像名字有用中文命名的情况。

将export LC_ALL=zh_CN.UTF-8

添加到/etc/profile

同时注意source的文件中是否有export LC_ALL=C

vm 生成失败

2017-05-25 11:01:29.577 21880 TRACE nova.compute.manager [instance: 2c5a8e62-62d0-430d-8747-795350bb6939] ProcessExecutionError: Unexpected error while running command.
2017-05-25 11:01:29.577 21880 TRACE nova.compute.manager [instance: 2c5a8e62-62d0-430d-8747-795350bb6939] Command: qemu-img convert -O raw /var/lib/nova/instances/_base/f5797db00aacfbe240bbfb0f53c2da80e4be6dfc.part /var/lib/nova/instances/_base/f5797db00aacfbe240bbfb0f53c2da80e4be6dfc.converted
2017-05-25 11:01:29.577 21880 TRACE nova.compute.manager [instance: 2c5a8e62-62d0-430d-8747-795350bb6939] Exit code: 1
2017-05-25 11:01:29.577 21880 TRACE nova.compute.manager [instance: 2c5a8e62-62d0-430d-8747-795350bb6939] Stdout: u''
2017-05-25 11:01:29.577 21880 TRACE nova.compute.manager [instance: 2c5a8e62-62d0-430d-8747-795350bb6939] Stderr: u'qemu-img: error while writing sector 1569792: No space left on device\n'
2017-05-25 11:01:29.577 21880 TRACE nova.compute.manager [instance: 2c5a8e62-62d0-430d-8747-795350bb6939]
2017-05-25 11:01:29.580 21880 INFO nova.compute.manager [req-9fa74abf-bcc1-4b7e-aaef-2e17b593a356 6aa5df16b47442c58efde791abd60497 66458b9ead64409fb9d2e0f2c6d67d39 - - -] [instance: 2c5a8e62-62d0-430d-8747-795350bb6939] Terminating instance


# df -h

发现磁盘所剩不多了

对策:清理磁盘

疏散失败

执行了个nova疏散命令,疏散不了,nova show 报这个错误:共享存储的状态不对。就是nova instances目录不具备共享存储,如没有配置了ceph共享存储,但是在执行nova evacuate命令的时候还是带了共享存储的参数。

| fault                                | {"message": "Invalid state of instance files on shared storage", "code": 500, "details": "  File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 354, in decorated_function |
|                                      |     return function(self, context, *args, **kwargs)                                                                                                                                            |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 3031, in rebuild_instance                                                                                            |
|                                      |     _(\"Invalid state of instance files on shared\"                                                                                                                                            |
|                                      | ", "created": "2017-05-17T01:25:17Z"}

有一定的概率无法显示控制台

操作步骤:控制台-虚拟机,点击虚拟机名称,点击【控制台】

预期结果:正常显示console页面

实际结果:有一定概率页面提示“Failed to connect to server”,点击在新窗口打开,可以打开console页面

nova控制节点的配置问题,其配置文件中memcache以及rabbitmq的hosts配置不对

在dashboard界面上,用镜像centos7、flavor–2,创建虚机error。并且抛出的错误是no host,即没有找到合适的调度节点。

若flavor的大小比镜像要求的还小,则会报错。

但又一次出现满足上述条件也报错。

有可能用的flavor被创建不合法的extra_specs

OS-FLV-DISABLED:disabled  False
OS-FLV-EXT-DATA:ephemeral   0
disk    20
extra_specs {"xxx": "123", "yyy": "321"}

而过滤选项又开启了ComputeCapabilitiesFilter过滤器。

import error

glance register.log报错:

2017-05-08 03:18:55.890 3185 ERROR glance.common.config [-] Unable to load glance-registry-keystone from configuration file /usr/share/glance/glance-registry-dist-paste.ini.
Got: ImportError('No module named simplegeneric',)

/usr/lib/python2.7/site-packages/simplegeneric.py 没有读权限,被谁给改了

keystone报错

Permission denied: AH00072: make_sock: could not bind to address [::]:5000

[root@controller0 ~]# systemctl start httpd.service
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
[root@controller0 ~]# systemctl status httpd.service
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sat 2016-05-28 20:22:34 EDT; 11s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 4501 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 4499 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 4499 (code=exited, status=1/FAILURE)

May 28 20:22:34 controller0 httpd[4499]: (13)Permission denied: AH00072: make_sock: could not bind to address [::]:5000
May 28 20:22:34 controller0 httpd[4499]: (13)Permission denied: AH00072: make_sock: could not bind to address 0.0.0.0:5000

可能是防火墙的配置问题或者selinux

若防火墙没有问题,检查selinux

检查selinux状态

1 [root@controller0 ~]# getenforce

2 enforcing #如果不为disabled 则表示为selinux正常运行

SELINUX=enforcing 改为 selinux=distabled

重启reboot

re-scheduled: Not authorized for image

CLI报错:

| fault                                | {"message": "Build of instance d5739cf7-9830-47fd-9a75-e9b1cb4bb421 was re-scheduled: Not authorized for image dcd85799-92f6-4294-91ec-48670a218651.", "code": 500, "details": "  File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2258, in _do_build_and_run_instance |

登陆到该计算节点,也报错

2017-05-18 15:01:24.867 40639 TRACE nova.compute.manager [instance: d5739cf7-9830-47fd-9a75-e9b1cb4bb421] ImageNotAuthorized: Not authorized for image dcd85799-92f6-4294-91ec-48670a218651.
2017-05-18 15:01:24.867 40639 TRACE nova.compute.manager [instance: d5739cf7-9830-47fd-9a75-e9b1cb4bb421]

于是在 /etc/nova/nova.conf中 增加

[DEFAULT]

auth_strategy=keyston

少了个e

应该是:

[DEFAULT]

auth_strategy=keystone

libvirtError: unsupported configuration: IDE controllers are unsupported for this QEMU binary or machine type

2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f] Traceback (most recent call last):
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2483, in _build_resources
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]     yield resources
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2355, in _build_and_run_instance
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]     block_device_info=block_device_info)
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2704, in spawn
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]     block_device_info=block_device_info)
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4758, in _create_domain_and_network
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]     power_on=power_on)
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4689, in _create_domain
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]     LOG.error(err)
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]     six.reraise(self.type_, self.value, self.tb)
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4679, in _create_domain
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]     domain.createWithFlags(launch_flags)
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]     rv = execute(f, *args, **kwargs)
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]     six.reraise(c, e, tb)
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]     rv = meth(*args, **kwargs)
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in createWithFlags
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2017-05-18 15:06:09.522 41033 TRACE nova.compute.manager [instance: c348b942-4553-4023-bbcb-296f3b1bf14f] libvirtError: unsupported configuration: IDE controllers are unsupported for this QEMU binary or machine type

将IDE改成virtio

qemu-kvm: Cirrus VGA not available

| fault                                | {"message": "Build of instance a1feb48a-b5f5-48ab-93a7-838bb46573fb was re-scheduled: internal error: process exited while connecting to monitor: 2017-05-18T10:33:34.222333Z qemu-kvm: Cirrus VGA not available", "code": 500, "details": "  File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2258, in _do_build_and_run_instance |

power架构的机器不支持Cirrus VGA

默认路由不能删除

默认路由怎么也删除不掉

有两个service确认下状态是否正常。network和NetworkManager

dashboard访问卡顿

定位到dashboard卡的原因是应为nova卡,

nova卡顿是因为nova无法与 memcached 建立连接,

进一步定位到memcached默认的最大连接数是1024,目前已达到最大连接数。

解决办法为编辑 /etc/sysconfig/memcached

参数修改为:

PORT="11211"
USER="memcached"
MAXCONN="65536"
CACHESIZE="1024"
OPTIONS=""

重启memcached

task_state一直处于scheduling

nova boot 执行某一个节点生成vm, task_state一直处于scheduling,vm_state一直处于building,有可能是强制调度的节点的nova-compute state down了。

Access denied for user ‘nova’@’%’ to database ‘nova_api’

初始化nova_api数据库的时候

初始化数据库

su -s /bin/sh -c "nova-manage api_db sync" nova

报错:

Access denied for user 'nova'@'%' to database 'nova_api'

root用户进入数据库,执行

> GRANT ALL PRIVILEGES ON nova_api.* TO 'nova'@'%' IDENTIFIED BY '2267593eb27be7c414fc';

解决

所有节点disk不足,返回0可用host

到该节点df -h发现磁盘空间剩余很多。nova判断节点的磁盘空间不是根据ll判断的,而是根据vm和其他数据一共占用的空间来计算的。

如:

[root@NFJD-PSC-IBMN-SV356 10d31bfa-961d-44ea-b554-7575315a8e2e]# ll -h
total 679M
-rw-rw---- 1 root root  16K Nov 17 17:32 console.log
-rw-r--r-- 1 root root 679M Nov 18 10:07 disk
-rw-r--r-- 1 qemu qemu 422K Nov 17 17:24 disk.config
-rw-r--r-- 1 nova nova  162 Nov 17 17:24 disk.info
-rw-r--r-- 1 nova nova 3.4K Nov 17 17:24 libvirt.xml
[root@NFJD-PSC-IBMN-SV356 10d31bfa-961d-44ea-b554-7575315a8e2e]# qemu-img info disk
image: disk
file format: qcow2
virtual size: 40G (42949672960 bytes)
disk size: 678M
cluster_size: 65536
backing file: /var/lib/nova/instances/_base/af019e4c89c44506c068ada379c040848416510e
Format specific information:
    compat: 1.1
    lazy refcounts: false

ll 这个文件出来的是600多兆,但是nova统计是按40G来统计的。因为nova代码要统计的是disk_over_committed。nova会把统计的磁盘信息存放在nova数据库的表compute_nodes中提供给调度器。

这里的virtual size减去disk size,便是over_commit_size。

可以看到,这里仅仅对qcow2格式的镜像做了overcommit处理,其它文件的over_commit_size等于0。

在nova调度服务的DiskFilter里面,用到了disk_allocation_ratio对磁盘资源做了超分,它和这里的overcommit不是一个概念,它是从控制节点角度看到的超额使用,而计算节点看不到,overcommit是计算节点看到了磁盘qcow2压缩格式之后所得到的结果,它最终上报的剩余空间是扣除了假设qcow2镜像文件解压之后的实际结果。所以会遇到实际上报的剩余空间小于肉眼看到的空间大小。

如果管理员部署时指定了计算节点,则不走调度流程,就会把虚拟机硬塞给该计算节点,强行占用了已经归入超额分配计划的空间,则最终可能导致计算节点上报的磁盘资源为负数。并且将来随着虚拟机实际占用的磁盘空间越来越大,最终可能就导致计算节点硬盘空间不足了。

novnc打不开问题定位

可能compute的防火墙被改了

在 /etc/sysconfig/iptables 添加

-A INPUT -p tcp --dport 5900:6100 -j ACCEPT

qemu-ga由于找不到对应的虚拟串行字符设备而启动失败,提示找不到channel

glance image-update –property hw_qemu_guest_agent=yes $IMAGE_ID# … 其他属性配置

务必设置property的hw_qemu_guest_agent=yes,否则libvert启动虚拟机时不会生成qemu-ga配置项,导致虚拟机内部的qemu-ga由于找不到对应的虚拟串行字符设备而启动失败,提示找不到channel。

并发创建虚机,有一些失败

在log中看到错误为rabbitmq不能端口连接不上

初步定位到是由于rabbitmq压力过大,导致rabbitmq 不能正常工作。现在采取一些措施来改善此情况。

①.把 rabbitmq 的集群的rabbitmq节点2 和节点3 由disc模式改为ram 模式。

②把rabbitmq 压力分散到3个rabbitmq节点。通过查看rabbitmq log 发现在没有修改之前,rabbitmq 压力主要在node01 上,其他两个节点几乎不会处理消息请求。

③把rabbitmq 监控的rate mode 功能关闭。这个rate mode功能监控消息队列中消息输速率。关闭对服务没有影响。

④nova-compute之前配置的[glance] api_servers=vip:9292,vip是管理网地址,当创建虚拟机等并发大时,会出现镜像下载占用管理网通道,导致rabbitmq等网络敏感服务消息阻塞,甚至消息超时,因此,将其api_servers配置为控制节点的存储地址,且把控制节点镜像服务由监听管理网改为监听存储网。

采取以上措施后,进行了测试

1.测试并发创建虚机100 个,虚机创建无一失败

2.测试并发创建卷100个,创建无一失败

一台控制节点内存使用率过高告警

一台控制节点内存使用率过高告警,发现是rabbitmq进程异常导致,消息队列中积压的消息过多导致内存增大无法释放,重启rabbitmq进程解决问题,最终解决问题需要修改rabbitmq配置文件,使得积压的消息存储在磁盘中而不是内存中。

对云主机进行resize操作没有成功

对一个vm做resize,即从一个小的flavor换一个大的flavor,没有成功

检查云主机所在节点的nova-compute.log

2017-07-03 17:50:08.573 24296 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6959, in _error_out_instance_on_exception
2017-07-03 17:50:08.573 24296 TRACE oslo_messaging.rpc.dispatcher     raise error.inner_exception
2017-07-03 17:50:08.573 24296 TRACE oslo_messaging.rpc.dispatcher ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command.
2017-07-03 17:50:08.573 24296 TRACE oslo_messaging.rpc.dispatcher Command: ssh 172.16.231.26 -p 22 mkdir -p /var/lib/nova/instances/54091e90-55f5-4f4d-8f66-31fbc787584f
2017-07-03 17:50:08.573 24296 TRACE oslo_messaging.rpc.dispatcher Exit code: 255
2017-07-03 17:50:08.573 24296 TRACE oslo_messaging.rpc.dispatcher Stdout: u''
2017-07-03 17:50:08.573 24296 TRACE oslo_messaging.rpc.dispatcher Stderr: u'@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\nIT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!\r\nSomeone could be eavesdropping on you right now (man-in-the-middle attack)!\r\nIt is also possible that a host key has just been changed.\r\nThe fingerprint for the ECDSA key sent by the remote host is\nc8:0b:98:9d:8d:d8:14:89:e6:fe:97:66:22:4e:57:13.\r\nPlease contact your system administrator.\r\nAdd correct host key in /var/lib/nova/.ssh/known_hosts to get rid of this message.\r\nOffending ECDSA key in /var/lib/nova/.ssh/known_hosts:34\r\nPassword authentication is disabled to avoid man-in-the-middle attacks.\r\nKeyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.\r\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r\n'

分析原因

可能是计算节点的nova用户的host key改变了

对策

对所有计算节点 删除/var/lib/nova/.ssh/known_hosts 有关计算节点的行,重新用nova用户ssh互相登陆,新增host key并确保nova用户计算节点之间ssh都是互信的

ceilometer安装好后,启动服务,状态出错

[root@NFJD-TESTN-COMPUTE-2 ceilometer-2015.1.5]# systemctl restart openstack-ceilometer-compute.service
[root@NFJD-TESTN-COMPUTE-2 ceilometer-2015.1.5]# systemctl status openstack-ceilometer-compute.service
● openstack-ceilometer-compute.service - OpenStack ceilometer compute agent
   Loaded: loaded (/usr/lib/systemd/system/openstack-ceilometer-compute.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2016-12-23 09:39:11 CST; 2s ago
  Process: 8420 ExecStart=/usr/bin/ceilometer-agent-compute --logfile /var/log/ceilometer/compute.log (code=exited, status=1/FAILURE)
 Main PID: 8420 (code=exited, status=1/FAILURE)

Dec 23 09:39:11 NFJD-TESTN-COMPUTE-2 ceilometer-agent-compute[8420]: File "/usr/lib/python2.7/site-packages/ceilometer/cmd/polling.py", line 82, in main_compute
Dec 23 09:39:11 NFJD-TESTN-COMPUTE-2 ceilometer-agent-compute[8420]: service.prepare_service()
Dec 23 09:39:11 NFJD-TESTN-COMPUTE-2 ceilometer-agent-compute[8420]: File "/usr/lib/python2.7/site-packages/ceilometer/service.py", line 117, in prepare_service
Dec 23 09:39:11 NFJD-TESTN-COMPUTE-2 ceilometer-agent-compute[8420]: cfg.CONF(argv[1:], project='ceilometer', validate_default_values=True)
Dec 23 09:39:11 NFJD-TESTN-COMPUTE-2 ceilometer-agent-compute[8420]: File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 1860, in __call__
Dec 23 09:39:11 NFJD-TESTN-COMPUTE-2 ceilometer-agent-compute[8420]: self._namespace._files_permission_denied)
Dec 23 09:39:11 NFJD-TESTN-COMPUTE-2 ceilometer-agent-compute[8420]: oslo_config.cfg.ConfigFilesPermissionDeniedError: Failed to open some config files: /etc/ceilometer/ceilometer.conf
Dec 23 09:39:11 NFJD-TESTN-COMPUTE-2 systemd[1]: openstack-ceilometer-compute.service: main process exited, code=exited, status=1/FAILURE
Dec 23 09:39:11 NFJD-TESTN-COMPUTE-2 systemd[1]: Unit openstack-ceilometer-compute.service entered failed state.
Dec 23 09:39:11 NFJD-TESTN-COMPUTE-2 systemd[1]: openstack-ceilometer-compute.service failed.

conf文件没有权限,修改权限,状态ok。没有权限是因为用root用户从另一台可用的计算节点scp的配置文件。

[root@NFJD-TESTN-COMPUTE-2 ceilometer-2015.1.5]# chown -R ceilometer:ceilometer /etc/ceilometer
[root@NFJD-TESTN-COMPUTE-2 ceilometer]# systemctl restart openstack-ceilometer-compute.service
[root@NFJD-TESTN-COMPUTE-2 ceilometer]# systemctl status openstack-ceilometer-compute.service
● openstack-ceilometer-compute.service - OpenStack ceilometer compute agent
   Loaded: loaded (/usr/lib/systemd/system/openstack-ceilometer-compute.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-12-23 09:42:32 CST; 2s ago
 Main PID: 9459 (ceilometer-agen)
   CGroup: /system.slice/openstack-ceilometer-compute.service
           └─9459 /usr/bin/python /usr/bin/ceilometer-agent-compute --logfile /var/log/ceilometer/compute.log

Dec 23 09:42:32 NFJD-TESTN-COMPUTE-2 systemd[1]: Started OpenStack ceilometer compute agent.
Dec 23 09:42:32 NFJD-TESTN-COMPUTE-2 systemd[1]: Starting OpenStack ceilometer compute agent...

.. note:: 文件权限问题,如配置文件在更换后没有配置文件权限,例如本来是root:nova的文件所有者,被换成了root:root,一定会出现服务无法正常运行的问题。

版本太新导致引用包出错

| fault                                | {"message": "Build of instance 15d9db88-d0a9-40a8-83e9-9ede3001b112 was re-scheduled: 'module' object has no attribute 'to_utf8'", "code": 500, "details": "  File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2258, in _do_build_and_run_instance |


2017-05-25 11:30:34.529 32687 TRACE nova.compute.manager [instance: 15d9db88-d0a9-40a8-83e9-9ede3001b112]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 207, in execute
2017-05-25 11:30:34.529 32687 TRACE nova.compute.manager [instance: 15d9db88-d0a9-40a8-83e9-9ede3001b112]     return processutils.execute(*cmd, **kwargs)
2017-05-25 11:30:34.529 32687 TRACE nova.compute.manager [instance: 15d9db88-d0a9-40a8-83e9-9ede3001b112]   File "/usr/lib64/python2.7/site-packages/oslo_concurrency/processutils.py", line 287, in execute
2017-05-25 11:30:34.529 32687 TRACE nova.compute.manager [instance: 15d9db88-d0a9-40a8-83e9-9ede3001b112]     process_input = encodeutils.to_utf8(process_input)
2017-05-25 11:30:34.529 32687 TRACE nova.compute.manager [instance: 15d9db88-d0a9-40a8-83e9-9ede3001b112] AttributeError: 'module' object has no attribute 'to_utf8'
2017-05-25 11:30:34.529 32687 TRACE nova.compute.manager [instance: 15d9db88-d0a9-40a8-83e9-9ede3001b112]
2017-05-25 11:30:34.531 32687 INFO nova.compute.manager [req-22c24896-bcad-4464-a2ab-c019c45ac0e3 2a2a6d4700034e53a198dac0a71aab6e fe3e50d5fc274d29948ff7fd46b214dc - - -] [instance: 15d9db88-d0a9-40a8-83e9-9ede3001b112] Terminating instance


[root@compute-power ~]# vim /usr/lib64/python2.7/site-packages/oslo_concurrency/processutils.py
284     cwd = kwargs.pop('cwd', None)
285     process_input = kwargs.pop('process_input', None)
286     if process_input is not None:
287         process_input = encodeutils.to_utf8(process_input)


[root@compute-power ~]# pip show oslo.concurrency
Name: oslo.concurrency
Version: 3.20.0
Summary: Oslo Concurrency library
Home-page: http://docs.openstack.org/developer/oslo.concurrency
Author: OpenStack
Author-email: openstack-dev@lists.openstack.org
License: UNKNOWN
Location: /usr/lib64/python2.7/site-packages
Requires: pbr, oslo.config, fasteners, oslo.i18n, oslo.utils, six, enum34


[root@bc36-compute-3 oslo_concurrency]# pip show oslo.concurrency
---
Name: oslo.concurrency
Version: 1.8.2
Location: /usr/lib/python2.7/site-packages
Requires: 
...

oslo.concurrency版本太新,换成老版本解决问题。

vm error

报错信息:

Flavor's disk is too small for requested image. Flavor disk is 16106127360 bytes, image is 21474836480 bytes.].

说明创建vm时所使用的Flavor(云主机类型)的磁盘空间不满足image镜像要求!

No valid host was found

2016-11-01 01:28:38.889 51843 WARNING nova.scheduler.utils [req-9eb2b8ec-216b-4073-95bd-1fbb51844faf 52ba7917bb284af7ad6ac313b7e8e948 0cd3632df93d48d6b2c24c67f70e56b8 - - -] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.

这个问题产生的很大原因有:

  • 计算节点的内存不足、CPU资源不够、硬盘空间资源不足造成的;将云主机类型规格调小点,发现就能创建成功。
  • 网络配置不正确,造成创建虚拟机的时候获取ip失败;网络不通或防火墙引起。
  • openstack-nova-compute服务状态问题。可以尝试重启控制节点的nova相关服务和计算节点的openstack-nova-compute服务;详细检查控制节点和计算节点的nova.conf配置是否有不当配置。
  • 这个报错问题的原因很多,具体要查看/var/log/nova下的日志详细分析。重点是nova-compute.log、nova-conductor.log日志

在nova-scheduler和nova-compute的日志中查看到”ImageNotFound: Image 37aaedc7-6fe6-4fc8-b110-408d166b8e51 could not be found”的报错信息!

root@controller ~]# /etc/init.d/openstack-glance-api status
openstack-glance-api (pid  2222) is running...
[root@controller ~]# /etc/init.d/openstack-glance-registry status
openstack-glance-registry (pid  2694) is running...

状态正常

[root@controller ~]# glance image-list
+--------------------------------------+---------------+-------------+------------------+-------------+--------+
| ID                                   | Name          | Disk Format | Container Format | Size        | Status |
+--------------------------------------+---------------+-------------+------------------+-------------+--------+
| 37aaedc7-6fe6-4fc8-b110-408d166b8e51 | cirrors       | qcow2       | bare             | 13200896    | active |
+--------------------------------------+---------------+-------------+------------------+-------------+--------+

正常工作,尝试upload一个镜像,也能够正常创建vm,原因何在呢??

因为在运维过程中,修改过glance的默认路径由/var/lib/glance/images修改为/data1/glance,并且将/var/lib/glance/images下的镜像都mv至/data1/glance下了,而此时尽管数据已经前已过去了,但是image的元数据信息却牢牢的记录在glance的image_locations表中,查看得知:

mysql> select * from glance.image_locations where image_id='37aaedc7-6fe6-4fc8-b110-408d166b8e51'\G;                                                                        
*************************** 1. row ***************************
        id: 37
  image_id: 37aaedc7-6fe6-4fc8-b110-408d166b8e51
     value: file:///var/lib/glance/images/37aaedc7-6fe6-4fc8-b110-408d166b8e51    #元凶
created_at: 2015-12-21 06:10:24
updated_at: 2015-12-21 06:10:24
deleted_at: NULL
   deleted: 0
 meta_data: {}
    status: active
1 row in set (0.00 sec)

真像:原来原有目录/var/lib/glance/images目录下的镜像都已经mv至/data1/glance下,而数据库中却依旧记录着就的路径内容,从而,衍生的一个问题:当nova尝试启动一台instance的时候,nova会到instance镜像缓存路径,默认/var/lib/nova/_base下查找是否有该镜像,如果没有则向glance发起result api请求,请求下载指定image的镜像到本地,glance则根据数据库中image_locations所定义的值去查找镜像,从而导致失败!

解决方法:更新glance的元数据信息

mysql> update glance.image_locations set value='file:///data1/glance/37aaedc7-6fe6-4fc8-b110-408d166b8e51' where image_id='37aaedc7-6fe6-4fc8-b110-408d166b8e51'\G;             
Query OK, 1 row affected (0.05 sec)
Rows matched: 1  Changed: 1  Warnings: 0

重建虚拟机,故障解决!!!

使用config drive 设定了ip,导致ip冲突不能创建vm

2017-06-22 11:58:50.819 18449 INFO nova.virt.libvirt.driver [req-4edc310f-8382-43ce-a021-8f04c55d5a51 25380a8cde624ae2998224074d4092ee 1cca7b8b0e0d47cb8d6eda06837960ed - - -] [instance: fb28867f-da0f-48f9-b8f8-33704c845117] Using config drive
...
2017-06-22 11:59:16.686 18449 TRACE nova.compute.manager [instance: fb28867f-da0f-48f9-b8f8-33704c845117] FixedIpAlreadyInUse: Fixed IP 172.16.1.4 is already in use.
2017-06-22 11:59:16.686 18449 TRACE nova.compute.manager [instance: fb28867f-da0f-48f9-b8f8-33704c845117]
2017-06-22 11:59:16.690 18449 INFO nova.compute.manager [req-4edc310f-8382-43ce-a021-8f04c55d5a51 25380a8cde624ae2998224074d4092ee 1cca7b8b0e0d47cb8d6eda06837960ed - - -] [instance: fb28867f-da0f-48f9-b8f8-33704c845117] Terminating instance
2017-06-22 11:59:16.698 18449 INFO nova.virt.libvirt.driver [-] [instance: fb28867f-da0f-48f9-b8f8-33704c845117] During wait destroy, instance disappeared.
2017-06-22 11:59:16.708 18449 INFO nova.virt.libvirt.driver [req-4edc310f-8382-43ce-a021-8f04c55d5a51 25380a8cde624ae2998224074d4092ee 1cca7b8b0e0d47cb8d6eda06837960ed - - -] [instance: fb28867f-da0f-48f9-b8f8-33704c845117] Deleting instance files /var/lib/nova/instances/fb28867f-da0f-48f9-b8f8-33704c845117_del
2017-06-22 11:59:16.708 18449 INFO nova.virt.libvirt.driver [req-4edc310f-8382-43ce-a021-8f04c55d5a51 25380a8cde624ae2998224074d4092ee 1cca7b8b0e0d47cb8d6eda06837960ed - - -] [instance: fb28867f-da0f-48f9-b8f8-33704c845117] Deletion of /var/lib/nova/instances/fb28867f-da0f-48f9-b8f8-33704c845117_del complete

挂载卸载卷失败

2017-05-09 14:19:13.957 12778 TRACE oslo_messaging.rpc.dispatcher VolumeDeviceNotFound: Volume device not found at [u'/dev/disk/by-path/ip-10.11.23.132:3260-iscsi-iqn.2000-09.com.fujitsu:storage-system.eternus-dx400:00C0C1P0-lun-78', u'/dev/disk/by-path/ip-10.11.23.133:3260-iscsi-iqn.2000-09.com.fujitsu:storage-system.eternus-dx400:00C1C1P0-lun-78'].

查了下可以正常挂载的节点的路由,发现compute-2节点的路由丢了,

10.11.23.128    10.11.231.1     255.255.255.128 UG    0      0        0 enp17s0f0.607
10.11.231.0     0.0.0.0         255.255.255.0   U     0      0        0 enp17s0f0.607

添加路由之后,可以成功挂载和卸载

[root@NFJD-TESTN-COMPUTE-2 ~]# ip a|grep 607
[root@NFJD-TESTN-COMPUTE-2 ~]# ip link add link enp17s0f0 name enp17s0f0.607 type vlan id 607
[root@NFJD-TESTN-COMPUTE-2 ~]# ip link set up enp17s0f0.607
[root@NFJD-TESTN-COMPUTE-2 ~]# ifconfig enp17s0f0 0.0.0.0
[root@NFJD-TESTN-COMPUTE-2 ~]# ifconfig enp17s0f0.607 10.11.231.26 netmask 255.255.255.0
[root@NFJD-TESTN-COMPUTE-2 ~]# route add -net 10.11.23.128/25 gw 10.11.231.1 dev enp17s0f0.607
[root@NFJD-TESTN-COMPUTE-2 ~]# ip a|grep 607
27: enp17s0f0.607@enp17s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1600 qdisc noqueue state UP 
    inet 10.11.231.26/24 brd 10.11.231.255 scope global enp17s0f0.607

vm挂载卷的时候失败

nova compute节点的nova-compute.log报错如下:

2017-05-09 15:18:01.952 10888 ERROR nova.virt.block_device [req-344ed875-4bc4-40bf-95e3-a0ff48df6059 62f52135115f4898bd0d82c1f0cd632b 6c149dcd3cf64171b8dd972dd03bbac0 - - -] [instance: ddcf7976-b282-45ac-a167-9865178ed629] Driver failed to attach volume 88442395-87d9-41f1-b9d8-0df4b941846a at /dev/vdc
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629] Traceback (most recent call last):
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 276, in attach
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]     device_type=self['device_type'], encryption=encryption)
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1210, in attach_volume
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]     self._connect_volume(connection_info, disk_info)
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1157, in _connect_volume
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]     driver.connect_volume(connection_info, disk_info)
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]     return f(*args, **kwargs)
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 505, in connect_volume
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]     if self.use_multipath:
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 505, in connect_volume
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]     if self.use_multipath:
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]   File "/usr/lib64/python2.7/bdb.py", line 49, in trace_dispatch
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]     return self.dispatch_line(frame)
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]   File "/usr/lib64/python2.7/bdb.py", line 68, in dispatch_line
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]     if self.quitting: raise BdbQuit
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629] BdbQuit
2017-05-09 15:18:01.952 10888 TRACE nova.virt.block_device [instance: ddcf7976-b282-45ac-a167-9865178ed629]

可能谁debug代码的时候加完断点,忘记把断点删除了

去代码中查找,果然如此

删除断点,重启服务,测试可以挂载卷


以上所述就是小编给大家介绍的《Openstack运维常见问题记录》,希望对大家有所帮助,如果大家有任何疑问请给我留言,小编会及时回复大家的。在此也非常感谢大家对 码农网 的支持!

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

Implementing Responsive Design

Implementing Responsive Design

Tim Kadlec / New Riders / 2012-7-31 / GBP 27.99

New devices and platforms emerge daily. Browsers iterate at a remarkable pace. Faced with this volatile landscape we can either struggle for control or we can embrace the inherent flexibility of the w......一起来看看 《Implementing Responsive Design》 这本书的介绍吧!

RGB转16进制工具
RGB转16进制工具

RGB HEX 互转工具

MD5 加密
MD5 加密

MD5 加密工具

正则表达式在线测试
正则表达式在线测试

正则表达式在线测试