深入理解nvme协议之一：nvme 协议重点寄存器和数据结构

熟悉nvme 协议是深入理解spdk 代码的基础。而理解协议中的重要寄存器和数据结构是理解协议的基础，下面就一起总结下协议中的重要寄存器和数据结构。

重要寄存器

下面寄存器都是位于pcie memory space, bar[0/1] 空间。

Offset 24h: AQA – Admin Queue Attributes

The Admin Submission Queue’s priority is determined by the arbitration mechanism selected, refer to section 4.11. The Admin Submission Queue and Admin Completion Queue are required to be in physically contiguous memory.

Offset 28h: ASQ – Admin Submission Queue Base Address

This register defines the base memory address of the Admin Submission Queue.

Offset 30h: ACQ – Admin Completion Queue Base Address

This register defines the base memory address of the Admin Completion Queue.

上面queue中的每个entry指向的地址落在controller memroy buffer. 就是下面下面两个寄存器相关的区域。

Offset 38h: CMBLOC – Controller Memory Buffer Location

This optional register defines the location of the Controller Memory Buffer (refer to section 4.7). If CMBSZ is 0, this register is reserved.

Offset 3Ch: CMBSZ – Controller Memory Buffer Size

This optional register defines the size of the Controller Memory Buffer (refer to section 4.7). If the controller does not support the Controller Memory Buffer feature then this register shall be cleared to 0h.

重要数据结构

nvme 重要的数据结构包括submit/completion queue 、scatter/gather list、completion queue entry、controller memroy buffer、name space list、controller list; 命令仲裁机制；

nvme queue

队列特点

总体来说有两个特点：

submit / complete queue 是生产区消费者模型；

queue 以循环队列的数据结构关联；

队列长度

受谁决定：indicated in a 16-bit 0’s based field that indicates the number of slots in the queue

最小长度：2 slots;

IO submit/completion queue最大长度：64K slots;

Admin submit/completion queue最多长度：4K slots;

队列优先级

If the weighted round robin with urgent priority class arbitration mechanism is supported, then host software may assign a queue priority service class of Urgent, High, Medium or Low. If the weighted round robin with urgent priority class arbitration mechanism is not supported, then the priority setting is not used and is ignored by the controller.

队列标识符

创建的时候，需要指定一个16 bit 的标识符；

Submission Queue Entry

每个submit queue entry 指定了一个queue entry的格式，每个entry 64 bytes。重要的字段包括下面几个：

命令类型字段 Dword0

指定发什么类型的命令，以Physical Range page 还是以scatter gather list形式访问数据

namespace identifer 字段

指定上述命令作用的目的区域

数据指针寄存器

指定上述命令中设计到数据的内存中的地址:

PRR entry:

• PRR Entry1: this field contains the first PRP entry for the command or a PRP List pointer depending on the command.
• PRP Entry2: specifies the Page Base Address of the second memory page if the data
  transfer crosses exactly one memory page boundary.

SGL entry: This field contains the first SGL segment for the command

Scatter Gather List (SGL)

这是对非连续分散地址的常见组织方式。

SGL segment

An SGL segment contains one or more SGL descriptors.

SGL descriptor

每个SGL descriptor 的字段如下表所示：

上面不同type的descriptor分类及其含义如下：

上面不同 sub type的descriptor分类及其含义如下：

五种类型的SGL descriptor 的用法如下：

• SGL Data Block descriptor： 描述一个数据块；
• SGL Bit Bucket descriptor：用来忽略部分源数据
• SGL Segment descriptor： 用来描述下一个链接的非末尾SGL segment
• SGL Last Segment descriptor：用来描述下一个末尾SGL segment

SGL 示例：