NVMe confusion
Posted: Thu Dec 17, 2020 12:06 am
So, I'm working on my NVMe driver again (now that I have more time since the college semester is over, yay!). I'm trying to figure out how to properly send commands and acknowledge responses. I've been reading the NVMe specification -- section 7.2 -- and it describes the general process. For reference:
I'm confused though on the doorbells. The spec notes (in sections 3.1.24 and 3.1.25) that "The difference between the last SQT write and the current SQT write indicates the number of commands added to the Submission Queue" and "The difference between the last CQH write and the current CQH entry pointer write indicates the number of entries that are now available for re-use by the controller in the Completion Queue". Does this mean that I need to calculate the difference between the old SQT and the new SQT and same for the CQHs? I.e.: say I write 6 commands in the admin queue. Would I write 6 to the queue tail doorbell, read all six responses, then write 6 to the queue completion head doorbell? And then if I write another command in, I write 1 to the submission doorbell and 1 to the completion doorbell to acknowledge the response?This section describes command submission and completion processing. Figure 432 shows the steps that are followed to submit and complete a command. The steps are:
1. The host places one or more commands for execution in the next free Submission Queue slot(s) in memory;
2. The host updates the Submission Queue Tail Doorbell register with the new value of the Submission Queue Tail entry pointer. This indicates to the controller that a new command(s) is submitted for processing;
3. The controller transfers the command(s) from in the Submission Queue slot(s) into the controller for future execution. Arbitration is the method used to determine the Submission Queue from which the controller starts processing the next candidate command(s), refer to section 4.13;
4. The controller then proceeds with execution of the next command(s). Commands may complete out of order (the order submitted or started execution);
5. After a command has completed execution, the controller places a completion queue entry in the next free slot in the associated Completion Queue. As part of the completion queue entry, the controller indicates the most recent Submission Queue entry that has been consumed by advancing the Submission Queue Head pointer in the completion entry. Each new completion queue entry has a Phase Tag inverted from the previous entry to indicate to the host that this completion queue entry is a new entry;
6. The controller optionally generates an interrupt to the host to indicate that there is a new completion queue entry to consume and process. In the figure, this is shown as an MSI-X interrupt, however, it could also be a pin-based or MSI interrupt. Note that based on interrupt coalescing settings, an interrupt may or may not be generated for each new completion queue entry;
7. The host consumes and then processes the new completion queue entries in the Completion Queue. This includes taking any actions based on error conditions indicated. The host continues consuming and processing completion queue entries until a previously consumed entry with a Phase Tag inverted from the value of the current completion queue entries is encountered; and
8. The host writes the Completion Queue Head Doorbell register to indicate that the completion queue entry has been consumed. The host may consume many entries before updating the associated Completion Queue Head Doorbell register.