questions about AHCI/SATA

[xeyes]

Hello everyone, have some questions about AHCI/SATA that I didn't find good answers for from the AHCI, SATA and ATA spec drafts.

1. Wiki's ATA article says "On some drives it is necessary to "manually" flush the hardware write cache after every write command.", is this still the case for SATA drives?

2. What are the differences between DMA and PIO cmds? Doesn't data always move inside data FISes on the sata bus and are DMAed to/from system memory by the HBA? There might be minor differences like PIO vs. DMA setup FISes but are there good reasons to use one vs. the other?

3. What should the direction (command header DW0 bit 6) be for cmds that don't move data between the device and system memory such as 0xE7? Is there a spec with these details?

4. Are the bits in port command issue (PxCI) register ORed together with the value written by the CPU? Or can the CPU clear bits there as well by writing 0?

5. I assume that the HBA doesn't detect data hazard among different cmds in different slots and the software is supposed to keep track of that?

If that's true, how should software enforce the ordering?

If the software always sets 1 bit at a time in CI (in quick succession without waiting for the the set bits to clear), would the order of completion be the same and the cmd whose bit is set earlier always completes earlier?

What if the software sets multiple bits in CI with a single one write? Does the HBA have any preference or the commands could complete in any order? There's a priority field in queued cmds but it's either high or low and seems to be quite coarse grained for a total of 32 slots?

Thanks!

[Octocontrabass]

1. Wiki's ATA article says "On some drives it is necessary to "manually" flush the hardware write cache after every write command.", is this still the case for SATA drives?

As far as I'm aware, it's not necessary for any drives. You do need to flush the write cache before turning the power off, though. I suspect the author of that statement forgot this step. (The flush may be performed as part of setting the drive to a low-power mode. You'll have to look at the various ATA and ACS specifications for details.)

2. What are the differences between DMA and PIO cmds? Doesn't data always move inside data FISes on the sata bus and are DMAed to/from system memory by the HBA? There might be minor differences like PIO vs. DMA setup FISes but are there good reasons to use one vs. the other?

PIO has worse error handling and may not be able to transfer more than one sector per command. The AHCI specification explains it in more detail, but in short you should always prefer DMA over PIO.

3. What should the direction (command header DW0 bit 6) be for cmds that don't move data between the device and system memory such as 0xE7? Is there a spec with these details?

According to the AHCI spec, it should only be set for writes, so use 0 for commands that are not writes.

4. Are the bits in port command issue (PxCI) register ORed together with the value written by the CPU? Or can the CPU clear bits there as well by writing 0?

According to the AHCI spec, writing a 1 sets the associated bit and writing a 0 does nothing. The AHCI spec additionally mentions that you should only set PxCI bits you want to change - do not write a 1 to a bit that is already set.

5. I assume that the HBA doesn't detect data hazard among different cmds in different slots and the software is supposed to keep track of that?

Correct.

If that's true, how should software enforce the ordering?

That depends on the commands you're using.

If the software always sets 1 bit at a time in CI (in quick succession without waiting for the the set bits to clear), would the order of completion be the same and the cmd whose bit is set earlier always completes earlier?

According to the AHCI spec, commands are typically posted in the order they're issued, and most commands complete in the order they're issued. The exceptions are port multipliers, which may cause commands to be issued in a different order, and NCQ commands, which may complete in a different order. Read the AHCI spec for details.

What if the software sets multiple bits in CI with a single one write? Does the HBA have any preference or the commands could complete in any order?

According to the AHCI spec, if you issue multiple commands with a single write, they may be posted in any order.

I notice a lot of your questions are answered by the AHCI specification...

[xeyes]

Thanks for taking the time to reply, but could you share a link to "the AHCI specification" if it is publicly available?

I was looking at "Serial ATA (AHCI) 1.3.1 specification" as can be obtained from https://www.intel.com/content/www/us/en ... 1-3-1.html and it seems that we are talking about two very (if not completely) different specs.

For example:

According to the AHCI spec, it should only be set for writes, so use 0 for commands that are not writes.

Write (W): When set, indicates that the direction is a device write (data from system memory to
device). When cleared, indicates that the direction is a device read (data from device to system
memory).

But 0xE7 or flush isn't meant to move data from device to system memory AFAIK.

According to the AHCI spec, writing a 1 sets the associated bit and writing a 0 does nothing.

Commands Issued (CI): This field is bit significant. Each bit corresponds to a
command slot, where bit 0 corresponds to command slot 0. This field is set by software
to indicate to the HBA that a command has been built in system memory for a
command slot and may be sent to the device. When the HBA receives a FIS which
clears the BSY, DRQ, and ERR bits for the command, it clears the corresponding bit in
this register for that command slot. Bits in this field shall only be set to ‘1’ by software
when PxCMD.ST is set to ‘1’.
This field is also cleared when PxCMD.ST is written from a ‘1’ to a ‘0’ by software.

Which doesn't seem to mention anything about the "side effects" of writing either 1 or 0.


Same for most other points

[Octocontrabass]

Write (W): When set, indicates that the direction is a device write (data from system memory to
device). When cleared, indicates that the direction is a device read (data from device to system
memory).

Hm, I must have missed that when I was looking at it yesterday. Maybe try it both ways and see if QEMU yells at you? You'll also tell the HBA to transfer 0 bytes of data, so I wouldn't be surprised if all existing HBAs just ignore that bit for non-data commands.

Which doesn't seem to mention anything about the "side effects" of writing either 1 or 0.

That's covered in section 5.5.1:

Software shall set PxCI.CI(pFreeSlot) to indicate to the HBA that a command is active. Software should only write new bits to set to ‘1’; the previous register content of PxCI should not be re-written in the register write.

Same for most other points

PIO vs DMA, section 1.7:

All transfers are performed using DMA. The use of the PIO command type is strongly discouraged. PIO has limited support for errors – for example, the ending status field of a PIO transfer is given to the HBA during the PIO Setup FIS, before the data is transferred. However, some commands may only be performed via PIO commands (such as IDENTIFY DEVICE). Some HBA implementations may limit PIO support to one DRQ block of data per command.

Setting and clearing bits by writing PxCI, section 1.5:

RW1 - Read/Write ‘1’ to set

Command issue order, section 5.3.3.3 and section 9.3.2:

The command selected shall be the oldest command yet to be issued (i.e. the command that had its PxCI bit set first). If multiple commands were issued on the same write to PxCI, the HBA may select the command to issue arbitrarily from amongst those commands when they are the oldest.

If multiple commands are issued to the same device by software in a single write of the PxCI register, the transmission order amongst that set of commands is arbitrary. To ensure a particular order of issue for a set of commands, the commands shall be issued by software in distinct writes of the PxCI register. There is no command ordering requirements among commands being issued to separate Port Multiplier ports.

Command completion order... isn't actually defined by the AHCI spec that I can see, it's probably in the SATA spec.

[Ethin]

Can't seem to find any mention of command completion in the SATA specification. I know the ACS-4 specification definitely doesn't define it, so if its not defined by either AHCI, SATA, or any of the ATA specifications, its most likely arbitrary. I suppose this is another reason to use NVMe over SATA...

[Octocontrabass]

Can't seem to find any mention of command completion in the SATA specification. I know the ACS-4 specification definitely doesn't define it, so if its not defined by either AHCI, SATA, or any of the ATA specifications, its most likely arbitrary. I suppose this is another reason to use NVMe over SATA...

The behavior is defined, just not necessarily in such clear terms. AHCI very clearly specifies the HBA state machine, and this state machine will be unable to issue more than one command at a time unless the command is one of the queued commands in the NCQ feature set.

Similarly, it's not possible to issue commands in parallel on IDE except the queued commands in the TCQ feature set. An IDE HBA won't stop you from trying - it just won't work.

[xeyes]

Hm, I must have missed that when I was looking at it yesterday. Maybe try it both ways and see if QEMU yells at you? You'll also tell the HBA to transfer 0 bytes of data, so I wouldn't be surprised if all existing HBAs just ignore that bit for non-data commands.

Flush seems to work bothways (got no errors, but can't tell whether the virtual disk was flushed or not), you are probably right that this is a don't care bit. The specs just forgot to mention it.

Thanks for pointing out the other section numbers, I guess we were looking at the same spec afterall and I need to get better at searching.

There's a new mystery, both qemu and virtualbox sets bit 0 of the error field/register in TFD upon a port reset, while not setting ERROR bit of the status register there. Seemingly violating ATA spec that says

...shall set the ERROR bit to one if any bit in the ERROR field is set to one. Otherwise, ... shall clear the ERROR bit to zero.

It is unlikely that both qemu and virtualbox have the same issue though.

The behavior is defined, just not necessarily in such clear terms. AHCI very clearly specifies the HBA state machine, and this state machine will be unable to issue more than one command at a time unless the command is one of the queued commands in the NCQ feature set.

Is the following understanding correct? For normal DMA cmds like 0x25 and 0x35, completion order is always the same as issue order which is in turn the same as the order of setting the bits in CI?

I get that queued DMA cmds like 0x60 and 0x61 won't always complete in the same order as issued.

[xeyes]

Can't seem to find any mention of command completion in the SATA specification. I know the ACS-4 specification definitely doesn't define it, so if its not defined by either AHCI, SATA, or any of the ATA specifications, its most likely arbitrary.

Totally agreed that the specs could have been made easier to follow, especially when there are so many cross references among them.

I suppose this is another reason to use NVMe over SATA...

This is very risky, if I destroy some SATA controllers or drives that 10 year old, I won't be too sad. Can't say the same for the new toys though.

Judging from the number of AHCI quirkinesses in various emulators (the most interesting one is simnow, which doesn't seem to be able to fire any normal operation interrupts other than the DPS one, making any non-data cmds unusable with interrupts ), even though NVME seems to be better defined, I doubt it is worth the hassle right now.

[Octocontrabass]

There's a new mystery, both qemu and virtualbox sets bit 0 of the error field/register in TFD upon a port reset, while not setting ERROR bit of the status register there.

That's normal. ATA drives automatically perform the EXECUTE DEVICE DIAGNOSTIC command when reset, and those values indicate diagnostics passed.

Is the following understanding correct? For normal DMA cmds like 0x25 and 0x35, completion order is always the same as issue order which is in turn the same as the order of setting the bits in CI?

For a single drive, yes.

When a port multiplier is involved, it works the same as single drives attached to separate ports. There's no overall order guarantee, but each individual drive will still complete its commands in the order they were posted to PxCI.

[xeyes]

There's a new mystery, both qemu and virtualbox sets bit 0 of the error field/register in TFD upon a port reset, while not setting ERROR bit of the status register there.

That's normal. ATA drives automatically perform the EXECUTE DEVICE DIAGNOSTIC command when reset, and those values indicate diagnostics passed.

Is the following understanding correct? For normal DMA cmds like 0x25 and 0x35, completion order is always the same as issue order which is in turn the same as the order of setting the bits in CI?

For a single drive, yes.

When a port multiplier is involved, it works the same as single drives attached to separate ports. There's no overall order guarantee, but each individual drive will still complete its commands in the order they were posted to PxCI.

Thanks again for the clarifications!