MMIO accesses to the same register in parallel?

[xeyes]

Tested my AHCI code on virtualbox and the port sometimes hangs. When it hangs, the set bit in port CI register never clears and no error bits are getting set.

Was quite a head scratcher until I took a look at the code on the other side.

In virtualbox's DevAHCI.cpp, access to the CI register is handled by the following code (only copied the relavent parts):

Code: Select all

PortCmdIssue_w(...)
{
...
pAhciPort->regCI &= ~uCIValue;
...
}

PortCmdIssue_r(...)
{
...
pAhciPort->regCI &= ~uCIValue;
...
}

Without otherwise serializing MMIO accesses in virtualbox code, these two read-modify-write accesses to regCI could result in data corruption if invoked from different guest CPUs and the time lines up.

Adding serialization in my code for accesses to this register seems to have resolved the hang, sort of proving that there's no general MMIO serialization in virtualbox protecting the 2 functions above.

Now, as in the title, the question is, is it okay to perform MMIO accesses to the same register in parallel?

[nullplan]

It is probably about as great an idea as you would expect a data race to be. Two simultaneous accesses must be atomic or serialized. Otherwise you are always going to have a race condition.

[Octocontrabass]

The hardware doesn't know which CPU is accessing MMIO. If the order of access doesn't matter, you should be able to do it in parallel.

There may be architecture-specific limitations, though. For example, the x86 LOCK prefix tends to not actually lock the bus for MMIO, so you should use some other form of synchronization when performing a read-modify-write.

[Demindiro]

Now, as in the title, the question is, is it okay to perform MMIO accesses to the same register in parallel?

If you use volatile accesses it should be fine (it's target dependent for LLVM, at least). AFAIK it is fine on all major platforms, especially on x86 where regular load/stores are always atomic.

In short:

Now, as in the title, the question is, is it okay to perform MMIO accesses to the same register in parallel?

Yes, provided you use volatile accesses (or maybe atomic, though volatile is the defacto way to do it).

(Of course, like Octocontrabass said, if the order matters you should use some synchronization primitive. Some hardware even provide dedicated bits in MMIO registers for that purpose).

In virtualbox's DevAHCI.cpp, access to the CI register is handled by the following code (only copied the relavent parts):

I'm pretty sure that's a bug and potentially a security hole in VirtualBox if there really is a race condition. I assume that VirtualBox does have some protection against that though.

[xeyes]

The hardware doesn't know which CPU is accessing MMIO. If the order of access doesn't matter, you should be able to do it in parallel.

There may be architecture-specific limitations, though. For example, the x86 LOCK prefix tends to not actually lock the bus for MMIO, so you should use some other form of synchronization when performing a read-modify-write.

Ordering seems okay in my case as data hazards among the ata commands are checked before the threads reach to the point of writing the CI register.

There's no point in RMW-ing CI though? Since the CPU can't clear bits in it.

[xeyes]

Now, as in the title, the question is, is it okay to perform MMIO accesses to the same register in parallel?

If you use volatile accesses it should be fine (it's target dependent for LLVM, at least). AFAIK it is fine on all major platforms, especially on x86 where regular load/stores are always atomic.

In short:

Now, as in the title, the question is, is it okay to perform MMIO accesses to the same register in parallel?

Yes, provided you use volatile accesses (or maybe atomic, though volatile is the defacto way to do it).

(Of course, like Octocontrabass said, if the order matters you should use some synchronization primitive. Some hardware even provide dedicated bits in MMIO registers for that purpose).

Yes I'm using volatile pointers and yes virtualbox is used as a x86 VMM in this case, compiler explorer seems like a really interesting site.

In virtualbox's DevAHCI.cpp, access to the CI register is handled by the following code (only copied the relavent parts):

I'm pretty sure that's a bug and potentially a security hole in VirtualBox if there really is a race condition. I assume that VirtualBox does have some protection against that though.

If parallel accesses are okay to perform, this code does look buggy. I didn't need the lock for other VMMs either. However, mainstream OSes don't seem to have any issue with this though, maybe they don't access this register in parallel?

Hopefully VirtualBox have protections against anyone trying to use this or other race conditions in their device code. That said, virtualbox isn't typically used on servers to run untrusted VMs either.