Page 1 of 1
MMIO accesses to the same register in parallel?
Posted: Tue Jul 12, 2022 1:48 am
by xeyes
Tested my AHCI code on virtualbox and the port sometimes hangs. When it hangs, the set bit in port CI register never clears and no error bits are getting set.
Was quite a head scratcher until I took a look at the code on the other side.
In virtualbox's DevAHCI.cpp, access to the CI register is handled by the following code (only copied the relavent parts):
Code: Select all
PortCmdIssue_w(...)
{
...
pAhciPort->regCI &= ~uCIValue;
...
}
PortCmdIssue_r(...)
{
...
pAhciPort->regCI &= ~uCIValue;
...
}
Without otherwise serializing MMIO accesses in virtualbox code, these two read-modify-write accesses to regCI could result in data corruption if invoked from different guest CPUs and the time lines up.
Adding serialization in my code for accesses to this register seems to have resolved the hang, sort of proving that there's no general MMIO serialization in virtualbox protecting the 2 functions above.
Now, as in the title, the question is, is it okay to perform MMIO accesses to the same register in parallel?
Re: MMIO accesses to the same register in parallel?
Posted: Tue Jul 12, 2022 8:51 am
by nullplan
It is probably about as great an idea as you would expect a data race to be. Two simultaneous accesses must be atomic or serialized. Otherwise you are always going to have a race condition.
Re: MMIO accesses to the same register in parallel?
Posted: Tue Jul 12, 2022 12:05 pm
by Octocontrabass
The hardware doesn't know which CPU is accessing MMIO. If the order of access doesn't matter, you should be able to do it in parallel.
There may be architecture-specific limitations, though. For example, the x86 LOCK prefix tends to not actually lock the bus for MMIO, so you should use some other form of synchronization when performing a read-modify-write.
Re: MMIO accesses to the same register in parallel?
Posted: Sat Jul 16, 2022 1:54 pm
by Demindiro
xeyes wrote:
Now, as in the title, the question is, is it okay to perform MMIO accesses to the same register in parallel?
If you use volatile accesses it
should be fine (
it's target dependent for LLVM, at least). AFAIK it is fine on all major platforms,
especially on x86 where regular load/stores are always atomic.
In short:
xeyes wrote:
Now, as in the title, the question is, is it okay to perform MMIO accesses to the same register in parallel?
Yes, provided you use volatile accesses (or maybe atomic, though volatile is the defacto way to do it).
(Of course, like Octocontrabass said, if the order matters you should use some synchronization primitive. Some hardware even provide dedicated bits in MMIO registers for that purpose).
xeyes wrote:
In virtualbox's DevAHCI.cpp, access to the CI register is handled by the following code (only copied the relavent parts):
I'm pretty sure that's a bug and potentially a security hole in VirtualBox if there really is a race condition. I assume that VirtualBox does have some protection against that though.
Re: MMIO accesses to the same register in parallel?
Posted: Sun Jul 17, 2022 2:41 am
by xeyes
Octocontrabass wrote:The hardware doesn't know which CPU is accessing MMIO. If the order of access doesn't matter, you should be able to do it in parallel.
There may be architecture-specific limitations, though. For example, the x86 LOCK prefix tends to not actually lock the bus for MMIO, so you should use some other form of synchronization when performing a read-modify-write.
Ordering seems okay in my case as data hazards among the ata commands are checked before the threads reach to the point of writing the CI register.
There's no point in RMW-ing CI though? Since the CPU can't clear bits in it.
Re: MMIO accesses to the same register in parallel?
Posted: Sun Jul 17, 2022 2:45 am
by xeyes
Demindiro wrote:xeyes wrote:
Now, as in the title, the question is, is it okay to perform MMIO accesses to the same register in parallel?
If you use volatile accesses it
should be fine (
it's target dependent for LLVM, at least). AFAIK it is fine on all major platforms,
especially on x86 where regular load/stores are always atomic.
In short:
xeyes wrote:
Now, as in the title, the question is, is it okay to perform MMIO accesses to the same register in parallel?
Yes, provided you use volatile accesses (or maybe atomic, though volatile is the defacto way to do it).
(Of course, like Octocontrabass said, if the order matters you should use some synchronization primitive. Some hardware even provide dedicated bits in MMIO registers for that purpose).
Yes I'm using volatile pointers and yes virtualbox is used as a x86 VMM in this case, compiler explorer seems like a really interesting site.
Demindiro wrote:
xeyes wrote:
In virtualbox's DevAHCI.cpp, access to the CI register is handled by the following code (only copied the relavent parts):
I'm pretty sure that's a bug and potentially a security hole in VirtualBox if there really is a race condition. I assume that VirtualBox does have some protection against that though.
If parallel accesses are okay to perform, this code does look buggy. I didn't need the lock for other VMMs either. However, mainstream OSes don't seem to have any issue with this though, maybe they don't access this register in parallel?
Hopefully VirtualBox have protections against anyone trying to use this or other race conditions in their device code. That said, virtualbox isn't typically used on servers to run untrusted VMs either.