Weird Triple Fault on MMIO read in VirtualBox

[ComputerFido]

Hey,

For some reason, I have been getting a strange issue where my VirtualBox triple faults whenever I read from MMIO I have mapped into virtual memory from a PCI BAR. I have checked the VirtualBox logs and the memory address for the device (XHCI controller) is correct, also I do not have this issue on QEMU/KVM. I have mapped the address into virtual memory with the flags write and present if that helps. The RIP in the register dump in the logs points to where I read from the MMIO.

VirtualBox Logs:
https://pastebin.com/d0Dijg8i

Thanks,
ComputerFido

[iansjack]

Using the built-in debugger, set breakpoints in your PF and GPF exception handlers. This will help you track exactly where the fault is and, if a PF, what memory access is causing it.

[ComputerFido]

Thanks iansjack,
After stepping over the instruction which caused the fault it just went straight to a triple fault. The virtual address was correct and everything, VirtualBox just went straight to a triple fault and didn't attempt to call my exception handler or anything. I also just tested it on real hardware and it also works fine.

[linuxyne]

Probably here?

[ComputerFido]

Probably here?

Yes that line is responsible, however it should be fine as it is essentially reading the first byte of the mapped memory. The instruction comes down to

Code: Select all

movzbl (%rax),%eax

The register dump in the VirtualBox logs shows that rax contains the correct virtual memory address (0xffffffffc007f000) and the PCI BAR lines up with the address in the logs, so it should not be an issue.

[iansjack]

There are only so many things that can be going wrong here:

1. Stack corruption, preventing interrupts and exception handlers being called.

2. IDT corruption, with the same effect.

3. Page table corruption, again fairly disastrous.

I would set a breakpoint just before the crash point then check each of these very carefully. I should also have added, put a break point on the double-fault handler, but it sounds like no exception handlers are being called at all. This points to some severe corruption of one of the above data structures. Often this is a result of uninitialized variables - some emulators and hardware set these to 0, some don't.

Edit: I suspect that these lines in the log are the important ones:

Code: Select all

00:00:46.004768 Guest GDT (GCAddr=ffffffff80104000 limit=37):
00:00:46.004771 0000 - read error rc=VERR_PAGE_TABLE_NOT_PRESENT GCAddr=ffffffff80104000
00:00:46.004774 0008 - read error rc=VERR_PAGE_TABLE_NOT_PRESENT GCAddr=ffffffff80104008
00:00:46.004775 0010 - read error rc=VERR_PAGE_TABLE_NOT_PRESENT GCAddr=ffffffff80104010
00:00:46.004777 0018 - read error rc=VERR_PAGE_TABLE_NOT_PRESENT GCAddr=ffffffff80104018
00:00:46.004778 0020 - read error rc=VERR_PAGE_TABLE_NOT_PRESENT GCAddr=ffffffff80104020
00:00:46.004779 0028 - read error rc=VERR_PAGE_TABLE_NOT_PRESENT GCAddr=ffffffff80104028
00:00:46.004780 0030 - read error rc=VERR_PAGE_TABLE_NOT_PRESENT GCAddr=ffffffff80104030

Edit2: Just one further thought - you did select "64-bit operating system" when creating the virtual machine?

[linuxyne]

Yes that line is responsible, however it should be fine

Let me rephrase. Are you sure that you want

Code: Select all

opRegs = (xhci_op_regs_t*)xhciVirtualAddress + capRegs->capLength;

and not

Code: Select all

opRegs = (xhci_op_regs_t*)(xhciVirtualAddress + capRegs->capLength);

?

Edit: Apologies. I get it now. The crash occurs even before the system has a chance to dereference opRegs. The crash occurs trying to dereference capRegs.

[ComputerFido]

Thanks,
I will take a look to see if anything gets corrupted, although it seems unlikely as the rest of my OS runs fine w/o that memory access. With the logs, as far as I know that is just VirtualBox attempting to dump the GDT of the guest OS, and I do have 64-bit selected.

[ComputerFido]

So unfortunately, the VirtualBox debugger wouldn't let me examine memory (VERR_PAGE_TABLE_NOT_PRESENT), so I was unable to reliably examine my stack, IDT and page tables. I am now using the Interrupt Stack Table for my double fault handler, to no effect, and I looked over my IDT code and everything seems fine, corruption of the IDT also seems extremely unlikely as all my interrupts (timer/scheduler, syscalls, ATA, page fault handler, etc.) all trigger and return fine. With paging, I tried to simply map the bottom 4Gb of address space with four 1gb pages as it seems to me unlikely to cause any paging errors, however all of these measures were to no avail.

Thanks,
ComputerFido

[Octocontrabass]

So unfortunately, the VirtualBox debugger wouldn't let me examine memory (VERR_PAGE_TABLE_NOT_PRESENT)

That's because your page tables were corrupted. You'll have to examine memory shortly before the corruption happens in order to catch what's causing it.

[ComputerFido]

So unfortunately, the VirtualBox debugger wouldn't let me examine memory (VERR_PAGE_TABLE_NOT_PRESENT)

That's because your page tables were corrupted. You'll have to examine memory shortly before the corruption happens in order to catch what's causing it.

As far as I know it is just the VirtualBox debugger as no matter when and where in the kernel I attempt to examine memory it throws that error, however it works fine for me to examine memory in my userspace processes.

Thanks,
ComputerFido