OSDev.org

Hi all,
I am developing a kernel for x86_64 architecture in rust and had written an NVMe driver quite a while ago which proved to work fine until when I recently decided to implement a proper memory frame & page allocation system (I previously used an allocator which just incremented the frame and page indices), as well as rewriting my PCI driver.

Since then, like the title suggests, any writes to the MMIO BAR for NVMe devices fail to go through. Important to add is that the MMIO BAR is populated with values that aren't garbage. It's likely that this is also the case for other PCI devices but I currently have no other drivers for any of them.

To be very specific this happens when I try to clear the CC.EN bit in the NVMe registers (but also other values that can be modified whenever CC.EN is set, which makes me think this isn't any issue regarding the NVMe driver itself).

I have checked pretty much everything I can think of and still haven't found the cause of this issue. I have used ChatGPT to debug the relevant code, but unfortunately to no success.
Here is a list of things I have checked:

• Memory allocation: This does not seem to be the problem, using the same implementation for non MMIO memory works as intended, I have also checked relevant page table flags and the Present, Writable, No cache are all set.

• PCI: BAR size is read and original value restored properly, IO & Memory Spaces are enabled along with Bus Master. The MMIO itself seems to have proper values when first read (version 1.4.0 & controller status 1 in NVMe registers).

• I am using volatile reads & writes everywhere regarding MMIO

• Initializing NVMe is currently the first thing done after initializing GDT, memory allocation and the kernel heap (interrupts are currently not yet implemented and are completely disabled except for exceptions), so it would seem highly unlikely anything is messing with it beforehand.

All of this is happening in a QEMU vm.
Any help is greatly appreciated.

If more context is needed by sharing certain pieces of code I will happily provide .

h3tR wrote: ↑Mon Apr 21, 2025 6:32 pmI have used ChatGPT to debug the relevant code, but unfortunately to no success.

Don't waste your time with fancy autocomplete.

h3tR wrote: ↑Mon Apr 21, 2025 6:32 pmI have also checked relevant page table flags and the Present, Writable, No cache are all set.

Page-level caching on x86 is a lot more complicated than it looks. The "no cache" bit is only one bit of a three-bit index into the PAT MSR that determines the actual memory type that's selected by your page table entry, and that memory type is combined with the memory type selected by the MTRRs to determine the effective memory type. The effective memory type may not be what you expect.

h3tR wrote: ↑Mon Apr 21, 2025 6:32 pminterrupts are currently not yet implemented and are completely disabled except for exceptions

You'll probably want your NVMe driver to use interrupts in the future. Don't put too much effort into getting it working without interrupts when you'll have to rewrite it later.

h3tR wrote: ↑Mon Apr 21, 2025 6:32 pmAll of this is happening in a QEMU vm.

Try using QEMU's trace log (-trace "pci_nvme_*") to see what it thinks you're doing.

I've found the issue by using this.

Octocontrabass wrote: ↑Mon Apr 21, 2025 7:25 pm Try using QEMU's trace log (-trace "pci_nvme_*") to see what it thinks you're doing.

I was using a struct to represent the registries after reading them and I was doing writing back the entire struct into mmio rather than the specific values I needed to modify. I'm not 100% what specifically caused the writes to not go through but it was either the compiler ordering the writes of the struct values in a weird way causing them to be misaligned or it could've been attempting to write to reserved and read-only registers that caused the NVMe device to tweak out.