XD/NX reserved bit on pages in the PML4 from the APs

[YogurtDonor]

Hey folks, my kernel is slowly coming together and I've been lurking on OSDev for a few months and the site and forums have been an invaluable resource. I've just spent the last few weeks struggling to get my APs to initialize into long mode, and I think my last hurdle may be worth signposting for others that follow. I don't have wiki write access so I'll document it here for now.

If like me you went for a UEFI only kernel, I didn't go through the real -> protected -> long sequence initially. Instead I started in long mode already and just needed to set my own PML4.

While doing this, I started off by utilizing the NX/XD bit on all my pages except those I tag as executable. UEFI must enable the XD bit in the EFER for you because I didn't.

The fun comes when you attempt to bring up the APs, go through all the dance, then set the PML4 you know works only to discover your code is executing fine, but your stack is not!

My clue came from the error code on the page fault only to discover that the value was 0xB. One of the bits indicates that a reserved bit is set. After checking against the reserved bits, it wasn't set and I was thoroughly confused.

After much head scratching and rubber duck programming with ChatGPT, it kept telling me that bit 63 was reserved, and I told it back that this is where the NX/XD flag is set. It replied that this bit is only valid when NX/XD is enabled in the EFER.

I think it's worth signposting this in a few places:

• In the section on Page Fault error codes I would include a note in both the row for Reserved write, and the the list of conditions under which a page fault occurs.
• On the SMP page I would link to the page fault error codes section saying there is more info here if you are struggling with page faults.
• Finally a note on the Setting Up Long Mode about how the NX/XD bit may set by UEFI but it won't be set on the APs, with a link to the above, and perhaps a bit where you manipulate the EFER in the section on switching from protected mode.

Now onto my next adventure - AHCI!

[Octocontrabass]

I don't have wiki write access

Now that your account is approved you should be able to get access.

UEFI must enable the XD bit in the EFER for you

There's no guarantee UEFI will do that. In fact, some CPUs don't support NX in the first place (although they might be too old for UEFI). You should always detect and enable the CPU features you want to use before you try to use them. You may even encounter PCs where the BSP and APs don't all support the same features.

[nullplan]

There's no guarantee UEFI will do that. In fact, some CPUs don't support NX in the first place (although they might be too old for UEFI).

But are there 64-bit CPUs that don't have NX? Personally, I assume NX is present. But if there are 64-bit CPUs without NX, then I will add something to refuse to run on such systems.

You may even encounter PCs where the BSP and APs don't all support the same features.

But then it is no longer symmetric multi-processing. My kernel isn't ready for that, either. If this legitimately is a possibility, then I probably need to verify that the AP supports all the same features as the BSP (at least the ones I'm using), and if not, send the AP back to sleep.

[Octocontrabass]

But are there 64-bit CPUs that don't have NX?

As far as I can tell, yes. There are also 64-bit CPUs with NX disabled via the MISC_ENABLE MSR, but at least that's something you can work around.

If this legitimately is a possibility,

I don't think it's possible to mix x86 CPUs with different capabilities, but I have seen some firmware disable MONITOR/MWAIT on only the BSP and not the APs. (I think that's also MISC_ENABLE, actually.)

[JAAman]

If this legitimately is a possibility,

I don't think it's possible to mix x86 CPUs with different capabilities, but I have seen some firmware disable MONITOR/MWAIT on only the BSP and not the APs. (I think that's also MISC_ENABLE, actually.)

yes, it is possible

in Intel's earliest P/E hybrid CPUs, the P cores support AVX512 but the E cores don't (to simplify the windows schedular implementation, AVX512 was disabled in the P cores, but it is physically there, and there is no technical reason it had to be disabled -- it was only disabled because it makes thread migration more complicated)

there is no technical reason why the P and E cores must have the same capabilities, only policy reasons have prevented it so far

[Octocontrabass]

Intel's boot blobs for Alder Lake didn't allow AVX512 to be enabled at the same time as E-cores, so you couldn't get a mixture of capabilities there. You're right, though, that doesn't prevent any future x86 CPUs from allowing such combinations.