IDT problems
Posted: Wed Feb 05, 2025 10:18 am
I'm having a bizarre problem with my IDT that is (I suspect, anyway, since I can't find any other reason for this) causing my system to fault out with a runaway instruction pointer when I try to enter user mode (I'm using SYSRET to do so, and I'm 100% certain my IA32_STAR MSR is set correctly because I verified that earlier). This is in 64-bit mode.
It's working fine whilst in the kernel. I load it at startup just after I set the GDT up, which is hardcoded and copied value-wise from an older project that worked. In that project, the IDT also worked fine and did not display this issue (I went back in and checked), and everything up to that point in the process is largely the same aside from the specific values of individual symbols differing due to different sizes.
I know the IDT is working because I write a simple handler that behaves as a catch-all to the tune of
and when I run a very simple test routine that divides by zero I get the expected result:
So I know I have a working IDT, but catch this:
Inexplicably, it's zero. I confirmed this by using SIDT to actually load the value from this register so that I know for a fact it isn't just QEMU trolling me. The IDT gates work fine — perhaps they're cached somewhere and the values are getting cleared out when I switch modes — but I can't for the life of me figure out what is going on here.
Also: my TSS contains ten copies of my kernel stack pointer and is valid as far as I know (I know it's getting used because when I test my scheduler the stack pointer under RSP0 is replaced with a value that's slightly lower and likely corresponds with the shifted frame that actually occurs when my routine returns). All of these (IDT, TSS, and stacks) are located within pages that I've pinned using global pages, though strangely I found that I had to copy them to the new paging tables anyway when I create a new frame for userspace (I verified the mappings using info TLB and they are, in fact, mapped to the correct locations).
I also am sure that some things are definitely not the cause as they were also present in the previous codebase where this did not happen:
Edit to clarify: I verified that the instruction pointer never reaches the code location by editing in two bytes at the pointer to _start (0xEB 0xFE) and noting that it did not, in fact, hang in place but instead still ran away from me.
The register state just before the sysretq instruction (frozen using an inserted hang loop):
The B register contains a pointer to the environment strings (currently an empty list, so the first entry is just a null pointer); the DI register contains argc (1) and the SI register contains a pointer to argv[0] which is itself a pointer to a string constant "TEST.ELF" which is the executable's file name. 0x4000D2 is the address of _start() and I've verified that it points correctly at my crt0.S code (again, the instruction pointer never gets there for some reason).
Here's the imgur with all the screenshots I took of this:
https://imgur.com/a/ifHZGWW
This includes a dump of the TLB as well; the last entry points to the structure I'm using to store frame information, and the virtual mappings from 0x400000 to 0x40D000 are where the executable I'm using to test (or attempt to test, lol) resides in memory. The executable is a variant on "hello world" that executes with stdout tied to the serial driver, so the expected result is that text should show up on my emulator's console in my IDE like all the other serial output does. Anyway, all of this is what led me to conclude that the problem is something related to the IDT, because it's possible that the instruction pointer did briefly enter that loop when I tested it that way only to be interrupted and then die when the interrupt handler turned out to be invalid due to the broken IDTR.
EDIT: I discovered that apparently my paging was only partially set up and that the TLB was deceptive (I hadn't enabled user access on the paging tables themselves which is apparently necessary). That said, the IDT problem still exists, and I still can't seem to get the SYSRETQ to work even with the paging tables set up that way (IRETQ works for a moment but then faults as soon as it gets beyond the first instruction when I remove the hang loop).
It's working fine whilst in the kernel. I load it at startup just after I set the GDT up, which is hardcoded and copied value-wise from an older project that worked. In that project, the IDT also worked fine and did not display this issue (I went back in and checked), and everything up to that point in the process is largely the same aside from the specific values of individual symbols differing due to different sizes.
I know the IDT is working because I write a simple handler that behaves as a catch-all to the tune of
Code: Select all
constexpr static bool has_ecode(uint8_t idx) { return (idx > 0x09 && idx < 0x0F) || idx == 0x11 || idx == 0x15 || idx == 0x1D || idx == 0x1E; }
[[gnu::target("general-regs-only")]] void handler(uint8_t idx, uint64_t ecode)
{
print_text(codes[idx]);
if(has_ecode(idx)) { print_text("("); print_number(ecode; startup_tty.print_text(")"); }
if(errinst) { print_text(" at instruction "); print_number(errinst, 16); }
if(idx == 0x0E)
{
uint64_t fault_addr;
asm volatile("movq %%cr2, %0" : "=a"(fault_addr) :: "memory");
print_text("; page fault address = ");
print_number(fault_addr);
}
while(1);
__builtin_unreachable();
}
So I know I have a working IDT, but catch this:
Inexplicably, it's zero. I confirmed this by using SIDT to actually load the value from this register so that I know for a fact it isn't just QEMU trolling me. The IDT gates work fine — perhaps they're cached somewhere and the values are getting cleared out when I switch modes — but I can't for the life of me figure out what is going on here.
Also: my TSS contains ten copies of my kernel stack pointer and is valid as far as I know (I know it's getting used because when I test my scheduler the stack pointer under RSP0 is replaced with a value that's slightly lower and likely corresponds with the shifted frame that actually occurs when my routine returns). All of these (IDT, TSS, and stacks) are located within pages that I've pinned using global pages, though strangely I found that I had to copy them to the new paging tables anyway when I create a new frame for userspace (I verified the mappings using info TLB and they are, in fact, mapped to the correct locations).
I also am sure that some things are definitely not the cause as they were also present in the previous codebase where this did not happen:
- My interrupt handler is implemented in C++ with extern "C" linkage. All of the IDT gates point to offsets in a function pointer table that call an ISR trampoline, which is generated using a macro. The trampoline pops the error code from the stack if needed (handled using preprocessor shenanigans) and then calls the handler. The handler itself is attributed to save all the general registers, and I set it up where any code that's used in the ISR will only ever use general-purpose registers so as to avoid needing a costly fxsave/fxrstor every time an interrupt fires.
- Said interrupt handler maintains a dynamic array of function pointers that are associated with each IRQ (potentially multiple at a time — the IRQ0/timer calls four separate routines located in various places, and this has caused no problems so far). It also does something similar for general interrupt handling.
- I rolled my own bootloader using UEFI. Yes, hate me if you want, call me a shill, but it ended up being easier than using GRUB because I actually have more control and can easily locate system pointers (XSDT, PCI descriptors, etc).
- My kernel main function does a few things before setting up the GDT: namely, it sets the kerrnel stack to a place that's allocated within the kernel's data/bss segments rather than whatever stack POSIX-UEFI had in the bootloader.
- The bootloader identity-maps roughly a gigabyte of physical memory for the kernel and swaps the paging tables before it does anything else. The identity-mapped memory is only visible in the kernel's paging tables, of course.
- I call _init to invoke the global constructors for several driver objects (some of which need the support of my heap allocator which is initialized early in the kernel main function) after setting up my IDT and GDT.
- My old project played around with SMP and did a little bit of setup in the bootloader by accessing the ACPI tables to find info about the system's processors. It never actually found anything that way (I ended up using the MADT), and any of the code that sets up the APIC and the like executes after the IDT is loaded. When I verified that the IDT was valid in the old version, that code had already executed (most likely) by the time I hit the monitor screen, but I'm unsure how that would affect it.
- The EFER in the old project has the SCE bit clear when I hit the point where I view the IDT register. I don't set the SCE bit until after the IDT is initialized anyway, and the above screenshot is also taken before I set that bit, so that still feels like a dead end. This is also true for the IA32_STAR and related MSRs, which are all populated after the point where I took that screenshot (I inserted a hang-loop right after the IDT initialization code in that instance).
Edit to clarify: I verified that the instruction pointer never reaches the code location by editing in two bytes at the pointer to _start (0xEB 0xFE) and noting that it did not, in fact, hang in place but instead still ran away from me.
The register state just before the sysretq instruction (frozen using an inserted hang loop):
The B register contains a pointer to the environment strings (currently an empty list, so the first entry is just a null pointer); the DI register contains argc (1) and the SI register contains a pointer to argv[0] which is itself a pointer to a string constant "TEST.ELF" which is the executable's file name. 0x4000D2 is the address of _start() and I've verified that it points correctly at my crt0.S code (again, the instruction pointer never gets there for some reason).
Here's the imgur with all the screenshots I took of this:
https://imgur.com/a/ifHZGWW
This includes a dump of the TLB as well; the last entry points to the structure I'm using to store frame information, and the virtual mappings from 0x400000 to 0x40D000 are where the executable I'm using to test (or attempt to test, lol) resides in memory. The executable is a variant on "hello world" that executes with stdout tied to the serial driver, so the expected result is that text should show up on my emulator's console in my IDE like all the other serial output does. Anyway, all of this is what led me to conclude that the problem is something related to the IDT, because it's possible that the instruction pointer did briefly enter that loop when I tested it that way only to be interrupted and then die when the interrupt handler turned out to be invalid due to the broken IDTR.
EDIT: I discovered that apparently my paging was only partially set up and that the TLB was deceptive (I hadn't enabled user access on the paging tables themselves which is apparently necessary). That said, the IDT problem still exists, and I still can't seem to get the SYSRETQ to work even with the paging tables set up that way (IRETQ works for a moment but then faults as soon as it gets beyond the first instruction when I remove the hang loop).
