OSDev.org

Hey all, I am writing a specialized OS for $COMPANY, and I have written remote GDB support for this project. My problem is that GDB seems to have a hard time understanding the way back into the guest stacks. We can assume amd64 for now, and I am using IST. The CPU exceptions are in CPL=3, so no privilege change is happening.
As you can probably guess, I am trying to pretend the exception handler is a function call by setting rbp to the stack pointer before the exception happens. I am building my guest program with -O0 -g, which will create stack frames even in leaf functions. The OUT instruction is what I'm using to trap out to handle the exception, and is not relevant to the GDB remote debugging. The stack I'm seeing in GDB isn't completely broken, but it only goes one frame back for the most part, and sometimes it skips the function that is causing an exception. In my test cases I am just using the UD2 instruction to generate exceptions and inspecting with GDB to see what's going on.
Seen above is backtrace before executing UD2. There is no problems with remote GDB before the CPU exception.

What do you think I am doing wrong here? Assuming no stack smashing, is it possible to get decent backtraces in programs?

For now I solved it another way, although I'm not happy with it. Remote GDB will now report the Return-RIP and Return-RSP when in an exception handler, which makes GDB happy again. It's annoying because now I need another way to report that an exception happened, possiby by sending the correct trap signal to GDB when executing the instruction, but at least the backtrace is useful again.