Hey all, I am writing a specialized OS for $COMPANY, and I have written remote GDB support for this project. My problem is that GDB seems to have a hard time understanding the way back into the guest stacks. We can assume amd64 for now, and I am using IST. The CPU exceptions are in CPL=3, so no privilege change is happening.
Code: Select all
%macro CPU_EXCEPT 1
ALIGN 0x10
push rbp
mov rbp, QWORD [rsp+32]
push rdx
add rsp, 16 ;; skip rdx, rbp
mov dx, 0xFF00 + %1
out dx, ax
sub rsp, 16
pop rdx
pop rbp
iretq
%endmacro
As you can probably guess, I am trying to pretend the exception handler is a function call by setting rbp to the stack pointer before the exception happens. I am building my guest program with -O0 -g, which will create stack frames even in leaf functions. The OUT instruction is what I'm using to trap out to handle the exception, and is not relevant to the GDB remote debugging. The stack I'm seeing in GDB isn't completely broken, but it only goes one frame back for the most part, and sometimes it skips the function that is causing an exception. In my test cases I am just using the UD2 instruction to generate exceptions and inspecting with GDB to see what's going on.
Code: Select all
Breakpoint 2, test_ud2 () at musl.c:21
(gdb) bt
#0 test_ud2 () at musl.c:21
#1 0x0000000000401321 in main (argc=2, argv=0x1ffcc8) at musl.c:45
Seen above is backtrace before executing UD2. There is no problems with remote GDB before the CPU exception.
What do you think I am doing wrong here? Assuming no stack smashing, is it possible to get decent backtraces in programs?