OSDev.org

I'm interested in debugging userland software running inside of my OS. I've managed to connect GDB to Qemu, but it's very tedious to debug userland programs because it requires me to halt the emulator while while the particular process I want to debug is mapped in virtual memory and I can't set userland breakpoints via GDB that target a specific process because it breaks on any userland process that hits that virtual address.

I'm suspecting that I'll need to implement something akin to the GDB remote protocol in my OS and somehow wire up GDB on the host OS to talk to the guest OS (my OS) in QEMU via wiring stdin/stdout to the virtual COM1 port?

Has anyone done something similar? How did you implement userland debugging?

For normal debugging, you can implement the INT3 (Debug exception) or a system call, then the kernel will write the debug info in COM ports.

For faults like GPF or Page faults, you can tell the user about them when the int handler is called.

You can also show the registers of the task, because you save them before calling the handler.

I haven't actually tried this, but....

How about defining a dummy global in the program you are debugging and set it to some magic number. Then set your breakpoints as conditional breakpoints when that variable equals the magic number. The chances of that triggering the breakpoint in a different program must be pretty close to zero.

It looks straight foward to set up to 4 break points via the x86 debug registers. I just need to make sure to set it on context switching.

IIUC, if I set the trap flag before `iretting` back into user space, it'll execute a single instruction and it'll generate an INT 1?

The steps required are:
a) Build a gdb stub.
b) The process I'm wanting to debug.
c) Can connect outside of QEMU to my host via GDB.
d) Give the stub the ability to manipulate registers and memory, set break points, single step, etc.

So:
a) I think this will be the most work implementing all of the instructions of the GDB protocol.
b) Maybe when the debugger connects I can prompt in my OS "which process do you want to connect to?"
c) My OS can communicate to the host via COM1. QEMU has a parameter where it can redirect the guest serial port to TCP, which matches what QEMU's inbuilt gdbserver does.
d) Making a set of system calls for this sounds straight forward.

Basically, you need to implement the debugging interface that your debugger requires. I use the OpenWatcom debugger, but it is customized with a few new messages I want, mainly that I want only the debugged thread to be stopped, not the whole OS.

For single stepping, you should use the TF flag. It will cause int 1. You also need to support hardware & software breakpoints. What you can do here depends on your processor, but x86 have support for breakpoints on memory references and similar.

I think you also needs a new "thread state" that indicates a thread is stopped because it is debugged. I use a debug list for this.

If you have shared code, your scheduler needs to load "local" breakpoints as it switches tasks.

I don't find COM ports practical for communication between target & debugger. I prefer to use the TCP/IP interface.

rdos wrote:Basically, you need to implement the debugging interface that your debugger requires.

Building my own GDBstub doesn't actually seem that difficult (see this journey of someone embedded a GDBstub into a Gameboy emulator). The protocol is a bit overwhelming but then they say just a few basic commands are needed.

rdos wrote:It will cause int 1. You also need to support hardware & software breakpoints.

I am on x86. Is there a reason to support both? Other than hardware breakpoints are limited to the 4 debug registers.

(AFAIK, software breakpoints simply replace the instructions with a predictable command such as INT3 / 0xCC?)

rdos wrote: I want only the debugged thread to be stopped, not the whole OS.

Debugging the kernel is easy because QEMU has a built in gdbstub I can connect to. For debugging user programs in my microkernel, I prefer that my gdbstub be a service (e.g. a "GDB Debug Server") but it leads to the problem that I can't debug my Debug Server. (Unless I have multiple debug servers running? )

If it was in the kernel, I can debug the debug code (via QEMU's gdbstub) but it requires my kernel knowing how to talk to the transport protocol, I don't have access to the C++ UI library (so I can draw a window that says "Hey GDB just connected. Which process would you like to launch or connect it to?"), which stops my microkernel from being very generic. (My own self imposed limitation.)

AndrewAPrice wrote:

rdos wrote:Basically, you need to implement the debugging interface that your debugger requires.

Building my own GDBstub doesn't actually seem that difficult (see this journey of someone embedded a GDBstub into a Gameboy emulator). The protocol is a bit overwhelming but then they say just a few basic commands are needed.

It's the same complexity as the Watcom interface. The protocol is not that hard to support. However, you also need a debug framework. In my experience, the debug framework has similar complexity as the debug protocol.

AndrewAPrice wrote:

rdos wrote:It will cause int 1. You also need to support hardware & software breakpoints.

I am on x86. Is there a reason to support both? Other than hardware breakpoints are limited to the 4 debug registers.

I think you should support both. You also need to prioritize which one to use. Some things, like data breakpoints can only be done with hardware breakpoints, while simple stops are easier to do with software (planting int 3 in the code).

Supporting breakpoints, handling single step & exceptions is part of the debug framework you need to implement, You also need a method to read & write registers in a debugged thread. If you don't save registers in the thread block, rather keep them on the stack, this will get more complicated to do.

AndrewAPrice wrote:

rdos wrote: I want only the debugged thread to be stopped, not the whole OS.

Debugging the kernel is easy because QEMU has a built in gdbstub I can connect to. For debugging user programs in my microkernel, I prefer that my gdbstub be a service (e.g. a "GDB Debug Server") but it leads to the problem that I can't debug my Debug Server. (Unless I have multiple debug servers running? )

If it was in the kernel, I can debug the debug code (via QEMU's gdbstub) but it requires my kernel knowing how to talk to the transport protocol, I don't have access to the C++ UI library (so I can draw a window that says "Hey GDB just connected. Which process would you like to launch or connect it to?"), which stops my microkernel from being very generic. (My own self imposed limitation.)

Generally, the debug server will run in it's own process. That means it must be able to read & write memory in other processes (and, of course, register state). In my design, I can single step into kernel with the application debugger. This is possible since the debugger has addons for understanding device drivers, and because single step & hardware breakpoints work in kernel space too. This is part of the reason I only stop threads that are debugged.

As for attaching the debugger to a process, I can do this both with ordinary programs and file system servers. Basically, the debugger is given the name of the process executable, and then the debug server looks for a process with that name, and if it finds it, attaches the debugger to it. If it is not running, it loads it from disc. This, of course, only works if a single instance of each program runs.

Also, for a serious project, you should not depend on running your OS in an emulator. Debugging should work on real hardware too, including debugging kernel on real hardware.

rdos wrote:Also, for a serious project, you should not depend on running your OS in an emulator. Debugging should work on real hardware too, including debugging kernel on real hardware.

One day! For now, just a hobby. The flexibility of implementing the "Debug Server" as a standalone process means I can just support serial port now (for ease of piping it to the host via QEMU) and easily change the transport channel to TCP later when my network stack is developed.

AndrewAPrice wrote:

rdos wrote:Also, for a serious project, you should not depend on running your OS in an emulator. Debugging should work on real hardware too, including debugging kernel on real hardware.

One day! For now, just a hobby. The flexibility of implementing the "Debug Server" as a standalone process means I can just support serial port now (for ease of piping it to the host via QEMU) and easily change the transport channel to TCP later when my network stack is developed.

Certainly. Just remember that basically no modern computer has real serial ports today, and the network adapters that QEMU supports are obsolete and writing drivers for them is (mostly) a waste of time.

Actually, moving to real hardware has become increasingly complex. Keyboards use USB (PS/2 is much easier to support), no real serial ports, so you need to use USB to serial converters, portables use Wifi network chips that lack documentation, you need to handle EFI booting and VBE is on it's way out in favor of complex video chip programming. I'd say even my mature OS would have problems running on modern portable computers, mostly because of lack of support for Wifi network chips and USB-based network adapters.

rdos wrote:Certainly. Just remember that basically no modern computer has real serial ports today, and the network adapters that QEMU supports are obsolete and writing drivers for them is (mostly) a waste of time.

Qemu has emulated an Intel PRO/1000-series NIC for a very long time, and it was made the default several years ago. Drivers for this line of NICs are forward-compatible to Intel's most recent 2Gbit and 10Gbit products with little more than PCI ID whitelists.

Also, serial ports are back in vogue with a lot of ARM platforms, if you go that route.

klange wrote:

rdos wrote:Certainly. Just remember that basically no modern computer has real serial ports today, and the network adapters that QEMU supports are obsolete and writing drivers for them is (mostly) a waste of time.

Qemu has emulated an Intel PRO/1000-series NIC for a very long time, and it was made the default several years ago. Drivers for this line of NICs are forward-compatible to Intel's most recent 2Gbit and 10Gbit products with little more than PCI ID whitelists.

I have my doubts about that. I used to have an Intel network driver, but it turned out not to be forward compatible. Also, the most common network chips typically are from RTL.

klange wrote: Also, serial ports are back in vogue with a lot of ARM platforms, if you go that route.

You can get them for embedded type x86 systems too, but on mainstream computers it's very rare.