kernel debugger

[yemista]

What are the basic steps for writing an internal kernel debugger? If anyone could point me to some documentation Id much appreciate it as I could not find any myself. Since I dont really know much about how it would function, my specs might be a little off, but basically, I want something that is compiled into the kernel, and contains function calls that are given virtual addresses, and the debugger will stop execution at that address. Upon hitting a breakpoint, you have the option to dump the registers, step to the next instruction, delete a breakpoint, or continue. Of course this would mean that at first the debugger would only be implemented at compile time, such as calling debug_set(addr), but thats all I am really looking for at the moment. Maybe if I ever get a shell youd be able to do it for command line, but for now I am just thinking about a basic debugger.

[mathematician]

To set a break point you replace the first byte of the instruction where you want execution to halt with a 0cch byte. When the processor hits that byte an int 3 is generated. Obviously you have to set that byte back to what it was before executing that instruction.

Setting the single stepping flag in the flags register will cause an int 1 to be generated after each and every instruction has been executed.

You can also use the debugging registers to set break points, but I have never used them.

So after you have hit a break point, replace the 0cch byte with what was there, set the single stepping flag, and return from int 3. After one instruction has been executed you will get an int 1, and then you can reset the break point, if you want to, before clearing the sigle stepping flag, and returning from int 1.


EDIT:
As an after thought I will add that to set break points you really need to implement a disassembler as part of the debugger; otherwise it would be difficult to know exactly where to put down the 0cch byte.

[alethiophile]

In case it helps, Linux will dump registers/memory when you type certain keystrokes at a text-mode command line.

[manonthemoon]

I suggest looking at this: http://mirror.href.com/thestarman/asm/debug/debug2.htm.

MS-DOS included a program called "debug", which had various commands for viewing and editing the system, setting breakpoints, etc. You might want to model your debugger after DOS DEBUG. It's very simple but powerful. Unfortunately, the link I posted only shows how to use DOS DEBUG (there's no source code, sorry!)

I just recently finished my kernel debugger, and its based heavily on DOS DEBUG.

Your bare-minimum debugger should have commands to view the registers (pushed on the stack after an exception) and display the contents of a given memory address.

Personally, I doubt breakpoints would be useful--I never use them. But if you think you need them, go for it. But first write code to display the registers and contents of a memory address.

EDIT: Just to clarify, DOS DEBUG was a separate program and was used to write and test small programs. I know you want an internal kernel debugger. I'm just pointing out a good place to start if you want to design a debugger. Specifically, look at the D, E, R, and G commands of DEBUG.