OSDev.org

Hello everyone,
I'm also writing an OS using assembly language and C, I'm now working on booting process and have finished the transfer from assembly codes to C codes. But I'm confronted with an annoying problem about debugging. I debug my codes with bochs and GDB, but the GDB doesn't work as I expected. The situation is like the below.
The control enters the C codes through a call instruction from assembly codes and If I issue a "backtrace" command in C function call, the GDB only prints the innermost frame, that is frame 0, and another frame which I can't figure out which one it is, because it only prints a random address (I can't figure out where this address locates) instead of a function or label name, no matter how deep the call stack actually is. And this isn't over.
If I try to exam local variable issuing "print XXX" command, the GDB just prints a garbage value!. So I can't exam the executing status efficiently during the debugging. Luckily, If I issue "print &xxx" command, the GDB can response a correct local variable address, and I can exam the value using "exam" command, but this is not viable as the amount of codes grow larger later.
During this situation, the assembly codes and C codes are all executed in real mode. If it would help, the following is my compiling options and link script.

gcc -g -Wall -mregparm=3 -march=i386 -m32 -mpreferred-stack-boundary=2 -fomit-frame-pointer -ffreestanding -fno-toplevel-reorder -fno-strict-aliasing -fno-stack-protector -include code16gcc.h -nostdinc -Ixxx

SECTIONS {
. = 0x0
.text {*(.text)}
.rodata{*(.rodata)}
.data{*(.data)}
.eh_frame : {*(.eh_frame)}
}

the file code16.gcc.h contains only one statement below
asm(".code16gcc");
Yes, it's the same with the file in boot directory of the Linux kernel.

berkus wrote:GDB expects linear 32-bit address space, not segmented 16 bit.

Second, I believe GCC cannot emit 16 bit real mode code, and your way of doing it is utterly wrong (i.e. you will not get what you expect to get as a result of this assembly).

Maybe I explain wrong. Actually, my booting codes are similar with the codes in the boot directory of the linux kernel. I found it C codes executed in real mode. I wonder how the linux kernel developers debug the booting codes with GDB.

@ berkus:

Indeed Linux (in arch/x86/boot/code16gcc.h) uses an "experimental" feature in binutils to generate 16bit code. (Experimental but around since 2001...)

@ harvey:

Try "set arch i8086".

I've read in some 2009 mailing list archives about a GDB patch somewhere that adds "set i386 real-mode on". Check if it has been merged into GDB, and if not, see if you can track down that patch.

Generally speaking, I can understand why the GDB maintainers don't bother much about 16bit Real Mode. The demand isn't very high, is it?

Solar wrote: @ harvey:

Try "set arch i8086".

I've read in some 2009 mailing list archives about a GDB patch somewhere that adds "set i386 real-mode on". Check if it has been merged into GDB, and if not, see if you can track down that patch.

Generally speaking, I can understand why the GDB maintainers don't bother much about 16bit Real Mode. The demand isn't very high, is it?

Yeah, I've tried "set arch i8086", but it seems not change anything.

I'll try the patch you mentioned. Yes, I know the demand isn't high. I just wonder How the linux kernel developers debug the booting codes which are executed in real mode and include both assembly and C. It's really way inefficient for me to debug such a large amount of C codes.

BTW, how do you debug the real mode codes? Take me for example. I debug codes (assembly + C, real mode and protected mode) using bochs with gdb-stub + GDB, although it's not efficient to debug C codes executed in real mode.

You don't actually need GDB to debug software. There are numerous other ways to do it.

gerryg400 wrote:You don't actually need GDB to debug software. There are numerous other ways to do it.

like what? I know ddd or kgdb, but they are based on gdb. And bochs internal debugger can only debug assembly codes as far as I know.

harvey wrote:

gerryg400 wrote:You don't actually need GDB to debug software. There are numerous other ways to do it.

like what? I know ddd or kgdb, but they are based on gdb. And bochs internal debugger can only debug assembly codes as far as I know.

You don't actually need a debugger to debug software. There are numerous other ways to do it.

Some examples are

1. Printing to the screen.
2 Asserting
3. Logging.
4. Code review (or stare-down debugging).
5. In-code unit testing.
6. Screen-of-death printing.

I don't use a debugger in my kernel. I get very few bugs where a debugger would be helpful. Most bugs I get these days are multicore timing or locking issues.

I suspect that the guys that wrote the Linux boot code used printf debugging.

Hi,

The problem with GDB is that it's designed for applications; where (for 80x86) the entire application is either 32-bit or 64-bit. This means it probably can't handle 16-bit 80x86 code at all, and it's incapable of handling mode switches (e.g. from 16-bit to 32-bit, from 32-bit to 64-bit, etc).

gerryg400 wrote:Some examples are

Doh - you beat me to it. I was too busy reading (and agreeing with) this guy's mini-rant on the lost art of desk checking.


Cheers,

Brendan

gerryg400 wrote:I don't use a debugger in my kernel. I get very few bugs where a debugger would be helpful. Most bugs I get these days are multicore timing or locking issues.

Probably more because you are not used to such a function than that you don't need it. How about if you write some complex device-driver (lets say an AHCI-driver), and it malfunctions. Maybe your memory-mapped lists are corrupt, or there are some kind of race-conditions. When you use the kprintf concept, you will be swamped with printouts if you would log everything, and most likely you would not log important tings related to the bug. The printouts might also alter the program flow so the bug "disappears". A kernel debugger is clearly superior here. However, the most important aspect is the ability to single-step the code in the debugger and make sure it does what you expect, and to make it a rule to always do this before accepting that the code is correct. This can never be superseeded with kprintf! You would simply run the code, and if it appeared to work, you accept it is correct, regardless if it also happens to write somewhere it is not supposed to, lock something it is not supposed to or whatever.

Solar wrote:You don't actually need a debugger to debug software. There are numerous other ways to do it.

Those are solutions I only resort to when real debugging doesn't work. I used "printf"-debugging as I looked for errors in the debug-trap-files simply because I couldn't setup source-level debugging in the environment. It took ten times as long to find the error because of this.

printf() debugging is a skill, just like step debugging is. You can do it right, or you can do it wrong. I don't think one is inherently superior to the other.

rdos wrote:...you will be swamped with printouts if you would log everything, and most likely you would not log important tings related to the bug...

That's part of the skill: Seeing which information should be logged at which level.

The printouts might also alter the program flow so the bug "disappears".

Another part of the skill involved. Your log statements should not have side effects, and your code should be stable enough not to react to stuff like compiler-introduced reorderings (a.k.a. "if I enable optimization my code breaks").

However, the most important aspect is the ability to single-step the code in the debugger and make sure it does what you expect, and to make it a rule to always do this before accepting that the code is correct.

I.e., playing manual unit test driver...

This can never be superseeded with kprintf!

Oh jolly well it can! Actually you don't even have to print anything, just check each submodule for its contract... however, that's Unit Testing, which is yet another crucial debugging technique. (Or rather, a development technique so that you don't have to do (as much) debugging.)

You would simply run the code, and if it appeared to work, you accept it is correct, regardless if it also happens to write somewhere it is not supposed to, lock something it is not supposed to or whatever.

You know, I have a maxime which I call "DNA": Do Not Assume. It's meant for coding practice, but in this case, I think it should also be applied to other developer's coding practice. Just because gerryg400 perfers printf() debugging over step debugging doesn't mean you should accuse him of shoddy workmanship. I very much doubt any of us has ever done a complete code coverage on a non-trivial work, manually or automatically.

I have worked in environments where step-debugging would just have had be swamped in unintelligible numbers, while a couple of well-placed printf()s (or rather, cout's in this case) quickly uncovered the bug. And I have worked in environments where step-debugging was a gift from heaven. (Actually, that was the same environment, just a different module of the product...) Chose the tool most appropriate for the problem.

rdos wrote:

gerryg400 wrote:I don't use a debugger in my kernel. I get very few bugs where a debugger would be helpful. Most bugs I get these days are multicore timing or locking issues.

Probably more because you are not used to such a function than that you don't need it. How about if you write some complex device-driver (lets say an AHCI-driver), and it malfunctions. Maybe your memory-mapped lists are corrupt, or there are some kind of race-conditions. When you use the kprintf concept, you will be swamped with printouts if you would log everything, and most likely you would not log important tings related to the bug. The printouts might also alter the program flow so the bug "disappears". A kernel debugger is clearly superior here. However, the most important aspect is the ability to single-step the code in the debugger and make sure it does what you expect, and to make it a rule to always do this before accepting that the code is correct. This can never be superseeded with kprintf! You would simply run the code, and if it appeared to work, you accept it is correct, regardless if it also happens to write somewhere it is not supposed to, lock something it is not supposed to or whatever.

I haven't written an AHCI driver and maybe when I get there, that will cause me to write an kernel debugger. Actually it won't be a kernel debugger because my drivers are in userspace.

I guess it wouldn't be very difficult to integrate remote GDB and have it work over a serial port but I haven't needed it for a while and there are other priorities.

At one point a few years ago I used TurboDebugger to debug the 32bit version of my OS over a serial port. It worked fine but when I switched to GCC I could no longer use it.

Solar wrote:printf() debugging is a skill, just like step debugging is. You can do it right, or you can do it wrong. I don't think one is inherently superior to the other.

Right.

Solar wrote:Another part of the skill involved. Your log statements should not have side effects, and your code should be stable enough not to react to stuff like compiler-introduced reorderings (a.k.a. "if I enable optimization my code breaks").

If the code debugged depends on timing, any insertion of code of whatever triviality might change the behavior. Printf is just a major distrurber because it takes time to execute.

Solar wrote:Oh jolly well it can! Actually you don't even have to print anything, just check each submodule for its contract... however, that's Unit Testing, which is yet another crucial debugging technique. (Or rather, a development technique so that you don't have to do (as much) debugging.)

It is a little hard to do unit testing for hardware-device-drivers meant for an OS. I mean, you just cannot plug-in the code in some sophisticated testing tools based on Windows or Linux, nor can you easily make it a standalone application for Windows or Linux.