OSDev.org

Hey guys, I was wondering if anyone here unit tests their code aggressively? Like, with GCOV and the works?

I've been pondering about how to unit test the bootloader itself, even the bootsector and was curious as to whether anyone else had done something similar.

~K

gcov doesn't do unit testing or any testing for that matter, nor does it create any tests. It lets you collect code coverage info.

I've written a test suite for my FAT FS code (and run it with gcov to see what's not covered by my test cases).
I've written a test for a memory manager as well (no coverage testing).
I've once tested the hell out of a red-black tree implementation (got 100% or nearly 100% code coverage).
...
For about 6 years I was paid to write tests for low-level stuff... It's an interesting experience.

Writing tests only ever makes sense when you believe you've found a bug. If the code is so complicated that it would require aggressive testing, you should probably use a different method to assure quality, such as formal verification.

Love4Boobies wrote:Writing tests only ever makes sense when you believe you've found a bug.

Nope. It serves equally well to notice bugs the instance they are created.

Combuster wrote:

Love4Boobies wrote:Writing tests only ever makes sense when you believe you've found a bug.

Nope. It serves equally well to notice bugs the instance they are created.

Sure but you can't know whether your code is correct or not in advance. You're left with three choices:

• Test everything, thus wasting valuable development time.
• Test arbitrarily. I doubt anyone would agree this is a good method.
• Only test the code that you suspect is wrong, which is what I suggested.

There are methods such as test-driven development where you explicitly never write code unless you have tests that confirms its existence and correctness. This way you quickly grow a large amount of tests and you may even gain more correctness through the simple virtue of writing your code such that it can be tested. However, I get the feeling that unit tests and bootloaders aren't perfect matches. Bootloaders are usually simple enough in principle, but they have to deal and be robust against BIOS bugs and older hardware platforms and it's not trivial to test such cases.

I don't think anyone sane would advocate test-driven development. Any experienced developer will tell you that most of the tests they've ever written turned out to be useless. TDD will only make this worse. Besides, a lot of the time, you don't even know what problem you're solving until you've started working on a solution, in which case TDD can result in a premature specification or in discarded tests. The justification I most often hear is that TDD is useful because the tests will ensure everything still holds together whenever there is an architecture or design change---but why not instead write the tests whenever such situations occur?

I found TDD is good for library / framework development. It's also sometime good for game development, like to ensure a certain random logic generate correct statistic outcome. Other than that, I mostly write unit tests when I'm tired

I said anyone sane. Anyway, correctness and packaging are orthogonal issues and I don't think game development is a particularly special type of development.

Back on to the original question - I have written unit tests for the more self-contained but tricky parts of my codebase (e.g. libc's printf/scanf code), and module tests for the TCP handling.

Other parts are left to be tested by running code that uses it (e.g. a LARGE part of my loading, graphics, and IPC code is tested by running the GUI).

Agressive unit testing is very hard for core OS services, because they usually either interact with hardware/firmware (which is very hard to simulate), or relies on so many other components in the system, as to make doing unit tests a waste of time. The far better solution is to write programs that ensure that the system behaves as expected, and run these programs on whatever hardware configuration you can find.

thepowersgang wrote:Agressive unit testing is very hard for core OS services, because they usually either interact with hardware/firmware (which is very hard to simulate), or relies on so many other components in the system, as to make doing unit tests a waste of time. The far better solution is to write programs that ensure that the system behaves as expected, and run these programs on whatever hardware configuration you can find.

The above can be a little misleading. I don't think he is suggesting integration testing doesn't serve a point.

Hi,

I wrote this code:
I stubbed out "getchar()" and unit tested it. My unit tests have 100% code coverage, therefore there are no bugs in this function.


Cheers,

Brendan

Brendan wrote: My unit tests have 100% code coverage, therefore there are no bugs in this function.

That may work for this function, but usually 100% coverage is either meaningless (say, we didn't pay attention to coverage of all paths through our code and only focused on basic block coverage or we failed to test some functionality, even though exercised the relevant code (e.g. didn't distinguish passed from failed)) or unreachable (because complexity doesn't give us a chance to test everything in any practical amount of time, add to that code being poorly written from the testability point of view). Coverage can tell you where you clearly failed to test. It can't guaranty correctness. So, I take it, a smiley was lost when writing that statement of yours. Was it not?

Brendan example was obviously meant to point out that as long as a test doesn't feed "U" into the system it doesn't find the crash.

Also got a great example. I've got a function that takes a char * as input and that does a FFT on it. Tested using a vector of 64 zeroes and it works 100%, with 100% test coverage.

I call it memset.