A quick unit testing question!

[elderK]

Neat answers. Thanks.

As for unit testing being worthwhile, I've find it incredibly so in my own work in recent years. Mostly when writing libraries. It does add a lot of time, as Boobies said, because I've taken the "test everything" approach.

I.e. Test the usual ways of utilizing some function or interface. Then feed it tons of bad data. Then ensure that you can trigger allocation failures at will, so you can test how your code reacts to that situation, etc.

I figure a potential middle ground would be to aggressively test components deemed critical and have some kind of percentage of important-ness for modules or other programs. The higher the percentage, the more aggressively tested it is.

I.e. If it's got a super high important-ness percentage, it'd be write-the-tests-first, test-everything approach.

As for say, testing a bootloader. I see that you could achieve that. But the abstraction is making it possible could be annoying. I.e. hiding BIOS invocation behind some procedure and when you're testing it, you'd link the loader to something else which provides a test stub for that invocation procedure.

Sorry, rambling. Stream of consciousness, yadda yadda.
~K

Edit:
While being able to provide a more substantial, provable, kind of trust in what you create (without reason, of course), I've found the real benefit of aggressively unit testing to be how it's influenced how I engineer what I create. There's a saying somewhere I'm sure that says: If it's really, really hard to test, it's probably designed badly.

I figure that's more about testing interfaces and such than the situation in which a given program is run. Still, with techniques such as dependency injection and such, you have a pretty massive amount of flexibility with which to test!

[alexfru]

Brendan example was obviously meant to point out that as long as a test doesn't feed "U" into the system it doesn't find the crash.

We haven't been told anything about use cases or constraints. The keyboard could have only numeric keys on it. Or it could produce non-ASCII character codes (perfectly allowed by the C standard, btw). Which brings us to another important point... If you don't know what's right and what's wrong or if you can't guaranty the right/desired behavior all the time or 80% of the time, you have a problem, you can't really test that which you don't know.

[Love4Boobies]

As for unit testing being worthwhile, I've find it incredibly so in my own work in recent years. Mostly when writing libraries. It does add a lot of time, as Boobies said, because I've taken the "test everything" approach.

I've asked this before: What is so special about libraries? Code is code; unit testing is equally useful or useless regardless of how you package your code.

Anyway, just so I'm not misunderstood: I don't think testing is not worthwhile. I think it's definitely worthwhile for non-critical code (for which better quality assurance methods exist, as I've already mentioned) but only for reported bugs, so they can be better understood and so as to have some certainty that they are truly gone after the attempt to fix them---and since the test has already been written, it can be kept in order to "make sure" that the same bugs are never reintroduced. Otherwise, writing them will severely handicap productivity without producing palpable results. Imagine you're emplying test-driven development and spending 50% of your time writing tests (more is not uncommon) and 5% of your tests catch bugs (I'm being very generous)---this means that, on average, 47.5% of your time goes out the window. Now imagine instead using that precious time for further development. Meanwhile (in parallel), the testing could be cheaply performed by testers and perhaps even users reporting bugs. Now you can use your developers to do proper development. There's a good reason why all the industry big players look down on this sort of thing.

I.e. If it's got a super high important-ness percentage, it'd be write-the-tests-first, test-everything approach.

If it's so critical, why not use formal verification instead?

[bluemoon]

What is so special about libraries?

Modules within a libraries usually can be isolated, it's relatively easy to write test for them.

Also, without sufficient test (at a sane level, I'm not talking about infinite test cases), you put your user into test pilots.

[Love4Boobies]

What is so special about libraries?

Modules within a libraries usually can be isolated, it's relatively easy to write test for them.

All modules, not just classes and/or routines in libraries, should strive to be loosely copuled.

Also, without sufficient test (at a sane level, I'm not talking about infinite test cases), you put your user into test pilots.

Aren't you a little perv? Anyway, what was this in answer to?

[bluemoon]

Aren't you a little perv? Anyway, what was this in answer to?

Forget all about these, perhaps I used an too extreme case. I'm a bit stressed due to the things happens here.

[Brendan]

Hi,

Brendan example was obviously meant to point out that as long as a test doesn't feed "U" into the system it doesn't find the crash.

Yes. For completeness:

Code: Select all

    int foo(int a) {
        char b;

        b = getchar();
        if(b != 0xAA) {
            b = b ^ 0x55;
        }
        return a/b;
    }

There's about 4 things wrong with this code:

• An overflow problem in "b = getchar();" (trying to put an int into a char)
• Failing to check if "getchar()" returned EOF
• If "char" is signed, then the condition "(b != 0xAA)" is nonsense (b could never be equal to 0xAA)
• A division by zero if "getchar()" returned 0x55 or 'U'

Of these, unit tests with 100% code coverage may detect none of the problems. Mostly, unit tests involve "unknown unknowns" - you don't know if there are bugs, so you make random guesses about which bugs to test for and hope you get lucky maybe; and while "100% code coverage" sounds impressive the only thing it's really good for is self delusion ("I believe it's right because it passes the tests" where that belief is misplaced).

I'm not saying unit tests are bad (they are a good idea in general); but perhaps what we should be doing is measuring coverage in terms of values tested. Rather than writing as little as 2 tests and saying we've got 100% code coverage; maybe we should say that there's 2**32 possible values for the input variable 'a' and 257 possible values returned by "getChar()", and that we've tested 2 of the 1103806595072 possible combinations.

Also note that a compiler or static analyser that tracks the potential ranges of values in variables would have detected most of the problems without the programmer doing anything extra. However, most of the problems are perfectly legal in C, so tools that detect these problems aren't practical for C (the tests would report far too many "false positives"). Of course this isn't a problem with C alone - most languages suck for the same reason.



Cheers,

Brendan

[Love4Boobies]

• An overflow problem in "b = getchar();" (trying to put an int into a char)
• Failing to check if "getchar()" returned EOF

These two are essentially the same problem, since getchar returns int so as to also account for EOF.

I'm not saying unit tests are bad (they are a good idea in general); but perhaps what we should be doing is measuring coverage in terms of values tested. Rather than writing as little as 2 tests and saying we've got 100% code coverage; maybe we should say that there's 2**32 possible values for the input variable 'a' and 257 possible values returned by "getChar()", and that we've tested 2 of the 1103806595072 possible combinations.

I agree with your point but the problem is in fact worse. C is an abstract machine and its semantics can vary from implementation to implementation (i.e., the number of bits in a byte; the number of bytes used to represent one type or another; how signed numbers are represented; other aspects of representation, like padding, endianness, and alignment; etc.---just to mention a few problems regarding types). So even if you managed to check all possible code paths for a particular C implementation using a supercomputer, you still won't know whether your code is correct. Perhaps changing the ABI will cause your code to break. In fact, tests are inherently limited in this regard since the problem is provably undecidable. Pessimism aside, one might have some luck checking a program against a particular ABI by using an automated test generation tool able to create tests suited for the code.

Also note that a compiler or static analyser that tracks the potential ranges of values in variables would have detected most of the problems without the programmer doing anything extra. However, most of the problems are perfectly legal in C, so tools that detect these problems aren't practical for C (the tests would report far too many "false positives"). Of course this isn't a problem with C alone - most languages suck for the same reason.

And, again, there's the problem of inherently undetectable problems. You either ignore these or introduce even more false positives.

[Thomas]

Hi,

You dont generally consider code coverage as a important matric with unit tests and that's not the objective of unit tests in the first place!!. It is fairly easy to get good coverage with unit tests. When you are building something - its important to ensure that you build on top a strong foundation, unit test help in this regard. You typically run converage with an intstrumented binary where functional testers execute their test cases ( ie typically 0 coding). Now code coverage becomes an important metric, because it to some extent shows the effectiveness of the tests.

--Thomas

[elderK]

I think this has got just a tiny little bit off topic, in a sense, as it wasn't meant to be at all about libraries versus monolithic programs.

I chose libraries there simply because I develop more of them than I do anything else. As for formal verification, aye. I'll admit that I haven't done nearly as much reading on that topic as I should have!

In the end, I guess it all comes down to the age old wisdom of "everything in moderation."

Write tests when it makes sense, whether that be in reaction to some discovered bug to make sure it never occurs again or to help check out particularly tricky corner cases, whatever.

As for code coverage, I do find it useful if only to know that my unit tests really are exercising the code as much as I expect. I feel that without having some proof of what code a given test exercises, you can't really be sure that your unit test actually exercises all paths of control to adequately test whatever.

Man. That was badly written. Alas.
~K

[Candy]

The only thing code coverage actually tells you is what code you didn't hit with a test. That may be untested code, it may be dead code or it may be the code causing crashes. In any case, you should ask yourself if you want to keep that code - but typically, the answer is yes.

[FrankRay78]

I'm currently experimenting with low-level unit testing, here's an example of dependency injection in C:

Code: Select all

typedef struct {
    void (*Show)();
    void (*Hide)();
    void (*SetPosition)(int x, int y, int width);
} Cursor;

Real implementation, which can only be run with direct access to the hardware port:

Code: Select all

void cursor_show()
{
    asm("outb %b0, %w1" :: "a"(0x0A), "d"(VGA_ADDRESS_REGISTER));
    asm("outb %b0, %w1" :: "a"(0x00), "d"(VGA_DATA_REGISTER));
}

void cursor_hide()
{
    asm("outb %b0, %w1" :: "a"(0x0A), "d"(VGA_ADDRESS_REGISTER));
    asm("outb %b0, %w1" :: "a"(0x20), "d"(VGA_DATA_REGISTER));
}

void cursor_setposition(int x, int y, int width)
{
    uint16_t pos = y * width + x;

    asm("outb %b0, %w1" :: "a"(0x0F), "d"(VGA_ADDRESS_REGISTER));
    asm("outb %b0, %w1" :: "a"((uint8_t)(pos & 0xFF)), "d"(VGA_DATA_REGISTER));
    asm("outb %b0, %w1" :: "a"(0x0E), "d"(VGA_ADDRESS_REGISTER));
    asm("outb %b0, %w1" :: "a"((uint8_t)((pos >> 8) & 0xFF)), "d"(VGA_DATA_REGISTER));
}

Mock implementation which the unit tests "inject":

Code: Select all

void cursor_mock_show() { }

void cursor_mock_hide() { }

void cursor_mock_setposition(int x, int y, int width) 
{ 
    cursor_mock_x = x;
    cursor_mock_y = y;
}

Factories that create the appropriate type:

Code: Select all

Cursor create_cursor() {
    Cursor cursor;
    
    cursor.Show = cursor_show;
    cursor.Hide = cursor_hide;
    cursor.SetPosition = cursor_setposition;

    return cursor;
}

Cursor create_cursor_mock() {
    Cursor cursor;
    
    // Initialise the cursor state to a known starting value
    cursor_mock_x = 0;
    cursor_mock_y = 0;

    cursor.Show = cursor_mock_show;
    cursor.Hide = cursor_mock_hide;
    cursor.SetPosition = cursor_mock_setposition;

    return cursor;
}

Kernel usage:

Code: Select all

asmlinkage int kernel_main()
{
    debug_message("Initialising");

    Cursor cursor = create_cursor();

    // INJECTION OF CURSOR:
    console_initialise(CONSOLE_WIDTH, CONSOLE_HEIGHT, (unsigned char*)CONSOLE_VIDEO_ADDRESS, (char)CONSOLE_WHITE_ON_BLUE, cursor);

Unit test usage:

Code: Select all

MunitResult console_cursor_should_initialise_to_zero_test(const MunitParameter params[], void* user_data_or_fixture)
{
    // Given
    unsigned char framebuffer[80 * 25 * 2];
    Cursor cursor = create_cursor_mock();
    
    // INJECTION OF MOCK CURSOR:
    console_initialise(80, 25, framebuffer, 0x00, cursor);

I'm interested in any discussion this may prompt.

Full source code located here: https://github.com/FrankRay78/InstructionOS