Any pointers to a good reference on this? Perhaps someone should add something to the wiki...Solar wrote:Unit tests / scenarios are a most valuable part of a software project, and peer review by mayhaps a dozen people from this board is a very poor second to that...
function/code test area?
- Love4Boobies
- Member
- Posts: 2111
- Joined: Fri Mar 07, 2008 5:36 pm
- Location: Bucharest, Romania
Re: function/code test area?
"Computers in the future may weigh no more than 1.5 tons.", Popular Mechanics (1949)
[ Project UDI ]
[ Project UDI ]
Re: function/code test area?
Erm... I'd have thought unit tests to be common knowledge by now...
JUnit
CppUnit
Boost Test
If that's all too much framework to you, you could of course write your own simple test drivers to go alongside with your sources, like I did for PDCLib. (Download link, the source repo is currently down for maintenance. See my signature to check if it's up again.)
JUnit
CppUnit
Boost Test
If that's all too much framework to you, you could of course write your own simple test drivers to go alongside with your sources, like I did for PDCLib. (Download link, the source repo is currently down for maintenance. See my signature to check if it's up again.)
Every good solution is obvious once you've found it.
- Love4Boobies
- Member
- Posts: 2111
- Joined: Fri Mar 07, 2008 5:36 pm
- Location: Bucharest, Romania
Re: function/code test area?
Well I've never actually used this technique. I did hear about it and I sort of immagined what it was all about telling from its name; however I wanted to be sure that I have the right idea. Thanks.
"Computers in the future may weigh no more than 1.5 tons.", Popular Mechanics (1949)
[ Project UDI ]
[ Project UDI ]
Re: function/code test area?
An very good addition to unit testing is coverage analysis. This is supported by GCC via some compiler switches and the GCOV tool. Using this you can see which parts of your software are really tested (on function or codeline basis) by your unit tests and extend the tests to hopefully reach up to 100% test coverage.
At my work this is a) a very good help and b) mandatory
At my work this is a) a very good help and b) mandatory
Re: function/code test area?
Yep, I second LMN on coverage. At my last job, we had >95% coverage with our unit tests, which meant you could make changes in basically unfamiliar code and be quite confident that, if the unit tests still worked, you hadn't broken anything.
In my current job, there's no such thing as unit tests, and every time I touch one of the algorithms, I wouldn't know if I had done something wrong until one of the customers comes complaining about bogus results. Not fun.
In my current job, there's no such thing as unit tests, and every time I touch one of the algorithms, I wouldn't know if I had done something wrong until one of the customers comes complaining about bogus results. Not fun.
Every good solution is obvious once you've found it.
Re: function/code test area?
I must admit that the concept of unit testing operating systems is quite unfamiliar to me.
Could one of you guys that have used it in such a low-level environment post a how-to or even a wiki-tutorial on how to get it setup and such for OS unit testing?
[edit] I didn't look up far enough, it seems Love4Boobies has beat me to it. [/edit]
Could one of you guys that have used it in such a low-level environment post a how-to or even a wiki-tutorial on how to get it setup and such for OS unit testing?
[edit] I didn't look up far enough, it seems Love4Boobies has beat me to it. [/edit]
Website: https://joscor.com
Re: function/code test area?
You can unit-test everything. Of course most frameworks expect a full user-space environment, but in the end unit-testing is merely the process of proving your functions do what they are expected to do, in code. It's one thing to have a kprintf() function that "works for me", and another to have code that tests every feature that my kprintf() is advertised to have, in every possible combination. ("OS Test Screen", so to speak. )
Every good solution is obvious once you've found it.
Re: function/code test area?
Hi
I have written a little bit about testing. You are free to integrate it into the wiki if appropiate.
I have written a little bit about testing. You are free to integrate it into the wiki if appropiate.
Testing
* "How to test?" is a question that cannot be answered in general. "When to test?" however, does have a general answer: as early and as often as possible.
o Stroustrup, Bjarne. The C++ Programming Language, 712.
This article will describe some techniques to test your hobby OS. Testing can be performed on many levels and I will tell about my approach to testing.
Testing the entire operating system
You can test your OS in broadly two ways: on real hardware or on simulated hardware. Both methods are valid and has its pros and cons.
Emulation and virtualization software is really, really good. It is routinely used to deploy large production environments [citation needed]. It also has the big benefit that you can test on the same machine you use for development without rebooting. This really speeds up the development speed and encourage more testing.
Simulation comes in two flavors: emulation or virtualization. Emulation emulate an entire PC and all its devices in software. This makes the performance quite bad – the speed of the emulated PC is several orders of magnitude worse than the host machine due to large overhead. Virtualization requires the host CPU to be identical to the guest CPU. This allows the virtualization software to run most of the code directly on the host CPU and only simulate critical CPU instructions and devices. Since most devices are much slower than the CPU and memory this gives very good performance. It is very close to running on the real hardware.
The benefit of emulation is that it is possible to stop execution at any point and inspect the state of the emulated machine. This allows debugging of very difficult low level problems without the help of specialized and expensive equipment. In practice the low speed is not a problem until you have quite a large system implemented.
During development it is often sufficient to test using simulation. But you have to test on real hardware too. Try to do if regularly. Bugs that crashes on real hardware but works fine in simulation are by nature very difficult to find and fix. So the smaller the change set to investigate the easier it will be.
But in reality OS development is more about software structures and algorithms than interfacing with the hardware. My experience is that the hardware access layer is quite “thin” and not very complex. The software layer is much more complex. But both things take quite a lot of time to get right.
The goal is to find a balance between development speed and testing confidence.
Unit testing
When starting out writing the first lines of code for the kernel it might seem impossible to unit test. But you would be surprised ow much it is actually possible to test. Your kernel will need some form of library and the usual approach is to use a subset of the standard libc or similar system library. If you port another libc you can probably be pretty sure that it works as advertised [hehe].
If you write your own libc please take the time to make unit test up front or at least write a test suite later. This will save you countless hours of frustrations. Chasing a bug in the kernel that turns out to be an off-by-one error or a reverse larger than character is really, really annoying. And it is actually easy to implement testing. Most if not all of the libc subset used in the kernel is platform independent – so the tests and the code can be implemented on your host platform using all the available tools. This is much easier and faster than debugging using printf() in the kernel.
Many other parts of the kernel can be tested quite easily too: you will need an assortment of lists, queues, stacks, bitmaps, etc. All these data structures and the operations on them can be implemented and tested in isolation.
You can write tests that verifies that the kernel system calls handles errors correctly. Later when the system starts working “by itself” you can create test applications that creates new processes and tests that it is handles correctly etc.
Other parts can also be tested but with a little more work. I currently test how I handle the GRUB memory map and determines how much physical memory is in the machine. It came in very handy recently. I had the system prepared to be tested but had not actually implemented the tests. My system didn't work on a test machine of [TODO]. He provided me with a transcription of the memory map and I was able to find the problem quite fast. I also added unit tests using all the memory map that I have access to. Since I now tests how my code handles this specific memory map I hope to prevent any errors on this machine in the future.
Implementing a test framework
I have implemented my own very simple testing frame work. It consist of a macro unit_test() which is called with a boolean as parameter. If true the test is assumed to fail and false that it succeeds – I know this is backwards but I accidentally started like that and now I have more than 500 tests and it is easier to keep going that change everything.
#define unit_test(expression) {if(expression) {textattr(RED + BLACK * 16); printf(" ==> file: %s, function: %s(), line: %i\nFAILURE: ", __FILE__, __FUNCTION__, __LINE__); ++__unit_test_failure_count; } else {textattr(LIGHTGREEN + BLACK * 16); printf("SUCCESS: "); ++__unit_test_success_count; } }
It is gcc specific since it uses __FILE__ etc. to be able to print sensible error information. It also uses a small subset of conio.h from [...]. I have implemented this for both my own OS and for Cygwin. It also requires to variables to be declared (int __unit_test_failure_count and int __unit_test_success_count).
I call it like this:
unit_test(condition);
printf("test description\n");
The result of running a series of tests will be a nicely colored output with green lines for success and red lines for failures. The failures will also print which file, function and line number was faulty. At the end it is possible to print the total number of test successes and failures.
I implement each test suite as its own executable and has a test runner that runs these executables. The individual test suite executables reports back to the test runner and tells how many tests failed and how many succeeded. The test runner is responsible for selecting which test suites should be run and I control it with simple command line parameters.
This gives me a somewhat easy to use and quite flexible testing frame work. I hope this will inspire you to implement more tests for your OS.
Final words
I consider testing to be a requirement. This is the only way that I have any chance of measuring stability. And stability is one of my main goals. Testing also gives you great confidence that changes will not accidentally break something.