Benchmarks, bug hunting and so on

[dr_evil]

Some time ago I read a thread where the poster challenged others to create as many threads as possible in their OS. Some people wrote that this "benchmark" is worthless because it just depends on the amount of memory available. And they were right.

Then I tried to create as many threads as possible on my OS. And discovered several off-by-1-bugs in my code. Then I tried to create as many processes in my OS and noticed some design problems of my memory manager.

I did those tests to find about my OS's deficiencies:

• - Creating many threads
  - Creating many processes
  - Allocate much memory in a processes (=memory pressure) and then create new processes/threads. Does the kernel crash?
  - Play IPC ping-pong between two threads, measure performance
  - Play IPC ping-pong between two processes, measure performance

Do you know any other things I could try in an OS to find bugs?



And another question: Is your kernel instrumented? Does it trace kernel events (like scheduling decisions, etc.)? How does it report this data?

[lukem95]

measure perfomance for each step of your OS when booting and carrying out regular tasks, if something seems to take an unusually long time, investigate why.

[inflater]

Try out every error handlers. Invoke faults, check if the registers are displayed properly, and if you have error handlers that can restore work after e.g. page fault instead of system freezing, test if they are working properly.

[JackScott]

I would also mention usability testing, if you OS is at that stage. Make sure that the interface is something other people can understand. Of course, that's the whole point of the Test Requests forum.

[dr_evil]

Luke: Boot logging (with a dependency-graph) is a great idea. But my OS boots in around 2 seconds... So there's no need to optimize this. Yet.



My error handlers work fine (afaik), usability is not yet a problem. I was more thinking about high-level tests. Any other ideas?

[Solar]

Check for, and state, scaleability. How is your scheduler affected when the number of processes increases? (I.e., how much time does the scheduler take to figure out which process to run next.) Can it do it in constant time regardless of the number of processes (O(1)), or does performance degrade?

What happens to your IPC infrastructure when processes deadlock? (A waiting for B which is waiting for C which is waiting for A...) What happens when many processes deadlock? What happens to performance?

And the old standby: border case tests. In their simplest form, when a function takes an int as parameter, test what happens when it is fed:

• INT_MIN
• INT_MIN + 1
• -1
• 0
• +1
• INT_MAX - 1
• INT_MAX

Similar strategies can be adapted to many other functionalities.

As for instrumentalization, make it both a start-up parameter and switchable by runtime. Offer options to log to terminal, file, and / or serial port. (Difficult these days when many systems no longer have a RS-232 port - USB / Network are much more difficult to do.)

[lukem95]

Have you got FS support? try reading 10000 blocks, then writing 10000 and measuring how long each read/write takes, what the data speed is etc, if you combine this with other data being recorded at the same time (when scheduler/interrupts occur) you can see where your system is eating performance etc.

[Candy]

Also take into account the CS equivalent of cornering speed. Testing how fast every part in isolation is does not tell you the speed of the system as a whole. Always test combined cases as well.

Windows 3.11 was very fast at reading from the harddisk, reading from floppy and reading from cd-rom. If you were copying from a cdrom to a harddisk while streaming from a network to a floppy, they all slowed to a crawl.

Windows XP still slows to a crawl if you copy with two threads lots of small files from a filesystem to another filesystem. It's not hard, it should be going very quickly but it is horribly slow.

[dr_evil]

What happens to your IPC infrastructure when processes deadlock? (A waiting for B which is waiting for C which is waiting for A...) What happens when many processes deadlock? What happens to performance?

What should happen?

In my OS they cannot deadlock by IPC. Priority inheritance, IPC design and OS checks make it impossible to deadlock. But an application by itself can naturally deadlock. Is there anything wrong with that?

[Solar]

In my OS they cannot deadlock by IPC. Priority inheritance, IPC design and OS checks make it impossible to deadlock.

That was the part I wanted to hear. Good work!

[lukem95]

In my OS they cannot deadlock by IPC. Priority inheritance, IPC design and OS checks make it impossible to deadlock.

That was the part I wanted to hear. Good work!

/me agrees