Measuring the Efficiency of an Operating System

[Edge]

I've recently put together an operating system kernel that is organized differently than your conventional kernel. I'm looking to measure the efficiency of my kernel in depth, and compare it to a kernel such as Linux, or equivalent.

What is the unit of measurement I should use, IPC? Are there any tools that can do this for you, or can I get a log from QEMU?

This is the one roadblock I've hit in this project. Any help would be appreciated.

[bluemoon]

In my opinion, each OS has different feature set, target usage and philosophy. In addition, modern OS may have different tuning for different usage.
Therefore it's not much you can compare.
But then, it depends on what target audience your benchmark report for. For end user, the number of desktop themes available to download may be even more meaningful than how long it take for IPC.

It's also not meaningful for benchmark any individual operation. Unless your OS outperform, say, 100 times faster than windows when copying a directory Then you can do such benchmark and put onto portfolio and attract users.

[Edge]

I understand. However, I still want to be able to compare the speed of my kernel - maybe performing a certain operation - to the Linux kernel or equivalent.

[bluemoon]

How about kernel call overhead? We have a thread on this yesterday.

[AndrewAPrice]

It depends what features your OS's have in common. If you just compared IPC speed - my OS may win because IPC is nothing more than a function pointer. My point is, make an exhaustive test:
- Allocate and free 500 MB of random objects a million times.
- Copy a 1GB directory of 50kb files from one drive to another.
- If a common tool is on both OS's (e.g. GCC) - compare how long it takes to compile the test suites.

[Brendan]

Hi,

What is the unit of measurement I should use, IPC? Are there any tools that can do this for you, or can I get a log from QEMU?

Use something like the Phoronix Test Suite that tests the system as a whole.

Don't bother with benchmarks that only test a CPU's floating point performance or test a computers RAM bandwidth or things like that (because you're not comparing CPUs or hardware). Don't worry much about micro-benchmarks like kernel API speed or IPC "ping pong" either - these tests are good for optimising (in the same way profilers are good), but are mostly meaningless for comparing OSs.


Cheers,

Brendan

[Edge]

Thanks guys. I figured this one out. I did what MessiahAndrw said. I had my OS environment perform a rigorous calculation and then I compared it to that of another OS environment.