OSDev.org

I have an idea... (dont know if its good though.. )

Many of you probably would like to have a set of benchmars for how good your sheduler, memory manager etc is working.

I have developed a few test myself for my kernel. I was thinking maby.

What about developing a set of processes to be executed. Each task is defined with same inner loop code and os specific calls are created in MACRO's. So that you need to customize one .h file with macros for accessing the kernels functions and then just compile and start running.

Tests i would like to do:

- Scheduler time
- Task Priority
- Task Starvation
- Priority inheritence problems
- Syscall performance
- Memory allocation performance

All of these could be defined in loops and structures defined as a plain c code just calling of macros in the places needed.

Then after all tests a performance report could be created and we could compare..

Anyone want to join in on this creating some test?

-
Thomas

I would love a standard set of benchmarks, the problem is to what standard do you write the applications? What about different types of priority handling... some use priority levels, while some use coloring, or other techniques. What part of the memory manager are you going to test, the malloc implementation, the virtual memory manager, physical manager? What kind of system calls are you giong to test? I would be interested in helping out, because I'm very big into optimization, and being able to compare and share my code with others would be a big help. A few things need to be decided upon first though, like how to set the task priority level, what kind of allocation are you looking at, and how do you determine what *good* is? I mean, is my allocator good if I can allocate/free blazing fast, but use a great amount of resources in order to do so? Well, I guess that depends on your definition of *good* . How can you tell in a normal OS how much memory is being used/wasted while making allocations. In my OS I plan on having a way to determine this, but I am sure not all OS's are like this, and if they have similar concepts, probably implemented way differently. Also, for a scehduler, how do you determine good... not using to much time in the scheduler is good, so is managing the time slices nicely, but giving the proper priorities is important as well. How do you propose you measure these specific parts. Like I said, I would like to help, but just a few things to think about, because OS's have different ways of doing things, it's hard to make something general purpose.

I get your point. But i think most designs around have a set of design goals, most of the goals i think is to create a crontrolled environment where either the control is in kernel, task or maby both. Maby a lot of assumptions..?

Dispite what design you may have, most operating system have a set of basic services. These services could easily be measured i think, and the result presented in a orderly manner. Then you can with your design goal in mind see if you get the results expected, or if you maby need to take a look at some unwanted result.

Basic test for testing
user/kernel switch (if applicable)
thread switch time in same process.
process switch time.
processing time, in different priority levels.
response time when waiting for lock?
message passing
memory allocation physical,virtual or both?

-
Thomas

thomasnilsen wrote:I get your point. But i think most designs around have a set of design goals, most of the goals i think is to create a crontrolled environment where either the control is in kernel, task or maby both. Maby a lot of assumptions..?

Dispite what design you may have, most operating system have a set of basic services. These services could easily be measured i think, and the result presented in a orderly manner. Then you can with your design goal in mind see if you get the results expected, or if you maby need to take a look at some unwanted result.

Basic test for testing
user/kernel switch (if applicable)
thread switch time in same process.
process switch time.
processing time, in different priority levels.
response time when waiting for lock?
message passing
memory allocation physical,virtual or both?

-
Thomas

user/kernel switch: Do you mean, calling a kernel function and returning, or, how do you measure the time to switch, and what are you switching for? (what part of a 'normal' kernel does this test?)
thread switch time: Well, this is a good idea, but who controls the thread switching? And how do you measure the time? Do you suggest having a NextThread() function that will bring you to another thread similar to giving up your time in a process?
process switch time: How do you know which order the processes are in to know it went from this one to the next one you are expecting... what if it detoured through a device i/o process in between?
processing time/different thread priorities: Again, how do you judge priorities and how they should respond? My kernel might have 3 levels of priorities.. idle, normal, and high... while someone else might have 128 levels of priority. How do you directly compare the 2?
response time when waiting for lock: What kind of lock?
message passing: What size messages? Or do you want to test many message types/sizes? Do you propose message passing from app->kernel, kernel->app, app->app? Do you mean standard messages like, mouse move, keyboard press, etc, or do you mean like.... pass a 4mb image between 2 processes?
memory allocation physical/virtual/both: I personally would like a way to measure memory used vs. memory allocated for a malloc memory testing program, same for virtual memory. For physical memory, do you want to measure things like allocating a contiguous block under 1mb (ala floppy disk dma memory), contiguous under 16mb (isa anybody?), and random above 16mb for everything else? Also, a way to measure de-allocation and fragmentation (although, this is getting VERY dependant on implementation), or at least keep making psuedo random allocation/deallocations to try to create fragmentation and then check actual vs. requested memory allocated, this would give an indication of how fragmented it is.

Other things to test: Loading and running a specific application (including allocation, linking, setting up memory map, etc and actually running). You could have one application spawn another application, taking the time right before the spawn, and having the second application take the time when it is running, and send a message to the first application with the time stamp it took to see how long it took to reach a known point, then do it multiple times to check how well caching works (if implemented of course, or maybe it's on HD cache anyways).

Well, that's all my thinking for now, lol, let me know .

You still have a point..

Lets take an example. In your kernel you have 3 levels of priority. This means that you assume that you only have three levels of priority in your code.. atleast you disregard the finer granularity of your tasks "real" priority. This would be easy to show in a test.. The result would not tell you anything more than the result of time pr task, and average wait time for lets say your highest priority one. As i said before. These tests wont tell you how "good" your solution is. They will tell you the results of the test, and then you should compare this to your design metrics and see if you get the results you expect. Maby 3 levels of priority was to little? or how big a difference would it have to say if you had 30 instead? Its these kinds of "results" i wont to drill down by doing this kind of test.

AND, im still in my mind very mouch interessted in seeing.. How mouch time is affordable to use on optimizing scheduler, and choosing next process. How would parameters and logic in my basic services improve/decrease performance?

I totally agree in most of your opinions, but i think you also agree in most projects around here the designs are based on very mouch the same principles. Multiple tasks. A set of kernel functions that tasks can use...

Anyway.. I want timing, responsetime, statistics that tell how mouch time i have saved and used for other tasks rather than spending it on a task that maby did'nt need it

cheers
Thomas

This is exactly the goal I am aiming for with my testing software (as well as regression testing), however I was *going* to implement it specificially for my OS...

Alright, I see what you're saying... so it doesn't matter how many priorities, you can just put the priority how you want and see what it does. It's not directly comparable to another kernel though, which is what I was thinking of. Anyways, it would be a great tool for optimizing and I plan on writing something similar anyways, so it'd be great to write a generic tool that everyone could use. I am very interested still, as soon as I figure out how to fix this problem i'm having, haha. I love ld... you set it to align(1) or align(0), and it aligns it anyways . Took me a while to track down the problem, thought it was something in my code!