OSDev.org

Hi,

I'm just wondering how common this is, to "prototype" your operating system in userspace before actually writing it for real? What I mean is, to implement most of the basic functions of your kernel as a set of userspace programs/libraries to run under another operating system prior to writing your standalone kernel.

I've thought of a few advantages of this:

• It allows one to thouroughly test and find and correct flaws in their design prior to spending time writing it for real
• It allows one to focus on one's design without worrying about operating system stuff like hardware drivers
• It allows one to finalise the specifics of their design, such as the exact layout of data structures, without worrying about trying to write that in kernel space at the same time
• It means that when one comes to write one's kernel for real, one already has a set of algorithms for working with their data structures that can be ported over to the kernel

I see this as being kind of like writing one's kernel in pseudocode/flowchart format beforehand, but in a way that can actually be tested.

Currently I'm working on a set of routines to work with the object references in my operating system, and then I'll implement my object data structures and thoroughly test them. I'm doing all of this in C, and then when I come to write the kernel I'll port it over to assembler and replace C library functions (like "malloc", "strcmp", etc.) with those appropriate for my kernel. So when I write the kernel, I won't still be trying to work out the details of the operating system's actual design. The only things that I won't be able to test is loading stuff from disk (that will involve buffering data from disk and keeping track of which buffers are in use by which processes - a fairly complex thing which I probably won't implment for a LONG time) and the execution of code designed to run under the operating system.

Regards,

onlyonemac

Hey,

I think that it can make sense to implement specific units on a for example your host system, because you can be certain that the surroundings work correctly and that any misfunctioning of your code is not caused by anything else on your own system. The only thing that might be a little tricky is to set up a proper building system & project structure to make it not messy. But all in all, I think it makes sense when you want to implement more complex stuff of which the dependencies are low and/or can be mocked. I did it too, for example when implementing some of my kernels allocators.

Greets,
Max

onlyonemac wrote:I'm just wondering how common this is, to "prototype" your operating system in userspace before actually writing it for real? What I mean is, to implement most of the basic functions of your kernel as a set of userspace programs/libraries to run under another operating system prior to writing your standalone kernel.

It is the (a?) right way to do it. I mentioned it some 15 years ago in this article of mine: OS Development Skeptically.

I certainly prototyped my filesystem code, using a disk image, and other stuff this way. It's far easier to debug userspace programs.

iansjack wrote:I certainly prototyped my filesystem code, using a disk image, and other stuff this way. It's far easier to debug userspace programs.

Exactly. So we don't have to worry about block device drivers while we're still trying to develop a filesystem.

I implemented my entire shell, vfs, and file system in windows so i could test it out. I then used the built in block device functions and/or file i/o (depending on if I wanted to work on a real disk or a virtual image). I could then format a disk and check in windows to see if it formatted correctly and copy files back and forth. I am currently working on my graphics driver interface under windows as well, although the setup for that is slightly more complex . I am using windows built in messaging to simulate my kernel messaging and writing my graphics driver as a standalone application, which then gets interfaced via a .dll file (user mode library) that knows how to call into the graphics driver. This way, I can have multiple apps link to the .dll which then creates the connection to my "graphics driver" so I can test multi-tasking out in the driver. It is a simple LFB driver for now (software rendering), so I just use SFML to grab a texture, update it manually and render it to screen. I see no reason not test my code before putting it live, unless it is something that isn't allowed under another OS. I even testing most of my VMM/PMM code out in windows, wrote a malloc implementation, etc before putting it into my OS. Of course, it's hard to do certain aspects which are easier to just fire up bochs and test, like my page fault handler. I *could* probably simulated this under windows as well, but it was easier to just do it and test in bochs. Once it works in bochs, I like to boot on some live hardware for a sanity check. I am not that far along, so take my methods with a grain of salt.

My window manager works on Linux using SDL as a backend so I can do quick prototyping before adding an entire kernel startup to the start time.

I also wrote my dynamic linker to run on Linux first and ported later. I am also slowly adding more compatibility pieces as I start to do things like benchmark modules - not so much prototyping, though.

It can take some doing to make things work just right, but if your debug tools are substantially better on Linux than your kernel, there's a lot of benefits to making something that works to remove a variable from the equation.

In the enterprise development world things like automated testing are so common that the development process very often includes many stubs and other "indirect" stuff. As a result the efforts required to prototype something under tightly controlled conditions are very small and many developers use such approach. OSDeving is behind, but tries to catch up. First - there are emulators, next there also is a lot of place for automated testing. The only step missing is the wide-spreading of such techniques. And of course, the best place to work with buggy code is the user space. But step in such direction leads straight to the artificial environment for development and testing, which is already common in enterprise development.

I basically have my entire kernel cross-compile to local target as well, with test cases included, and run it as an executable. That tests whatever I have - and I can use dependency injection for the bottom most bits of hardware, so anything except for actual hardware drivers can be tested, and hardware drivers I may be able to test if they have a configurable memory map and blocking operations on it, by simulating the device in a separate thread.

Personally I have in fact stopped using this technique for now as I have switched to writing my kernel in C where it is easy enough to implement most of the functionality directly in kernel-space (rather, there is no real difference between writing it for kernel-space and for user-space, except for things such as memory allocation which I've got kernel-space stubs for at the moment). Also since I got sound output to work with QEMU, I've been able to do all of my testing in QEMU and frankly I think the total time taken for each launch of QEMU when debugging a module is less than the time that it would take to port that code to user-space - or at least the decrease in workload is worth whatever extra time it takes.

qemu doesn't let use you things like valgrind, though.

Kevin wrote:qemu doesn't let use you things like valgrind, though.

Well, I don't use valgrind, and if I ever do it'll probably be when there's a bit more kernel to my kernel.

Basically, you are doing Test driven development. If well done, and if your apps works outside your kernel, you can use those apps to test your kernel