"Prototyping" in userspace?

Discussions on more advanced topics such as monolithic vs micro-kernels, transactional memory models, and paging vs segmentation should go here. Use this forum to expand and improve the wiki!
Post Reply
onlyonemac
Member
Member
Posts: 1146
Joined: Sat Mar 01, 2014 2:59 pm

"Prototyping" in userspace?

Post by onlyonemac »

Hi,

I'm just wondering how common this is, to "prototype" your operating system in userspace before actually writing it for real? What I mean is, to implement most of the basic functions of your kernel as a set of userspace programs/libraries to run under another operating system prior to writing your standalone kernel.

I've thought of a few advantages of this:
  • It allows one to thouroughly test and find and correct flaws in their design prior to spending time writing it for real
  • It allows one to focus on one's design without worrying about operating system stuff like hardware drivers
  • It allows one to finalise the specifics of their design, such as the exact layout of data structures, without worrying about trying to write that in kernel space at the same time
  • It means that when one comes to write one's kernel for real, one already has a set of algorithms for working with their data structures that can be ported over to the kernel
I see this as being kind of like writing one's kernel in pseudocode/flowchart format beforehand, but in a way that can actually be tested.

Currently I'm working on a set of routines to work with the object references in my operating system, and then I'll implement my object data structures and thoroughly test them. I'm doing all of this in C, and then when I come to write the kernel I'll port it over to assembler and replace C library functions (like "malloc", "strcmp", etc.) with those appropriate for my kernel. So when I write the kernel, I won't still be trying to work out the details of the operating system's actual design. The only things that I won't be able to test is loading stuff from disk (that will involve buffering data from disk and keeping track of which buffers are in use by which processes - a fairly complex thing which I probably won't implment for a LONG time) and the execution of code designed to run under the operating system.

Regards,

onlyonemac
Last edited by onlyonemac on Tue Dec 01, 2015 9:40 am, edited 1 time in total.
When you start writing an OS you do the minimum possible to get the x86 processor in a usable state, then you try to get as far away from it as possible.

Syntax checkup:
Wrong: OS's, IRQ's, zero'ing
Right: OSes, IRQs, zeroing
User avatar
max
Member
Member
Posts: 616
Joined: Mon Mar 05, 2012 11:23 am
Libera.chat IRC: maxdev
Location: Germany
Contact:

Re: "Prototyping" in userspace?

Post by max »

Hey,

I think that it can make sense to implement specific units on a for example your host system, because you can be certain that the surroundings work correctly and that any misfunctioning of your code is not caused by anything else on your own system. The only thing that might be a little tricky is to set up a proper building system & project structure to make it not messy. But all in all, I think it makes sense when you want to implement more complex stuff of which the dependencies are low and/or can be mocked. I did it too, for example when implementing some of my kernels allocators.

Greets,
Max
alexfru
Member
Member
Posts: 1111
Joined: Tue Mar 04, 2014 5:27 am

Re: "Prototyping" in userspace?

Post by alexfru »

onlyonemac wrote:I'm just wondering how common this is, to "prototype" your operating system in userspace before actually writing it for real? What I mean is, to implement most of the basic functions of your kernel as a set of userspace programs/libraries to run under another operating system prior to writing your standalone kernel.
It is the (a?) right way to do it. I mentioned it some 15 years ago in this article of mine: OS Development Skeptically.
User avatar
iansjack
Member
Member
Posts: 4685
Joined: Sat Mar 31, 2012 3:07 am
Location: Chichester, UK

Re: "Prototyping" in userspace?

Post by iansjack »

I certainly prototyped my filesystem code, using a disk image, and other stuff this way. It's far easier to debug userspace programs.
onlyonemac
Member
Member
Posts: 1146
Joined: Sat Mar 01, 2014 2:59 pm

Re: "Prototyping" in userspace?

Post by onlyonemac »

iansjack wrote:I certainly prototyped my filesystem code, using a disk image, and other stuff this way. It's far easier to debug userspace programs.
Exactly. So we don't have to worry about block device drivers while we're still trying to develop a filesystem.
When you start writing an OS you do the minimum possible to get the x86 processor in a usable state, then you try to get as far away from it as possible.

Syntax checkup:
Wrong: OS's, IRQ's, zero'ing
Right: OSes, IRQs, zeroing
Ready4Dis
Member
Member
Posts: 571
Joined: Sat Nov 18, 2006 9:11 am

Re: "Prototyping" in userspace?

Post by Ready4Dis »

I implemented my entire shell, vfs, and file system in windows so i could test it out. I then used the built in block device functions and/or file i/o (depending on if I wanted to work on a real disk or a virtual image). I could then format a disk and check in windows to see if it formatted correctly and copy files back and forth. I am currently working on my graphics driver interface under windows as well, although the setup for that is slightly more complex ;). I am using windows built in messaging to simulate my kernel messaging and writing my graphics driver as a standalone application, which then gets interfaced via a .dll file (user mode library) that knows how to call into the graphics driver. This way, I can have multiple apps link to the .dll which then creates the connection to my "graphics driver" so I can test multi-tasking out in the driver. It is a simple LFB driver for now (software rendering), so I just use SFML to grab a texture, update it manually and render it to screen. I see no reason not test my code before putting it live, unless it is something that isn't allowed under another OS. I even testing most of my VMM/PMM code out in windows, wrote a malloc implementation, etc before putting it into my OS. Of course, it's hard to do certain aspects which are easier to just fire up bochs and test, like my page fault handler. I *could* probably simulated this under windows as well, but it was easier to just do it and test in bochs. Once it works in bochs, I like to boot on some live hardware for a sanity check. I am not that far along, so take my methods with a grain of salt.
pcmattman
Member
Member
Posts: 2566
Joined: Sun Jan 14, 2007 9:15 pm
Libera.chat IRC: miselin
Location: Sydney, Australia (I come from a land down under!)
Contact:

Re: "Prototyping" in userspace?

Post by pcmattman »

My window manager works on Linux using SDL as a backend so I can do quick prototyping before adding an entire kernel startup to the start time.

I also wrote my dynamic linker to run on Linux first and ported later. I am also slowly adding more compatibility pieces as I start to do things like benchmark modules - not so much prototyping, though.

It can take some doing to make things work just right, but if your debug tools are substantially better on Linux than your kernel, there's a lot of benefits to making something that works to remove a variable from the equation.
embryo2
Member
Member
Posts: 397
Joined: Wed Jun 03, 2015 5:03 am

Re: "Prototyping" in userspace?

Post by embryo2 »

In the enterprise development world things like automated testing are so common that the development process very often includes many stubs and other "indirect" stuff. As a result the efforts required to prototype something under tightly controlled conditions are very small and many developers use such approach. OSDeving is behind, but tries to catch up. First - there are emulators, next there also is a lot of place for automated testing. The only step missing is the wide-spreading of such techniques. And of course, the best place to work with buggy code is the user space. But step in such direction leads straight to the artificial environment for development and testing, which is already common in enterprise development.
My previous account (embryo) was accidentally deleted, so I have no chance but to use something new. But may be it was a good lesson about software reliability :)
User avatar
Candy
Member
Member
Posts: 3882
Joined: Tue Oct 17, 2006 11:33 pm
Location: Eindhoven

Re: "Prototyping" in userspace?

Post by Candy »

I basically have my entire kernel cross-compile to local target as well, with test cases included, and run it as an executable. That tests whatever I have - and I can use dependency injection for the bottom most bits of hardware, so anything except for actual hardware drivers can be tested, and hardware drivers I may be able to test if they have a configurable memory map and blocking operations on it, by simulating the device in a separate thread.
onlyonemac
Member
Member
Posts: 1146
Joined: Sat Mar 01, 2014 2:59 pm

Re: "Prototyping" in userspace?

Post by onlyonemac »

Personally I have in fact stopped using this technique for now as I have switched to writing my kernel in C where it is easy enough to implement most of the functionality directly in kernel-space (rather, there is no real difference between writing it for kernel-space and for user-space, except for things such as memory allocation which I've got kernel-space stubs for at the moment). Also since I got sound output to work with QEMU, I've been able to do all of my testing in QEMU and frankly I think the total time taken for each launch of QEMU when debugging a module is less than the time that it would take to port that code to user-space - or at least the decrease in workload is worth whatever extra time it takes.
When you start writing an OS you do the minimum possible to get the x86 processor in a usable state, then you try to get as far away from it as possible.

Syntax checkup:
Wrong: OS's, IRQ's, zero'ing
Right: OSes, IRQs, zeroing
Kevin
Member
Member
Posts: 1071
Joined: Sun Feb 01, 2009 6:11 am
Location: Germany
Contact:

Re: "Prototyping" in userspace?

Post by Kevin »

qemu doesn't let use you things like valgrind, though.
Developer of tyndur - community OS of Lowlevel (German)
onlyonemac
Member
Member
Posts: 1146
Joined: Sat Mar 01, 2014 2:59 pm

Re: "Prototyping" in userspace?

Post by onlyonemac »

Kevin wrote:qemu doesn't let use you things like valgrind, though.
Well, I don't use valgrind, and if I ever do it'll probably be when there's a bit more kernel to my kernel.
When you start writing an OS you do the minimum possible to get the x86 processor in a usable state, then you try to get as far away from it as possible.

Syntax checkup:
Wrong: OS's, IRQ's, zero'ing
Right: OSes, IRQs, zeroing
Boris
Member
Member
Posts: 145
Joined: Sat Nov 07, 2015 3:12 pm

Re: "Prototyping" in userspace?

Post by Boris »

Basically, you are doing Test driven development. If well done, and if your apps works outside your kernel, you can use those apps to test your kernel
Post Reply