Design and Implementation of an Extensible OS

[jedld]

Hi,

My OS (DEX-OS) design stems from concepts found in Extensible Operating Systems (Hot-swappable / plug-in style modules), examples of OS'es like these are KEA and XINU. Initially, I intended it only for educational purposes since I want it to be easy for students to compare schedulers and scheduling algorithms but I discovered it has other advantages as well. My OS is already functional and features multitasking, ATA/floppy, ramdisk support, Virtual Consoles and Paging. There is also an "unstable" port of the Tiny C Compiler for dex-os as well as a stable port of nasm and some popular games

As an extensible operating system, I have implemented kernel mode plug-ins/grafts/extensions, user-mode plug-ins or KEA-style portals are still in my planning horizon. My scheduler is fully dynamic and can be changed even when processes are running. Also my kernel malloc function can be changed on the fly... once. Of course there are difficulties encountered when creating hot-swappable modules and many components need to be studied carefully in order to find a way to "disconnect" it from the system.

Visit my site or join the project at:
http://sourceforge.net/projects/dex-os/

I also have some documents about my design as well as my source code. There is also a floppy disk image for convenience.

Feel free to post at sourceforge or here about my design, developing an extensible operating system is very tricky and I need all the help I can get. Pype e-mailed me and told me that we have similar goals, I hope dex-os and the clicker project can work together to solve problems on making modules dynamic. An example problem: In the dynamic kernel malloc module, how do you deal with the realloc function? (This problem is illustrated in memory/kheap.c of the dex-os source code).

Thank you very much.

[Pype.Clicker]

welcome aboard, jedld, and thanks for providing us this introduction talk on your OS. I hope it will reveal useful for people here who do not work on Clicker (which is ... the vast majority of them )

for those who wonder what "KEA" is, i just found http://www.hpl.hp.com/personal/Alistair ... ccds96.pdf

[Pype.Clicker]

An example problem: In the dynamic kernel malloc module, how do you deal with the realloc function? (This problem is illustrated in memory/kheap.c of the dex-os source code).

Hmm. Haven't tried modularity *that* far atm, but here's how i would do it...

• on arrival/departure of a provider, the access to the interface is 'locked'. All threads that were executing one of the interface method can complete its job but no new thread is allowed (that's the job of the rwshare monitor in koLib)
• once no executors are present, the server switch can take place.
• the 'new' server asks the 'old' server its list of free frames and add them one by one in its own system
• once the enumeration is done, the 'new' server is ready for use and the interface is 'unlocked'

All we need is an 'enumerate free zones' method for inter-servers communication in this case.


Note that it's a quite slow approach. A more sophisticated method would be that the 'new' server partially 'obscures' the old one: it always receives newly freed zones and defers allocations to the 'old' server when it cannot handle them.
We could also imagine that the 'new' server would request the old server to give him free zones close to a newly freed zone so that merging can occur.

Does it sound interresting ?

[DennisCGc]

Pretty difficult, if you ask me
Those are very high (I think) goals.
Just downloaded the new release 1.0.2, but that doesn't work well, it stops with "Mounting boot device". (I know, I should post that in OSDEV testing )
I don't have that modularity ::)
What do you mean with dynamically scheduler ? ???
And.. what is your OS design ? ::)

[Pype.Clicker]

Pretty difficult, if you ask me Those are very high (I think) goals.

He! If it was for doing something simple, we wouldn't be coding an OS, would we ? I'm proud to have high goals. It took time to put things in place, but it gives a major feature that makes a system (almost) unique.

And.. what is your OS design ?

Jedld wrote a 'system design.txt' file in its source release... i bet his OS design is described there...

@dynamic scheduler: he just explained it: the scheduler of DEX-OS (much like the one of Clicker) is just a block that can be replaced by another (equivalent) block while the system is running. No need for changing the runlevel, nor rebooting or stopping process or whatever. You just plug the new scheduler in and the system automatically applies the new policy. You unplug it and the system reverts to the former policy.

[DennisCGc]

He! If it was for doing something simple, we wouldn't be coding an OS, would we ? I'm proud to have high goals. It took time to put things in place, but it gives a major feature that makes a system (almost) unique.

Hmm, that's true but it "sounds" difficult to me

@dynamic scheduler: he just explained it: the scheduler of DEX-OS (much like the one of Clicker) is just a block that can be replaced by another (equivalent) block while the system is running. No need for changing the runlevel, nor rebooting or stopping process or whatever. You just plug the new scheduler in and the system automatically applies the new policy. You unplug it and the system reverts to the former policy.

Ah, I see, but why should it be replaced ? ???
Sounds like a dumb question, but I want to know it.

Jedld wrote a 'system design.txt' file in its source release... i bet his OS design is described there...

I should read it, when I have time ;D

[ASHLEY4]

I wish you had posted a month ago, My OS "Dos Extreme"
uses DEX in the logo for short >:(.

I will have too re do my logo now :'(.

Dennis, "(Hot-swappable / plug-in style modules)",
It is in the vane of how linux does things, Take a simple idea, Make it sound hard and people think you are clever'er than you really are.

V2os did it year's go (in a very simple way) and is as simple has geting a exe to run in your OS.

ASHLEY4.

[jedld]

Indeed linux does support dynamic modules so to speak, but what are "modules" in linux. Modules in linux are mostly device drivers, an extensible operating system like KEA, SPIN and the *future* dex-os takes it far more than just that, it supports hot swappable device drivers and hot swappable kernel implementation.
What is the rational behind this? Extensible oeprating systems ride on the premise that "the applications knows best" and "there is no one best algorithm", and maybe it would pose an advantage if the application could control programmatically the operating system policies governing it.
In my case I initially created it for educational purposes since students can experiment and compare algorithms without making changes to the kernel, since the changes can be done programmatically by an application, you can create a hypothtetical application that could find the best memory management algorithm and the best scheduler given a list of memory management plug-ins and scheduler plug-ins given the current processes in memory. Such things couldn't be done with a kernel binded in super-glue.
Extensible operating systems researchers are also aiming to provided protection mechanisms to this so called plug-ins or kernel grafts to increase system stability. Of course extensibility is not only limited to schedulers and kernel mallocs but to other components as well, like the vmm, or the I/O manager.

here is a link for some more info:
http://www.cs.unm.edu/~riesen/prop/node26.html

[jedld]

With regards to the kernel memory allocator, switching kernel memory allocation algorithms itself is simple but unfortunately there some complications; This is because memory allocation algorithms have this property of "implementation specific persisitency" which means that applications depend on something specific to an implementation in this case, it is the address of the allocated heap. You cannot remove immediately the previous memory allocator because that would invalidate all memory addresses given by that memory allocation algorithm. when applications or kernel modules free all memory allocated by the previous algorithm then that is only the time wherein you can safely remove the previous one. So my current concept, which has a lot of problems, is that new malloc requests go to the new algorithm and if there is a free request, the heap interface determines who owns that free request and sends it to that algorithm. The realloc presents an interesting challenge since we are forced to use functions from both algorithms ... I have detailed this problem at the docs section in sourceforge.net for those who want to know. Another solution I believe is to not use the malloc paradigm at all but unfortunately I am not sure of any other way without making it to cumbersome.

Regards.

P.S: Even DOS could be considered extensible (in some way)since applications really hack the OS so much you'd wonder what DOS is really responsible for. But such techniques are so crude and unstable it makes me frown >:(.

[arafel]

Don't you guys think, such extent idea of fully swappable-modular os is just a bit unpractical in real world implementation?

Does anyone really will benefit from such kernel design? i guess if there someone ever will want to implement it's own 'whatever_part_of_kernel' in existing kernel. it can be achieved easily (of course it depends of ...) by rewriting corresponding part of kernel code. (also i suppose that in such 100% modular design some kind of extensive framework is required, which will probably hit the system performance..).

p.s in my opinion linux implementation of hot-swappable modules is extended up to perfect level. devices/parts that really "needs" to be hotswappable - implemented as such. other left alone.

[jedld]

In terms of practicalities sake, well that's one of the reasons I'm trying to do it, to see if it is indeed feasible. It is just that there are advantages to this system design that merits us to try and explore.

[chris]

In terms of practicalities sake, well that's one of the reasons I'm trying to do it, to see if it is indeed feasible. It is just that there are advantages to this system design that merits us to try and explore.

Agreed . jedld, your OS sounds interresting and you speak of ideas and questions I've been asking myself. I will remember to look into it when I have time.

P.S. Welcome to the boards.

[ASHLEY4]

The Linux quote, is not meant as in linux uses it, But is meant in that linux, Takes a simple task and make's it look hard,By useing long words, So they look clever,er than they really are.
(Do not get me wrong im a linux user,and love it, But the words it use's and the way it does things is so long winded )

So i will interpret it. Loads and unloads drivers,kernel-modules ETC ,with out rebooting (On the fly).

PS: If i take Dos (no EMS) run my kernel which start's in real mode set all the things i need for pmode ,set the memory model i want go to pmode,do some things and then go back to real mode dos and run a com file.
This is doing the same as your OS.

Remember KISS

ASHLEY4.

[mystran]

You cannot remove immediately the previous memory allocator because that would invalidate all memory addresses given by that memory allocation algorithm. when applications or kernel modules free all memory allocated by the previous algorithm then that is only the time wherein you can safely remove the previous one. So my current concept, which has a lot of problems, is that new malloc requests go to the new algorithm and if there is a free request, the heap interface determines who owns that free request and sends it to that algorithm.

Umm, excuse me, but what level of memory allocation are you talking about? I mean, usually there is page-allocator on the bottom. Then there's kernel malloc, then there's the virtual memory system, and then there's user level malloc or garbage collectors or whatever.

Now, let's take one at a time. Replacing the page-allocator on the fly is quite simple, just move all free pages from one allocator to another (in effect allocating all pages from the old allocator to the new one) and provide some interface for the new allocator to reuse any relevant data in case there's some. But then again, this level is usually just a stack of free pages, so what's the point.

In kernel level malloc, one will want to redirect all the allocation to the new malloc when switching. Here we have the problem that different allocators are going to use different schemes for dividing memory, so it's non-trivial to transfer allocations from one system to another. In any case, to replace such allocator, I'd probably implement some object-system, which has an interface consisting of malloc/free/realloc equivalents, and then explicitly use one implementation using a pointer. One could define the "default" allocator, switch it on the fly, or request all kernel modules/whatever to allocate memory from the other allocator and move data over. It's not fast, but that's probably the most reasonable thing to do, if one really wants to move over.

There's a problem with this though if one allows pointers between different modules, but even then one can always add a level of indirection similar to what some compacting garbage collectors do. This takes a small performance hit though. Other possibilities include to add a lot of glue to all interfaces just to communicate memory changes, or to require all allocators to handle existing allocations of any kind gracefully, and build an interface for transferring stuff over from one to another.

Now, virtual memory allocating is a similar problem, but since we are now dealing with pages, we have same size of basic unit in every allocator, so it's just a question of building an interface for transferring allocations over from one allocator to another, and then requiring all allocators to implement this interface. If performance (of allocator changes) is not issue, then this is almost trivial, and I can't imagine anyone wanting to change all the time. If virtual memory works on process level, one can even use different allocators for different processes, although this certainly adds overhead to shared memory and copy-on-write scenarios.

Finally, application-level malloc library has nothing to do with operating system kernels (as long as we have some interface to virtual memory management to request mappings for more pages).

So, which memory manager were you talking about?

[Pype.Clicker]

Takes a simple task and make's it look hard,By useing long words, So they look clever,er than they really are.

I know i'll get a "i didn't meant to offense anyone", but you actually did. Why would we fake anything ? Don't you think at this point we've gone over the "i'll conquer the world" stuff ?

Think of it briefly: what does Linux offer ? a system where you may (if lucky) disable a set of functions and then replace it by another set of function. Jedld's solution aswell as mine focus of a no-interruption of service approach: clients of a component *do not even notice* the server changed.

If you would for instance change the way the filesystem is behaving, you'd have no other option than first unmounting all the partitions that use that filesystem. That's precisely what we want to avoid.

Why bothering ? What do you do when you want to take a break at coding and play a game ? you reboot and switch to a MS product or to a game console because it had been better tuned for gaming. What if tomorrow, by the power of hot-swappable component, you could stay on the same OS and simply switch to another set of preferences and then going back when the game is over ?

Wouldn't that worth fighting for ?
Wouldn't that worth ... coding for ?