Returning to my microkernel

[AndrewAPrice]

For nearly a year I haven't made much progression on my OS, but I'd like to resume it and I'm going to do a complete rewrite and plan for a much as possible from the beginning.

It's going to be a micro kernel, and I want to keep the kernel itself relatively small. The kernel will be split in to 5 sections (purely abstraction sections):

• Interrupt Management - Sends events to drivers that are listening for a certain interrupt, and handle system calls.
• Process Management - Handles creating/destroying processes, prioritises processes, also parsing ELF headers.
• Device Management - Handles devices and their drivers.
• Timer/scheduler - Handles switching between processes.
• IPC - Handles processes trying to talk to each other: events, messages, pipes, shared memory.

There are going to be 3 levels of processes. The scheduler won't let a higher level process do work until all lower level processes are in an idle/sleep state. They are all ran in user mode and only vary in the types of system calls they can make.
Starting from the lowest level, the levels are;

• Drivers:
  Can map physical memory range in to it's own address (e.g. DMA), write to/read from ports, can create/destroy devices, and do everything servers can.
  Servers:
  Can communicate with drivers, and do everything applications can.
  Applications:
  Can communicate with servers and other applications.

One thing I'm concerned about with the drivers is the performance penalty if each time they want to write out/read in a port they must make a system call. It would be easy enough though to group of bunch of in/outs into a single system call.

I'm going to identify processes (and pipes) by a name, which doesn't exist on the file system (though nothing says it couldn't be mounted on the VFS). The name will be separated in to their categories. For example; "servers\vfs". Programs will be able to define their own names, though they will be given a default one. For example a program with the filename "texted.bin" will be loaded as "applications\texted.bin". If that name already exists, it'll append a number on the end; "applications\texted.bin1", etc.
Possibly the name could include the user's name aswell, for example; "applications\Andrew\texted.bin". When you query the kernel for a list of processes, you'll be able to use wildcards (e.g. "applications\Andrew\*" to get every application executed under my account). These names will resolve in to a PID or (PiID in the case of a pipe), and you'd use the PID for communicating with the process.

As for devices, I'm thinking that drivers could register a device. And when you query the kernel about a device, e.g.
"devices\disk\0" it will return the device's ID along with the driver's PID, and you simply send messages to the driver (e.g. Read chunk from device 1).

So, that's my plan for the kernel. What do people think? I'd like it to be criticised and all suggestions are welcome.

[Laksen]

Why do port io through syscalls? Just give higher IO privileges to drivers

[AndrewAPrice]

Is that possible without running the driver in ring 0?

[AJ]

Hi,

Yes it is - you can either:

• Each time a GPF occurs in ring 3, check whether the IO was allowed and perform the IO (slow for multiple reads).
• Use the IO privilege bitmap in the TSS.
• Make careful use of IOPL (which has the disadvantage of giving all drivers access to all ports!).

Cheers,
Adam

[Laksen]

Sure. Place it in CPL3. Drivers should then have IOPL3 to access IO Ports, and servers and applications can get IOPL0. You would have to clear the IO Permission bitmap to make sure the drivers have access to all ports though(if you want that)

[AndrewAPrice]

Thanks for that

I've just finished my list of system calls (attached). It's mostly about Memory Management, Device Management (which could actually be a server but is part of the kernel for now), Process Management, and IPC.

I've tried to make it as standalone and server independent as possible (e.g. the kernel does not know anything about the GUI, input, FS, etc). This is broken on 2 conditions though; the first is when it spawns a new process and it communicates with server/vfs to load the executable, and the second is when an error occurs and it sends the event to server/messenger (which could dump the error to console, or in the future display an error window).

[itisiuk]

looks good.
although seems to be a lot of ways todo IPC.
would it not be better just to use one method??

also on the topic of microkernels.
how do u implement the memory manager and does it use virtual address spaces i.e. a pgdir per process ??
i always got confused on this part thats why i never tryed writing a microkernel.

[AndrewAPrice]

Memory management is done in the kernel, and so is process management. And each process (driver/server/app) runs in it's own address space, which is why shared memory is important (e.g. imagine having to stream a window's bitmap over a pipe).

The kernel does somewhat rely on individual processes, e.g. it will contact "server/messenger" when it wants to display a dialogue box (e.g. "This application has tried to divide by zero (naughty, naughty).. So, I've gone ahead and killed it!"). If I wanted to log these messages, I'd just have to change the messenger server.

Other than that, the kernel does not even know there is a file system (though I may break this rule with an exec() function or similar).

[itisiuk]

I see your point about trying to send a bitmap over a pipe or though the registers it would be very slow, i didnt think
about that at all.

this got me thnking about how to implement shared memory between process.
so far the only way i could think of doing it was to implement a shared memory region
say between 0x1000 and 0x10000 just for example.
then have the kernel map the requested shared mem size say 0x2000 from the 1st free location in the shared region in the two process needing shared mem.
i.e
process 1 and 2 could map mem in this region between 0x1000 and 0x3000
process 3 and 4 could have 0x3000 0x4000 but not map between 0x1000 and 0x3000 as process 1 and 2 is using that.
and again process 1 and 2 could have 0x4000 to 0x5000 with the same condition as above.

however no shared mem could be mapped other than between 0x1000 and 0x10000 for all processes so other data could not collided with it.

soz for the bad explanation i was just writing this down as i was thinking it.

anyway youve inspired me to try and do a microkernel now.
i got bored trying todo my monalithic kernel even though i nearly have it fully working with a shell
and the possiblity of graphics as the vm86 handler works finally.

[edit]

well ive just stripped down my old kernel to just the startup, mem manager and process hander and have decided to use
the minix ipc and am busy implementing it now.

anway i was wondering if execeptions had to be handled with a stack switch or could i use my old code which does and just halts
unless theres a vm86 gpf which does the vm86 task until finished.

[/edit]

[AndrewAPrice]

In my kernel, then memory you want to share to/from must be 4kb aligned (my allocator will have an option to allocate things aligned) and the kernel will give the shared memory an ID number. Then you contact the window manager and say "Hey, I'm creating a window that's 100x100. The title is 'A Window', and it's bitmap is located in shared memory #493."

The window manager will say "Yo, kernel, give me access to shared memory #493" and it'll map that shared memory in to the process's memory and return a pointer to it. The shared memory has a reference counter in the kernel, and the memory is only released when all process's referencing the shared memory has released it.

Another highly desirable feature of my kernel (though it won't come until a while later) is when a server or driver becomes unresponsive (e.g. it hasn't received the next event after 500ms (which isn't unreasonable since a server/driver's main priority is being responsive to events)) and the kernel will send a message to the messenger (along with debug info), kill the process, and restart it. Which shouldn't be an issue if the VFS crashes, because the VFS is loaded as a GRUB module anyway, and I reserve that are of memory, so I can reload the process just as it was on bootup.

[AndrewAPrice]

@itisiuk:
Halting should be fine. Right now the kernel freezes just after it sends something to my messenger, and I'm going to keep it like that until my kernel is stable enough that I 99% trust it won't crash.

[itisiuk]

Another highly desirable feature of my kernel (though it won't come until a while later) is when a server or driver becomes unresponsive (e.g. it hasn't received the next event after 500ms (which isn't unreasonable since a server/driver's main priority is being responsive to events)) and the kernel will send a message to the messenger (along with debug info), kill the process, and restart it. Which shouldn't be an issue if the VFS crashes, because the VFS is loaded as a GRUB module anyway, and I reserve that are of memory, so I can reload the process just as it was on bootup.

i like that idea, nice and simple really, the way things should be .

Halting should be fine. Right now the kernel freezes just after it sends something to my messenger, and I'm going to keep it like that until my kernel is stable enough that I 99% trust it won't crash.

think il just keep my exception handler like that then, with the debug msg's of course.

your shared mem manager seems a little complicated, well for me anyway. think ill have to build upto something like that once i get done with the basics.

anyway i think youve got a good idea there with how your microkernel is going to work.

[elderK]

Andrew, your approach to handling unresponsive servers leaves me with a few questions.

But before I ask them, I will make sure that I understand your kernel's layout correctly...

If I understand it right, you're microkernel is basically just a tiny, tiny monolith. Exporting the drivers and suchlike out into user space. But, the core functionality is still embedded in the kernel core (thus, lil' monolith). Right?

In the event that one of your servers become unresponsive and fails to act on whatever event, you kill it - restart it, right?

... What happens to it's data? What if it crashed handling a transaction? You are lucky in that your servers do /not/ control core functionality (ie, memory management - you really wouldn't want to lose the tracking data...). Do you checkpoint data? Or do you simply say 'Bubyeeeeeeee server's private data!'.

Just curious.
~K

[elderK]

Also, now that I think about it - what about hand-offs?

Ie: You have Block Server... or whatever. It controls block devices. Seeks, reads, to/from Hard drives, whatever...

Say this server was written by John Bloggs. It's reliable and trusty - but lacks several features that you want from Mr. Zuggs server? (Say, block caching or read/write overlaying, whatever...). Running both of these servers simultaneously is stupid and dangerous as they both demand control for the same hardware and so, you need to gently phase control from one, to the other (because no doubt, Blogg's server is still handling transactions.)

It may be in this case that the Block server is tracking data about device usage, scheduling writes and so on. If this /is/ the case, you want that data to be exported to Zugg's server - so it doesn't have to gather all of that intelligence again.

How do you handle transitions from one server to another, when those servers demand the same hardware? (memory, ports, etc).

~K
This is something I've been doing some thinking on a lot lately, as it is very important in the operation of my kernel. Ie: What happens when you replace the Memory Manager, Scheduler, Process Manager or IPC Control? How do you 'phase' between them? How do you 'hand off' control? How to deal with programs still actively _using_ the server being removed? Etc... It's fun. And quite doable!