Your OS design

[KeeperOC]

By moving large chunks of the kernel into user space, and having the remaining kernel preempt the running system, check for any outstanding system calls and then return execution to userspace processes. I'm almost certainly going to use some system calls for efficiency's sake, but it is possible to implement those abstractions without directly calling the kernel, just letting it respond in it's own time.

[Colonel Kernel]

By moving large chunks of the kernel into user space

Then it's not a kernel anymore...

it is possible to implement those abstractions without directly calling the kernel, just letting it respond in it's own time.

So it's based on polling. What's the advantage of doing it this way?

[_OScoder]

How would you implement IPC without a system call? How would the kernel know when to take a peek in the shared memory and see what requests are there? How would the client know when to take a peek in the shared memory and see when the kernel's response has been delivered?

If the kernel is running as a process, it could block until the 'stream' was written to. Streams (areas of shared memory) could be 'frozen', but marking them as read-only; thus, when a stream is written to, it will cause a page fault, and then kernel can instruct the sceduler to unblock the relavent thread.

Thanks,
OScoder

[Colonel Kernel]

when a stream is written to, it will cause a page fault, and then kernel can instruct the sceduler to unblock the relavent thread.

Is the relevant thread the one that was writing to the shared memory region? If so, let's say the page fault handler makes the shared memory region read-write and lets the thread finish writing there. How would the kernel know when that thread has finished writing? If that's not the relevant thread (i.e. -- the PF handler doesn't resume the faulting thread), then the only pieces of information the kernel has is which thread tried writing to the area, and at what address -- no message has been "passed" to the kernel in this case.

Whether it's a page fault or a system call, it's still a trap of some kind to the kernel. At least in the system call case you can use optimized instructions for that purpose (e.g. -- syscall/sysret/sysenter/sysexit on AMD/Intel). I still don't see the reason to do things this "IPC-oriented" way.

[KeeperOC]

By moving large chunks of the kernel into user space

Then it's not a kernel anymore...

No, it's a microkernel.

it is possible to implement those abstractions without directly calling the kernel, just letting it respond in it's own time.

So it's based on polling. What's the advantage of doing it this way?

That's the question asked by the post that caused this whole sub-thread - what are the advantages/disadvantages of a kernel which uses no calls/traps etc., the obvious answer being a lack of efficiency, but an increase in kernel abstraction (thus greater flexibilty). We hadn't got round to that because we were discussing your assertion that such a kernel was not even possible.

Andrew

[0Scoder]

Ok, this small discussion is kinda going offtopic from the subject of kernel design I think. I'll post this then say no more on the subject. The way I designed the blocking system to work is:

Thread A has no messages in stream A -> Thread A blocks on that stream (through the kernel stream) -> kernel marks stream A's pages as unwritable, and stores a list of the threads to be unblocked when the stream is written to -> Thread B writes to this stream -> page fault to kernel -> kernel locates stream through its memory map, and finds that thread A was blocked on it -> kernel wakes up thread A -> thread B continues as if nothing happend.

I can see no problems with this initially,
OScoder

[Colonel Kernel]

No, it's a microkernel.

Not the parts that are in user space. It doesn't matter; I think I know what you mean.

That's the question asked by the post that caused this whole sub-thread - what are the advantages/disadvantages of a kernel which uses no calls/traps etc., the obvious answer being a lack of efficiency, but an increase in kernel abstraction (thus greater flexibilty). We hadn't got round to that because we were discussing your assertion that such a kernel was not even possible.

I made no such assertion, I just asked leading questions. ;D

To me, the "greater abstraction = greater flexibility" argument doesn't hold water. I've seen this used as a justification for over-engineering things in horrible ways just for the sake of some aesthetic notion of "more abstract is better".

What concrete advantages are there? I agree that a microkernel architecture is very flexible, but microkernels usually implement IPC in terms of system calls, not page faults and/or polling.

Ok, to put it another way: I don't see any advantages. I guess I'm asking what made this somehow look like it might be a good idea.

@OScoder:

Thread A has no messages in stream A

Ok.

Thread A blocks on that stream (through the kernel stream)

I'm not sure what you mean by "through the kernel stream". How would thread A block without a system call? The only way I can think of is if it caused a page fault.

kernel marks stream A's pages as unwritable, and stores a list of the threads to be unblocked when the stream is written to

Ok.

Thread B writes to this stream -> page fault to kernel -> kernel locates stream through its memory map, and finds that thread A was blocked on it -> kernel wakes up thread A

Ok.

thread B continues as if nothing happend.

Huh? I thought the kernel woke up thread A... Are you assuming that threads A and B are on different processors?

You didn't really answer my original question of how thread A (the receiver) gets the data that thread B (the sender) was trying to write into the queue...

I don't think this is offtopic as your idea needs to be understood before anybody else can offer more opinions on its advantages and disadvantages.

[_OScoder]

I don't think this is offtopic as your idea needs to be understood before anybody else can offer more opinions on its advantages and disadvantages.

Ok, if you say so

I'm not sure what you mean by "through the kernel stream". How would thread A block without a system call? The only way I can think of is if it caused a page fault.

Ok, this is kind of bringing unneeded complications, but the kernel will also be listening on a stream, which it can block on.

Huh? I thought the kernel woke up thread A... Are you assuming that threads A and B are on different processors?

Ok, in my design when a thread is woken, a variable is changed in that threads scheduling information, so when the scheduler next gets to that thread it is run instead of being ignored (so it makes no difference to the time thread B runs for.

You didn't really answer my original question of how thread A (the receiver) gets the data that thread B (the sender) was trying to write into the queue..

Sorry about that. Quite simply, it has to check for it. It has to attempt to read the queue every so often to see whether a message is there. I suppose it could be designed differently though?

Hope this makes sense now,
OScoder

P.S If I were to implement this I would most likely perform a compromise. Unofficial system calls could be used through the standard library for speed optimisation. Thanks also for your criticism so far, it has been helpful for thinking my design over!

[Colonel Kernel]

Thanks also for your criticism so far, it has been helpful for thinking my design over!

I'm glad it's been helpful... it's better for me to be helpful than just annoying and pedantic. ;D

Ok, this is kind of bringing unneeded complications, but the kernel will also be listening on a stream, which it can block on.

Does this stream have anything to do with the stream threads A and B are reading/writing to in your example, or is it a different stream altogether? Also, if the kernel "blocks" on a stream, that says to me that it would cause a page fault. I think your kernel would therefore have to handle page faults in kernel-mode as well. In this case, you have sort of a "kernel process" with "kernel threads" that can block, as well as the "real kernel" that runs in a purely event-driven way. This is pretty heavyweight stuff for a microkernel IMO.

Ok, in my design when a thread is woken, a variable is changed in that threads scheduling information, so when the scheduler next gets to that thread it is run instead of being ignored

Since threads "block" by causing page faults, this means that your scheduler would have to make sure the fault is resolved before resuming the thread, right? In other words, if thread A was writing to a page when it "blocked", the scheduler would have to change the permissions on that page to make it read-write before resuming thread A.

Sorry about that. Quite simply, it has to check for it. It has to attempt to read the queue every so often to see whether a message is there. I suppose it could be designed differently though?

That sounds like polling too. The latency could get pretty bad, plus polling wastes CPU time.

I would suggest instead just going with the traditional microkernel approach -- put your OS services in user-land processes, and have system calls for IPC. Otherwise, the rest of this just looks like a solution in search of a problem. Why are you trying to avoid system calls in the first place?

[0Scoder]

Does this stream have anything to do with the stream threads A and B are reading/writing to in your example, or is it a different stream altogether? Also, if the kernel "blocks" on a stream, that says to me that it would cause a page fault. I think your kernel would therefore have to handle page faults in kernel-mode as well. In this case, you have sort of a "kernel process" with "kernel threads" that can block, as well as the "real kernel" that runs in a purely event-driven way. This is pretty heavyweight stuff for a microkernel IMO.

It is a different stream altogether (I think I am failing to explain everything here correctly). Yes, this does mean there may have to be kernel threads, but isn't this an advantage? It means that functions implemented in the kernel could be implemented elsewhere at very little cost. Why would it make the kernel much heavier though? It can't be that costly to allow it to have it's own threads, and there has to be a part that handles exceptions anyway so the (relatively small) event driven bit cannot be escapen.

Since threads "block" by causing page faults

Ok, I have may have mis-explained here. Threads block by telling the kernel to block on them (through the kernel stream as mentioned before).

In other words, if thread A was writing to a page when it "blocked", the scheduler would have to change the permissions on that page to make it read-write before resuming thread A

Yes, that is the case. It would work as follows:
Thread A tries to write to stream A -> page is marked as read-write, so exception is called -> page-fault handler detects, through the memory location, that a stream caused the page fault and so forwards the information to the stream manager part of the kernel -> stream manager detects that stream was blocked on -> after checking permissions, marks streamA as read/write -> wakes up blocked threads -> resumes threadA

That sounds like polling too. The latency could get pretty bad, plus polling wastes CPU time.

Ok, that's why the blocking idea is there, so if a thread is waiting for something, it can block. however, it may not be the most efficient solution.

I would suggest instead just going with the traditional microkernel approach -- put your OS services in user-land processes, and have system calls for IPC.

Ok, I'll certainly try to take that into account!

Otherwise, the rest of this just looks like a solution in search of a problem. Why are you trying to avoid system calls in the first place?

Ok, at first my design was based on the concept of IPC through shared memory, where the kernel interferes very little (making it faster that having to invoke the kernel for each message pass occurs). This was intended for applications that transfere a large amount of data between each other (for example the different network layers) as opposed to events. Of course, as part of the design, I need to evaluate precisely how much kernel 'interference' would be usefull as opposed to counter-productive. I also wondered about extending my abstrction as far as possible, by making streams the ONLY way of communicating with anything, even the kernel. There are some reasons that the kernel should 'officially' be only accessible by streams, though they are beyond the scope of this post (since they would have to be explained as well).

Thanks,
OScoder

[Combuster]

How about the following implementation:

A user program has three (possibly sets) pages: one to write kernel commands to, two to read the results from

The user program will write the commands

Next it will try to read the results from one page (next time it will read the other one), but as this page is marked as unavailable, it will pagefault. The kernel now wakes the kernel thread

The kernel thread should check which pages have been accessed. The kernel thread then reads and parses each page, performing the operations, and loading the respective thread's result page, while unmapping the other (if its used). After the commands page is read, it is marked as not accessed to save time on the next iteration.

If the kernel sees there are no commands left to parse, it toggles its own sleep flag.
Then it tries to reads some invalid page in order to page fault and be scheduled out.

No explicit kernel calls will be required, other than the pagefault
the kernel may be configured to wake up the relevant thread on any page fault (thread number where the address should be, present cleared or supervisor only) and schedule the current thread. (the kernel might save recent pagefaults somewhere where the kernel threads can find it). I think this would be a relatively fast way of doing this.


Well, I suppose I should introduce my own OS while i'm already posting in here.
Mainly, i want it microkernel based, with some exokernel traits: Modular, applications can fine-tune their environment to their own liking

The two things that are not regular are these:
the userspace applications work like dll's - they may have their own threads, but they can also serve to be called by others (the 'drivers'). when a program requires another module it can decide to swap contexts on a call by means of a pagefault, but it can also decide to link in the dll (and its rights) entirely, so that you can get a program which does not need context swaps or anything to do what it wants. Of course there should be security checks on this mechanism, but it should be application dependent ( browser with very strict security, 3d games with the least security ), of couse you can tie down security for server applications.

secondly, the most innovative aspect: the gui
instead of making a desktop I'd for a change build a user environment in 3d where the user can walk around and do his daily work. Admitted, it is radical, but i haven't seen it been done before and i like trying new things. Console will of course be done first.

The only thing I haven't quite worked out yet is a replacement for its current name: 'MOS' (My Operating System)

Comments are of course appreciated

[earlz]

secondly, the most innovative aspect: the gui
instead of making a desktop I'd for a change build a user environment in 3d where the user can walk around and do his daily work. Admitted, it is radical, but i haven't seen it been done before and i like trying new things. Console will of course be done first.

I think that it might be a bit too radical, idk how you could possibly make it look professional

aside from that it sounds quite cool

[Warrior]

Besides the fact that I don't know how you'll get 3D I'll post my thoughts:

I believe 3D should be done as "Everything comes to you".

You want a document? Instantly have it.
You want a picture? Have it, edit it, etc.. all easily

The key word is me having to do less to get more.

I'm a very lazy user and I want to have as much done for me as possible while still being able to control how it's done if I so feel like it.

[pianohacker]

If you do get around to implementing this, remember Microsoft Bob, and look very, very carefully at it's mistakes. Remember of course, that this could be taken to mean that the virtual reality metaphor might not be a good one. After all, it's a fairly heavy interface over the actual computer.

I'd also like to propose my own idea (rather than double-post). One of the basic tenets of UNIX is that Everything is a File, including objects, devices, kernel interfaces, etc. What if that maxim were reversed? Suppose every thing is an object: devices/serial1, devices/disk1/partition1, control/kernel/acpi/battery, users/bill/, etc. File systems, USB drives, etc. could be loaded entirely inside their own directories, or on top of the existing namespace. The objects would have metadata, etc. Objects could either be created statically, or implemented through callbacks. Whaddya think?

[AndrewAPrice]

secondly, the most innovative aspect: the gui
instead of making a desktop I'd for a change build a user environment in 3d where the user can walk around and do his daily work. Admitted, it is radical, but i haven't seen it been done before and i like trying new things. Console will of course be done first.

A 3D console? You could try two methods:
a) Instead of splitting the screen into X and Y, split it into X, Y, and Z.
or
b) When the screen scrolls, the top line could move back into the monitor, and then you can watch the letters go further back, then loop down, and wrap up like a scroll of paper.