OSDev.org

I am probably legend here for all the complicated IPC messaging design I've done, and shortly before my long pause in osdev, I finally implemented asynchronous messaging into my kernel, because I got fed up with all the nasty special cases that have to be taken care of, if synchronous system is used.

So, in case somebody actually happens to care (and I like using the forum for 'documentation'), I'm going to explain what I currently have, and how I'm going to change it. This is a big step for me, because once I'm happy with the messaging, I can move my attention (finally) to userspace things (like how am I going to run device drivers in userspace).

Currently, each thread has a messagelist for incoming messages. The service supports following calls:

- SendMessage(object, operation, w1, w2);

Send a message to object (stored in object owner's messagelist). Operation is 'nifgen' generated method hashcode, with two bits of type information, which allow "dynamic" typing of the <w1,w2> pair (immediates, long message, handle transfer, and a reserved transfer type).

SendMessage never blocks, but can fail if the messagelist is 'full', or some other nasty condition occurs (say, object invalidated).

- GetMessage(&message);

Copy (and remove) the first message from list. Wait for a message if none is available.

- PeekMessage(&message);

Same as GetMessage, but fails without blocking if none is available.

- WaitMessage(object, op, &message);

Same as GetMessage, but skips (and leaves into the list) messages not matching object and/or operation (there are wild cards for both, so GetMessage is really just WaitMessage(ANY, ANY, &message);

Then there's DispatchMessage(message), which handles message dispatching to object-specific handler function painlessly (look I got OO-messaging in kernel), if the receiver feels like using such a service.

Service thread thus looks something like this: Now, the system uses dynamic allocation of messages, so that it can deal with the WaitMessage(..) logic. Thanks to implementation details, the message object is always fixed sized (4 machinewords + list handling overhead).

WaitMessage(...) currently used in one place, scheduler_sleep(), which specifically wants to wait for MSG_kernel_timer to wake it up.

Ofcourse there is just one timer per thread, and this is also a bit questionable, because the timer has to bypass the official interface, because scheduler_sleep() doesn't bother implementing a full object, so the whole thing is a hack. Finally, this means that since SendMessage calls malloc() to allocate a message object, sending a message from the timer interrupt handler is strictly speaking 'dubious', especially with all the trouble to have a (nearly) O(1) timer system.

----

So, here comes today's listen to your brain lesson:

If I remove WaitMessage(..), and implement the timer wakeups separately (probably needs about 10 lines of code at most), I can instead have a fixed sized circular buffer per thread, allocated at thread creation time, allowing me to calculate fixed per thread costs, which is nice.

But it also means I never have to allocate anything just to send a (small) message. It might fail, but it might fail in any case. It also means I don't need to trash my heap with tons of small message objects, and it makes it easier to kernel to track of resource usage (for big, explicitly allocated things like threads, it's easy to set limits based on memory size or whatever). Not to mention that SendMessage() becomes truly constant-time which is nicer.

I can think of one other use for WaitMessage in kernel though, and that's paging wait. But if that needs another 10 lines of special support, then I need about 20 lines more special purpose code total. As for user processes, that can be reimplemented easily in userspace.

Conclusion: Sometimes special purpose code seems to be worth it, if it simplifies the general case.

Should give up with mu-character, as this board seems to hate it.

mystran wrote: Should give up with mu-character, as this board seems to hate it.

Works fine if you change character encoding to UTF-8

I don't know if the character encoding is specified (read: I'm too lazy to look) anywhere in the code for this board, but if it is, it isn't UTF-8.

(Sorry for lack of comment on design (if indeed that is what you're looking for), but I'm not anywhere close to that stage!)

Yeah well, I wouldn't mind comments, ofcourse, but this thread serves a purpose even if nobody comments, I guess.

Actually, it turns out, that scheduler_sleep(), the only function that currently needs WaitMessage(..) is:

- not a good way to sleep; you want to process MSG_kernel_timer from a thread event loop instead, waiting for messages yourself

- not well behaved: it trashes any previous timer you might have had; this could be fixed, but see previous point

- currently only used by 'ticker' (which spins a bar every second on screen, so you see the system is alive) and 'hogtask' which is a priority testing task that serves no real purpose.

So basicly, the required fix for getting rid of WaitMessage(..) is remove scheduler_sleep(), rewrite 'ticker', and either rewrite or remove 'hogtask'.

Old 'ticker' code:
New ticker code: Now it's event-driven, and not terribly more complicated. Ofcourse it's still not any more accurate than it used to be; it might lag arbitarily it if doesn't get scheduled immediately, but it's just a livelyness indicator and I'm feeling lazy.

Oh, and if somebody looks at my code and wonders, then yes, there's a special case: kernel sends certain messages (currently timer expirations) to threads with 'object' set to zero.

It does this bypassing the SendMessage(...) interface, which would normally set the target thread to object owner. This is by design, allowing certain messages to be sent to thread itself (instead of an object it owns).

Which reminds me: I should probably make a clear (kernel internal) interface for doing that, so that it doesn't require touching the lower-level messaging routines.

Just a quick note, aren't peekx routines normally used for grabbing something from x but not removing it?

Actually, the GetMessage(...) / PeekMessage(...) naming is basicly identical to that used by Microsoft Windows.

Since my messaging is probably closest to that of Win32 (from popular systems anyway), I didn't see a point in trying to come up with better naming.

Actually, Win32 PeekMessage(...) can be configured to either remove, or not remove messages. I might support that at some point, but for now PeekMessage(...) is just a non-blocking version of GetMessage(...), unless someone comes up with a really good alternative (better than TryGetMessage(...) anyway).

You could have just one function, GetMessage, with a timeout parameter. 0 means return immediately on success or failure, 1 to MAX_NUM-1 means wait up to that many milliseconds for a message to be available, and MAX_NUM (whatever it is based on the type you're using) means to block indefinitely until a message is available. Then you get three different types of functionality available under one interface with no ambiguity about what Peek means and whatnot.

I know. In fact, the system call that implements GetMessage has a timeout parameter, and if you set it to 0, you get PeekMessage instead.

The reasons for having separate functions are practical:

- when you see a GetMessage/PeekMessage, you always know if it can block or not.

- with two different functions, you can easily grep for GetMessage/PeekMessage. This is very useful.

In any case, you shouldn't rely in message order anyway, so you can just resend the message to yourself if you want to keep it in queue. Since kernel-level messaging just copies 4 machinewords (4*4bytes), this isn't terribly expensive.

In userspace there's going to be another queue internally (to allow batch-transfer of messages and selective waiting and whatnot), so there's no performance issue there either (and one can have that non-removing PeekMessage, ofcourse).

As it is, user processes aren't allowed to block indefinitely. If you specify (~0) to have the maximum timeout, you will get MSG_kernel_timer after about 497 days, if you don't receive some other message before that.

Now I'd just have to figure out how am I going to match requests to replies, when that isn't obvious (eg. there's one timer per thread, so no need to match anything).

As it is, currently you create an object, send it with the request, and have reply sent to that new object. But it would be nice if one didn't have to create a new object for every single message that expects a reply (which has to match request).

Sending replyID would be the obvious solution, but I need the identifiers unforgeable, especially since the receiver doesn't know how sent a message, just that the sender must have had a capability to the target object.

Oh well.

Associate a target object with each message, but keep the target object ID or whatever in kernel space so that the user program only knows it is replying to a certain message.

As far as kernel space messages go I wouldnt try to prevent forging there, but its doable for user space.

Hmmh, so, basicly, I could:

1. allocate a temporary one-time object (objectID really) in send systemcall, and return the id to sender as return value.

2. send that reply-object as part of the message

3. automatically destroy the object when it's used

Ok, so only theoretical problem is some process never responding to anything, but even then it'll run out of handle-space (which is intentionally finite), at which point it has to destroy the objects, which constitutes some type of reply, I guess.

Bonus: allow reply-objects to be sent to other processes just like normal objects (basicly, don't bother trying to prevent it) and we can have proxy-objects forward messages, without ever having to process the reply.

Sounds like a plan. In fact, sounds like a pretty good plan. Remaining question is actually just whether a reply should be a semi-normal message, or a special reply message. But I guess that just needs about 15 minutes of brainwork.

My kernel can by definition "forge" anything it wants. Anything running under the kernel has to trust it anyway.

mystran wrote: Actually, Win32 PeekMessage(...) can be configured to either remove, or not remove messages. I might support that at some point, but for now PeekMessage(...) is just a non-blocking version of GetMessage(...), unless someone comes up with a really good alternative (better than TryGetMessage(...) anyway).

PostMessage is basically the non-blocking way in Microsoft, maybe the name is more suited. As I understand with Microsoft's version they use PeekMessage with DispatchMessage as a pair to basically a poll and dispatch off queued or as they say posted messages.