Open source Midori?

[Candy]

I think some of you may have seen discussions flying around about Midori. It is hitting *very* close to home for me; it's practically the exact design I had in mind and that I was working out the last year or two.

Given that it has a lot of the same choices, I think it has a good theoretical basis to continue on. So is anybody else inspired by their proposals and if so, would there be enough interest to start up a group project for an open source Midori OS?

http://joeduffyblog.com/2015/11/19/asyn ... verything/ plus previous posts.

[Candy]

... No interest at all?

[Boris]

Hello, I just read the article. I'm quite shoked actually, because I wanted to try OS development to try a task oriented kernel architecture. ( with primitives like continueWith and cancellation tokens)
IMHO, asynchronous tasks are powerful, but often misused.
In my kernel, I wanted asynchronous tasks for everything that has an undefined time of execution, or thing that could fail if we wanted immediate resolution.

For example, I wanted the sbrk function to take a token of cancellation, meaning the sbrk function could block until physical memory become available or until the application decide it has waited too much.

Unfortunately for you, im new in OS development. for now, I have only a two part kernel, the first part activates paging and loads the second part. But I'm interested.

[Combuster]

I think time is my personal issue, rather than it being a project I would not see myself work on.

[dschatz]

I am interested in many of the ideas worked on in Midori. In particular, I'm a fan of capability-based authority and async interfaces. I think a big drawback with their design is that it cannot support legacy software - they use (compiler-enforced) software isolation, and cannot support blocking code. I'm looking into ways to get the benefits without these drawbacks.

[Brendan]

Hi,

I am interested in many of the ideas worked on in Midori. In particular, I'm a fan of capability-based authority and async interfaces. I think a big drawback with their design is that it cannot support legacy software - they use (compiler-enforced) software isolation, and cannot support blocking code. I'm looking into ways to get the benefits without these drawbacks.

When Midori's developers say "everything is non-blocking" what they really mean is that waiting for a message to arrive is the only thing that causes a thread to block; and because everything waits for messages to arrive everything is blocking.


Cheers,

Brendan

[SpyderTL]

I had not realized that Midori had been "abandoned", although I had pretty much assumed it because of the lack of updates over the past few years.

One of the reasons I started my own project was because MS had abandoned their Singularity project, and I wanted to see what a "managed" OS would look like. So I started with the idea of an Object based OS, and worked backwards to figure out the quickest way to get there. My biggest goal was to be able to allow the end user to create and manipulate objects from a text-based command prompt, and then expand from there. Once the user could create objects and call methods on them, the rest has been pretty much just creating more classes to talk to more hardware, and to perform common tasks like downloading data from the Internet, and parsing file system tables.

I'm pretty happy with the way things have worked out, but there's still a ton of work to do, and I've still got a lot of ideas that I haven't even started implementing yet. But, let us know what you decide on your project.

[dschatz]

When Midori's developers say "everything is non-blocking" what they really mean is that waiting for a message to arrive is the only thing that causes a thread to block; and because everything waits for messages to arrive everything is blocking.

The article says

Because in Midori the only way to "wait" for something was to use the asynchronous model, and there was no hidden blocking

My interpretation of this is that you cannot "wait for messages" but rather you provide an explicit dataflow to occur when a message arrives. So suppose I want to port code that uses a synchronous read(). As a developer, I need to implement read() on top of the provided asynchronous read, but they don't provide any blocking wait. I see no way to preserve the stack until the asynchronous read is complete.

Now this has its advantages - namely, when I invoke some component, I can be reasonably certain that control will be returned to me in a timely matter. But a drawback is that it cannot support legacy, blocking code.

[Combuster]

But a drawback is that it cannot support legacy, blocking code.

You'd have to go to quite some lengths to get a compiler to convert all I/O calls to continuations (or run a vm under the same merit), but supporting legacy is not strictly impossible.

Then again, getting rid of some legacy can be a good thing.

[dschatz]

You'd have to go to quite some lengths to get a compiler to convert all I/O calls to continuations (or run a vm under the same merit), but supporting legacy is not strictly impossible.

Then again, getting rid of some legacy can be a good thing.

I don't think it's sufficient for a compiler to convert all I/O calls. It also needs to convert all function calls in a call stack that may block. Basically, turning blocking code into non-blocking code requires duplicating the entire call chain around the block. This is sometimes called "stack-ripping". Using a VM is feasible but makes it difficult to compose legacy code with new code (e.g. use existing libraries). In general, though, I agree, getting rid of some legacy is good, but my own philosophy is that a successful approach will be incremental, not reboot_the_world().

[Candy]

You'd have to go to quite some lengths to get a compiler to convert all I/O calls to continuations (or run a vm under the same merit), but supporting legacy is not strictly impossible.

Then again, getting rid of some legacy can be a good thing.

I don't think it's sufficient for a compiler to convert all I/O calls. It also needs to convert all function calls in a call stack that may block. Basically, turning blocking code into non-blocking code requires duplicating the entire call chain around the block. This is sometimes called "stack-ripping". Using a VM is feasible but makes it difficult to compose legacy code with new code (e.g. use existing libraries). In general, though, I agree, getting rid of some legacy is good, but my own philosophy is that a successful approach will be incremental, not reboot_the_world().

I've given some thought how to port GCC to such a system. GCC has blocking operations - when you start thinking about it even printf() and read() must be asynchronous - so to support it you have to be able to block.

You basically invert the common paradigm of implementing tasks on a thread pool - you implement threads as tasks that have their own stack ownership and that manually switch back to the regular stack when suspending, and have a continuation attached that switches back & continues the thread. Then you have a thread that is implemented as a sequence of tasks. You also make wrappers for all asynchronous functions that are synchronous and that link to the thread suspension mechanism - basically, make async call, attach continuation for resume, return.

You do a similar thing for new applications - you replace main() with a task entry point task_main() which takes a shared_ptr to the application as argument (or equivalent in whatever your language is). When that application object is deallocated everything is cleaned up for that application instance. That allows you to have literally zero threads in most applications.

[dschatz]

I've given some thought how to port GCC to such a system. GCC has blocking operations - when you start thinking about it even printf() and read() must be asynchronous - so to support it you have to be able to block.

You basically invert the common paradigm of implementing tasks on a thread pool - you implement threads as tasks that have their own stack ownership and that manually switch back to the regular stack when suspending, and have a continuation attached that switches back & continues the thread. Then you have a thread that is implemented as a sequence of tasks. You also make wrappers for all asynchronous functions that are synchronous and that link to the thread suspension mechanism - basically, make async call, attach continuation for resume, return.

You do a similar thing for new applications - you replace main() with a task entry point task_main() which takes a shared_ptr to the application as argument (or equivalent in whatever your language is). When that application object is deallocated everything is cleaned up for that application instance. That allows you to have literally zero threads in most applications.

This is what stackful coroutines are. To be explicit, this is a user-level thread with much of the associated problems therein - e.g. how does mmap()ing a file work? What happens when a coroutine page faults? Typically the kernel is unaware of the user-level threading and suspends the entire (kernel) thread despite the fact that other coroutines are available.

[Combuster]

mmap()

Of course, there's no point in discussing issues with synchronous system calls on a kernel that has no synchronous system calls

[dschatz]

mmap()

Of course, there's no point in discussing issues with synchronous system calls on a kernel that has no synchronous system calls

Just to be clear, it doesn't much matter if the call to mmap() itself is synchronous or asynchronous, but the demand paging aspect of it is where the problem lies. The solution, of course, is to implement mmap() as if mlock() occured afterwards so that faults cannot happen. This comes with performance issues.

[Brendan]

Hi,

When Midori's developers say "everything is non-blocking" what they really mean is that waiting for a message to arrive is the only thing that causes a thread to block; and because everything waits for messages to arrive everything is blocking.

The article says

Because in Midori the only way to "wait" for something was to use the asynchronous model, and there was no hidden blocking

My interpretation of this is that you cannot "wait for messages" but rather you provide an explicit dataflow to occur when a message arrives. So suppose I want to port code that uses a synchronous read(). As a developer, I need to implement read() on top of the provided asynchronous read, but they don't provide any blocking wait. I see no way to preserve the stack until the asynchronous read is complete.

My interpretation is that you can "wait for messages" (and it does block), but they've obfuscated the event loop under syntactical sugar ("await") in an attempt to make the actor model seem like procedural programming.

Now this has its advantages - namely, when I invoke some component, I can be reasonably certain that control will be returned to me in a timely matter. But a drawback is that it cannot support legacy, blocking code.

For legacy non-blocking code, I think you'd just do something like (e.g.) "fp = await Async.Open(filename);".


Cheers,

Brendan