OSDev.org

Combuster wrote:Lets honour your request and make your life difficult in return:

Brendan wrote:No it's not that simple. Answer these questions:
[*]What actually determines who gets CPU time when (each individual process or the kernel)?

The end user. It chooses which tasks the computer should do and therefore what the computer must put time spending.

Combuster wrote:

[*]What actually determines when to read data, whether it be a high priority read or a prefetch, (each individual process or the kernel)?

The user and the process. The kernel can't know what to read until it is explicitly told to. If there's no file loaded ever, the kernel has no clue what to prefetch either.

Combuster wrote:

[*]What actually determines when a CPU should enter a power management state (each individual process or the kernel)?

When the user is around or not.

Combuster wrote:

[*]What actually determines when load should be migrated between CPUs (each individual process or the kernel)?

The end user?

Brendan wrote:Hi,

Combuster wrote:

This must be what makes "exo-kernel" so magical - it exists in 2 mutually exclusive states at the same time (possibly in some parallel universe full of pixies and unicorns), where processes are in control and the kernel isn't but at the same time the kernel is in control and processes aren't.

And you missed that I never mentioned that the boss was not in charge - the tasks of the boss simply changed. The boss is in charge of keeping things running, the tasks are in charge of how they use the resources given. Therefore they are both in charge of something. It is that simple.

No it's not that simple. Answer these questions:

• What actually determines who gets CPU time when (each individual process or the kernel)?
• What actually determines when to read data, whether it be a high priority read or a prefetch, (each individual process or the kernel)?
• What actually determines when a CPU should enter a power management state (each individual process or the kernel)?
• What actually determines when load should be migrated between CPUs (each individual process or the kernel)?

From what Rusky is saying; the kernel is what actually determines all of these things, and the processes themselves have no direct control over any of it (and only have indirect control via. suggestions/hints that the kernel is free to ignore and/or postpone). This makes perfect sense as the kernel is the only thing that has the information needed to make efficient global optimisations - each process is isolated and shouldn't care how many other processes are trying to use the CPU/s or read data from which devices or whatever else. Basically, the kernel is in charge and not the processes, and this makes perfect sense.

Now let's draw some diagrams:

Code: Select all

    _____________
   |             |
   |  Kernel and |
   |    Drivers  |
   |_____________|
        |
        | Kernel API
        |
        |   CPL=0
--------|-------------
        |   CPL=3
        |
    ____|________
   |             |
   | Library     |
   |_____________|
   |             |
   |  Process    |
   |_____________|

For this diagram, each process can have a different library (that uses the kernel API), where each different library can provide a different interface to the process (e.g. a POSIX interface, a Win32 interface, etc). Is this a diagram of a monolithic kernel (e.g. where the library is typically a standard C/C++/POSX library, but may not be), or is this a diagram of an exo-kernel?

Let's assume this is a diagram of a monolithic kernel. The Kernel API could be anything (it mostly only exists to keep the library happy; although it is possible for a process to use the bare kernel API without any library). For example, the kernel API could provide higher level abstractions and include things like "fopen()" and "malloc()" (where the library is a very minimal layer). The kernel API could also provide lower level abstractions - for example, it might only provide "open()" and "sbrk()" where the library has to implement "fopen()" and "malloc()" on top. For another example, the kernel API might provide extremely low level abstractions - e.g. it might only expose raw blocks and pages, where the library has to implement "open()" and "fopen()", and "sbrk()" and "malloc()" on top of that. None of this changes the fact that it's still a monolithic kernel - it's just a sliding scale of from "very high level kernel API abstractions" to "very low level kernel API abstractions". If fact nothing really prevents a monolithic kernel's API from providing multiple levels of abstraction at the same time - e.g. it could provide very high level abstractions in addition to providing a very low level abstractions.

Now; think about a monolithic kernel that provides a very low level abstractions via. its kernel API; and tell me how that is different to an exo-kernel. It's impossible for an exo-kernel to do anything that a monolithic kernel can't do, because they're essentially exactly the same thing.

The real question here is whether a monolithic kernel's API should provide very high level abstractions, or very low level abstractions, or something in between, or a mixture of high and low level abstractions. The most likely answer to this is "something in between" - if it made sense to provide very low level abstractions then all of the existing monolithic kernels would be providing those low level abstractions already (possibly in addition to their existing higher level abstractions, for the sake of compatibility); or at a minimum they'd be evolving towards lower level abstractions; so that they can give the "massive number of people" that write their own library (and don't just use the system's standard C/C++/POSIX library) a lot more flexibility.

Of course I'm been a little sarcastic - some people actually do write their own library instead of just using the system's standard C/C++/POSX library. Examples of this are the Wine project (implementing the Win32 API on *nix systems) and Microsoft's POSIX subsystem (implementing a standard C/C++/POSX library on Windows). It's not like you need very low level abstractions for this.

So what are the benefits of having very low level abstractions? The main benefit is, if you're a researcher and not writing a "real world OS" and don't need to care about malicious code, or compatibility, or features that people have grown to expect (e.g. like your web server being able to files stored on an NFS file system); then you can make it seem like low level abstractions provide better performance in some cases without acknowledging the fact that the sacrifices you've made to gain that extra performance are unlikely to be practical in a real OS.


Cheers,

Brendan

Brendan wrote:[*]What actually determines who gets CPU time when (each individual process or the kernel)?

Brendan wrote:[*]What actually determines when to read data, whether it be a high priority read or a prefetch, (each individual process or the kernel)?

Brendan wrote:[*]What actually determines when a CPU should enter a power management state (each individual process or the kernel)?

Brendan wrote:[*]What actually determines when load should be migrated between CPUs (each individual process or the kernel)?[/list]

• Who decides which disk blocks to read, write, or allocate, when and where? Can you do a zero-touch, all-files-at-once asynchronous-I/O-based cp?
• Who decides which physical pages to swap out, discard, or otherwise free (and how to do so)? Can you use an algorithm other than LRU to decide, depending on your application?
• Who decides how to handle application-caused interrupts/exceptions? (signal handlers decide too much performance-wise)
• Who decides the internals of the network stack? Is it the kernel, possibly providing zero-copy interfaces like sendfile/splice, or the applications, who can implement sendfile-like operations for even non-file situations? Is it the kernel, providing lots of ioctls and flags for things like delayed TCP ack, or is the application, who can just do it however is optimal (e.g. also combining the fin packet with the last data packet)? Is it the kernel, who has separate TCP retransmission buffer, or the application, who can retransmit directly from its own buffers?
• Who pulls the data out of/puts the data in network packets? (Linux packet mmap is a very exokernel-like interface) Can you implement precomputed checksums for zero-touch disk cache->network packet transmission? Can you discard rather than copy packets, skip checksum verification, etc. for stress testing tools? Can you implement a zero-overhead TCP/IP forwarder for e.g. a network load balancing algorithm?

Brendan wrote:For another example, the kernel API might provide extremely low level abstractions - e.g. it might only expose raw blocks and pages, where the library has to implement "open()" and "fopen()", and "sbrk()" and "malloc()" on top of that. None of this changes the fact that it's still a monolithic kernel - it's just a sliding scale of from "very high level kernel API abstractions" to "very low level kernel API abstractions". If fact nothing really prevents a monolithic kernel's API from providing multiple levels of abstraction at the same time - e.g. it could provide very high level abstractions in addition to providing a very low level abstractions.

Brendan wrote:if it made sense to provide very low level abstractions then all of the existing monolithic kernels would be providing those low level abstractions already (possibly in addition to their existing higher level abstractions, for the sake of compatibility); or at a minimum they'd be evolving towards lower level abstractions; so that they can give the "massive number of people" that write their own library (and don't just use the system's standard C/C++/POSIX library) a lot more flexibility.

Brendan wrote:Of course I'm been a little sarcastic - some people actually do write their own library instead of just using the system's standard C/C++/POSX library. Examples of this are the Wine project (implementing the Win32 API on *nix systems) and Microsoft's POSIX subsystem (implementing a standard C/C++/POSX library on Windows). It's not like you need very low level abstractions for this.

So what are the benefits of having very low level abstractions? The main benefit is, if you're a researcher and not writing a "real world OS" and don't need to care about malicious code, or compatibility, or features that people have grown to expect (e.g. like your web server being able to files stored on an NFS file system); then you can make it seem like low level abstractions provide better performance in some cases without acknowledging the fact that the sacrifices you've made to gain that extra performance are unlikely to be practical in a real OS.

Rusky wrote:

Brendan wrote:[*]What actually determines who gets CPU time when (each individual process or the kernel)?

The kernel provides CPU time, because the exokernel's job is to "securely multiplex hardware."

Rusky wrote:

Brendan wrote:[*]What actually determines when to read data, whether it be a high priority read or a prefetch, (each individual process or the kernel)?

Individual processes issue read schedules (for regular and prefetch reads) that are merged by the kernel.

Rusky wrote:

Brendan wrote:[*]What actually determines when a CPU should enter a power management state (each individual process or the kernel)?

The kernel "securely multiplexes hardware." It's not really to the advantage of any individual application to care about PM.

Rusky wrote:

Brendan wrote:[*]What actually determines when load should be migrated between CPUs (each individual process or the kernel)?[/list]

Individual applications can request specific CPUs, the kernel, "securely multiplexing hardware," hands out CPU time, and also notifies the applications so they can decide what to do with the particular CPU they're running on.

Rusky wrote:Let's look at some more questions:

Rusky wrote:The Xok benchmarks may be missing some features (although I'm still not sure what, the Cheetah vs Harvest certainly isn't), but it goes both ways- Cheetah implements several features not present in OpenBSD. There's also a slightly newer paper from 2002: http://pdos.csail.mit.edu/papers/exo:tocs.pdf It has a few more examples of exokernel-specialized applications. They also add IIS/NT to their Cheetah benchmark, which does use zero-copy, and which even their baseline Socket/Xok server outperforms on slower hardware.

Rusky wrote:

Brendan wrote:For another example, the kernel API might provide extremely low level abstractions - e.g. it might only expose raw blocks and pages, where the library has to implement "open()" and "fopen()", and "sbrk()" and "malloc()" on top of that. None of this changes the fact that it's still a monolithic kernel - it's just a sliding scale of from "very high level kernel API abstractions" to "very low level kernel API abstractions". If fact nothing really prevents a monolithic kernel's API from providing multiple levels of abstraction at the same time - e.g. it could provide very high level abstractions in addition to providing a very low level abstractions.

When you provide those "very low level kernel API abstractions" and still protect applications from each other, you are an exokernel. At that point, you may as well move the high level abstractions out of the kernel to 1) make them more easily modified.debugged, 2) reduce system call overhead, and 3) reduce the surface kernel attackers can target.

Brendan wrote:

Rusky wrote:Individual applications can request specific CPUs, the kernel, "securely multiplexing hardware," hands out CPU time, and also notifies the applications so they can decide what to do with the particular CPU they're running on.

There's no reason a monolithic kernel can't do this (but most don't/won't because it's much less able to adapt quickly/dynamically and has higher overhead).

Brendan wrote:

Rusky wrote:When you provide those "very low level kernel API abstractions" and still protect applications from each other, you are an exokernel. At that point, you may as well move the high level abstractions out of the kernel to 1) make them more easily modified/debugged, 2) reduce system call overhead, and 3) reduce the surface kernel attackers can target.

No; it makes you a monolithic kernel that implemented some lower level abstractions where it's beneficial in practice (rather than merely beneficial in some researcher's synthetic "apples to oranges" benchmark).

Brendan wrote:

Rusky wrote:The Xok benchmarks may be missing some features (although I'm still not sure what, the Cheetah vs Harvest certainly isn't), but it goes both ways- Cheetah implements several features not present in OpenBSD. There's also a slightly newer paper from 2002: http://pdos.csail.mit.edu/papers/exo:tocs.pdf It has a few more examples of exokernel-specialized applications. They also add IIS/NT to their Cheetah benchmark, which does use zero-copy, and which even their baseline Socket/Xok server outperforms on slower hardware.

A research paper that suggests existing monolithic kernels should implement broken networking with no security at all for the sake of performance (that does not say that exo-kernels are necessary for this, or that exo-kernels are good in any way whatsoever)?

Note that the paper says it compared Xok/ExOS to OpenBSD 2.2 (from 1997), and Microsoft's IIS version 2 (1996), and 200 MHz Pentium Pro CPUs (from 1995). The paper might be more recent (the copyright says 2002) but even if the paper is more recent I'd be willing to bet that their benchmarks were taken "as is" from old research done in 1997 and were not repeated on software and hardware from this century.

Rusky wrote:

Brendan wrote:

Rusky wrote:Individual applications can request specific CPUs, the kernel, "securely multiplexing hardware," hands out CPU time, and also notifies the applications so they can decide what to do with the particular CPU they're running on.

There's no reason a monolithic kernel can't do this (but most don't/won't because it's much less able to adapt quickly/dynamically and has higher overhead).

Actually it has less overhead. Instead of picking a thread and restoring its state to the newly chosen CPU, the kernel just jumps to a fixed entry point in the process, which can then make the thread decision itself and restore exactly the state it needs.

Rusky wrote:

Brendan wrote:

Rusky wrote:When you provide those "very low level kernel API abstractions" and still protect applications from each other, you are an exokernel. At that point, you may as well move the high level abstractions out of the kernel to 1) make them more easily modified/debugged, 2) reduce system call overhead, and 3) reduce the surface kernel attackers can target.

No; it makes you a monolithic kernel that implemented some lower level abstractions where it's beneficial in practice (rather than merely beneficial in some researcher's synthetic "apples to oranges" benchmark).

Now you're just arguing over definitions. "Exokernel" is defined as a kernel that avoids abstraction and instead provides low-level interfaces. The motivation doesn't matter- it's still using exokernel principles, and if it's a dominant characteristic of your design, it's an exokernel. I honestly don't care what you call it, as long as you understand what I mean when I say it.

Owen wrote:I'll also note that the choice of OpenBSD for comparison seems a little disingenuous. The OpenBSD project is not known for chasing performance - they're known for chasing correctness and security. If I were doing such a comparison, I'd want to include Linux, and maybe Solaris (both are known for fast networking stacks - especially Linux these days).

I'd also want to include a modern, high performance web server - say, nginx (also known for performance).

Brendan wrote:
So you mean it's a complete pile of trash with no support for sane pre-emption, has severe "denial of service like" problems at every possible opportunity, and does a virtual address space switch (large amount of overhead) to hopefully maybe if you're lucky avoid a few trivial register loads? Do they also do a task switch to a process to notify the process that it didn't get CPU time?

Combuster wrote:

Owen wrote:I'll also note that the choice of OpenBSD for comparison seems a little disingenuous. The OpenBSD project is not known for chasing performance - they're known for chasing correctness and security. If I were doing such a comparison, I'd want to include Linux, and maybe Solaris (both are known for fast networking stacks - especially Linux these days).

I'd also want to include a modern, high performance web server - say, nginx (also known for performance).

But also, it isn't fair to demand a comparison of a 15-year-old alternative kernel design to todays standards of kernels - especially that they have accumulated 15 years of bloaty guessware to make things act faster, if they haven't actually included some concepts from that design itself - a selection the optimisations you would typically expect in the libos have been added as alternative paths in the linux kernel, as well as entirely new close-to-hardware interfaces.

Combuster wrote:

Brendan wrote:So you mean it's a complete pile of trash with no support for sane pre-emption, has severe "denial of service like" problems at every possible opportunity, and does a virtual address space switch (large amount of overhead) to hopefully maybe if you're lucky avoid a few trivial register loads? Do they also do a task switch to a process to notify the process that it didn't get CPU time?

i.e. did not read.

Brendan wrote:confirmation bias and not proof at all.
(...)
I.e. getting bored with people that repeatedly fail to make any logical/sane argument for exo-kernels

Combuster wrote:I've already said all the things I considered needed to be said regarding exokernels. Now all that's left is dealing with your ad nauseam arguments - because all you're doing is being frustrated and repeating all your earlier arguments and fallacies in a different form.

Rusky wrote:You're still playing word games. Why are monolithic kernels allowed to make low-level interfaces and not be considered exokernels, while exokernels are not allowed to provide abstractions and still be called exokernels? Why are ideas that follow exokernel concepts not allowed to be considered exokernel-like just because they were invented before exokernels?

Rusky wrote:Scheduler activations are well-researched, and were abandoned because it was complex to support in an existing operating system alongside 1:1 threading, not because of poor performance or stability. Comments like "does a virtual address space switch to hopefully maybe if you're lucky avoid a few trivial register loads" and "Do they also do a task switch to a process to notify the process that it didn't get CPU time?" show that you do not understand what they are.

zeitue wrote:Questions:
Do Exokernels run all the user space stuff in just a single address space?
If you Exokernels run programs in individual address spaces then do the libOSes use only a single address space per libOS?

zeitue wrote:Thoughts:
You might take a look at the AmigaOS family, which are pretty much Exokernels, this includes AROS, AmigaOS, and MorphOS.
AROS, for example is implemented as a bunch of libraries which form the OS in user space and even exec which is which is their kernel is a library in user space. AROS programs will link themselves to exec.library at start, exec is mapped into each programs address space and this gives the program access to resources such as other libraries, memory, objects, ....

zeitue wrote:AROS is not just an Exokernel but also a hosted kernel, that's right AROS can run on either bare hardware or on top of a host like Linux or BSD or even Android. One advantage I can see in the Exokernel design is that you can code hosted or on bare hardware, so you can develop it faster using user space tools such as debuggers and tracers, as well as not having to reboot in between testing components. In addition libraries can be written hosted and used on both the chosen host and the Exokernel itself.