I keep recoding my OS!

[Korona]

I think I've written about this before, but: frequent iteration should always be prioritized over coming up with a great design from scratch. Of course, every once in a while, one should sit down and think about design and whether one's current design is on the right track. However, without frequent iteration and simply trying out things it is not possible to understand the limitations and difficulties of different designs. It is also important to accept that in any project, a large portion of the code is not in an optimal shape. That is not because programmers are lazy or because programmers are stupid (especially the latter is of course not true), it is just a consequence of the fact that it's impossible to understand all requirements when coding from scratch.

[nexos]

IMO language doesn't matter too much, I think using it good ol' plain C and bits of ASM is good. My main goal is speed, security, and stability. These may seem obvious, but stability / security and speed appear to be mutually exclusive. I want to try to mix them together as best as possible. Oh, I'll remember to read the Intel manuals before writing code .

[nexos]

I think I've written about this before, but: frequent iteration should always be prioritized over coming up with a great design from scratch. Of course, every once in a while, one should sit down and think about design and whether one's current design is on the right track. However, without frequent iteration and simply trying out things it is not possible to understand the limitations and difficulties of different designs. It is also important to accept that in any project, a large portion of the code is not in an optimal shape. That is not because programmers are lazy or because programmers are stupid (especially the latter is of course not true), it is just a consequence of the fact that it's impossible to understand all requirements when coding from scratch.

Exactly. I think I've done enough research into OSes and made enough research OSes to where I have the ability to make something good.

[eekee]

@Korona: I agree; I've (finally, after much protesting ) come to the same conclusions.

@nexos: Some languages are special, Lisp and Forth especially so. You can code an ordinary OS in these languages, perhaps with more effort, but being interactive and having simple syntaxes, I think they can enable scripting components together in a way which puts Unix to shame.

[nexos]

OK, I have a question, and no flaming here . For what I want, I am debating between a hybrid kernel and a microkernel. I want my kernel to scale well to large NUMA clusters, use asynchronous I/O, and support distributed systems. What would you choose for that kind of system? I also want good structure, and an interesting project, so I am leaning in the microkernel direction currently. Luckily, I plan to write the user space utilities (i.e., ls, cp, mv, rm, wc, and so on) plus a filesystem and partition managing app first, so I have time to figure this out .

[bloodline]

OK, I have a question, and no flaming here . For what I want, I am debating between a hybrid kernel and a microkernel. I want my kernel to scale well to large NUMA clusters, use asynchronous I/O, and support distributed systems. What would you choose for that kind of system? I also want good structure, and an interesting project, so I am leaning in the microkernel direction currently. Luckily, I plan to write the user space utilities (i.e., ls, cp, mv, rm, wc, and so on) plus a filesystem and partition managing app first, so I have time to figure this out .

Probably best to start with a micro kernel design, as you can always move parts into kernel space later (and make it a hybrid) if you have issues.

[nexos]

Good thought, @bloodline. I have been leaning towards a microkernel anyway, I'll go for that.

[bloodline]

Good thought, @bloodline. I have been leaning towards a microkernel anyway, I'll go for that.

I’m obviously biased as I prefer the micro kernel design, but I think it’s much easier to start with a micro kernel and then add monolithic features to it than start with a monolithic kernel and try and get it to exhibit micro kernel like behaviour...

If I’m not very much mistaken, all the major modern hybrid kernels (which I think covers almost every OS other than Linux) started out as pure micro kernel designs!

[thewrongchristian]

OK, I have a question, and no flaming here . For what I want, I am debating between a hybrid kernel and a microkernel. I want my kernel to scale well to large NUMA clusters, use asynchronous I/O, and support distributed systems. What would you choose for that kind of system? I also want good structure, and an interesting project, so I am leaning in the microkernel direction currently. Luckily, I plan to write the user space utilities (i.e., ls, cp, mv, rm, wc, and so on) plus a filesystem and partition managing app first, so I have time to figure this out .

You could call it NexHurd

[moonchild]

OK, I have a question, and no flaming here :) . For what I want, I am debating between a hybrid kernel and a microkernel. I want my kernel to scale well to large NUMA clusters, use asynchronous I/O, and support distributed systems. What would you choose for that kind of system? I also want good structure, and an interesting project, so I am leaning in the microkernel direction currently.

The main problem with microkernels is performance. However, they don't scale any worse than traditional kernels; so I think they would work well for your purposes.

[bloodline]

OK, I have a question, and no flaming here . For what I want, I am debating between a hybrid kernel and a microkernel. I want my kernel to scale well to large NUMA clusters, use asynchronous I/O, and support distributed systems. What would you choose for that kind of system? I also want good structure, and an interesting project, so I am leaning in the microkernel direction currently.

The main problem with microkernels is performance. However, they don't scale any worse than traditional kernels; so I think they would work well for your purposes.

Obviously I’m not an expert, but when I read the papers regarding poor performance of the micro kernel design vs a monolithic design, they focus on problems caused by having a single CPU. When you can have when servers and tasks can genuinely run concurrently (i.e. on an SMP Machine), these performance issues become negligible, and more than this, the design of the micro kernel lends itself to multiple threads of concurrent execution.

[nexos]

OK, I have a question, and no flaming here . For what I want, I am debating between a hybrid kernel and a microkernel. I want my kernel to scale well to large NUMA clusters, use asynchronous I/O, and support distributed systems. What would you choose for that kind of system? I also want good structure, and an interesting project, so I am leaning in the microkernel direction currently. Luckily, I plan to write the user space utilities (i.e., ls, cp, mv, rm, wc, and so on) plus a filesystem and partition managing app first, so I have time to figure this out .

You could call it NexHurd

I like it . It is kind of going to be like a modern GNU.

The main problem with microkernels is performance. However, they don't scale any worse than traditional kernels; so I think they would work well for your purposes

Yeah, I've been thinking about the performance a bit, and trying to mitigate some concern. I think PCID will help some, I plan on writing a user threading system integrated with kernel threads, and several scheduler optimizations to mitigate these concerns. However, a distributed monolithic kernel sounds hard . I think that is a microkernel's big advantage.

[thewrongchristian]

OK, I have a question, and no flaming here . For what I want, I am debating between a hybrid kernel and a microkernel. I want my kernel to scale well to large NUMA clusters, use asynchronous I/O, and support distributed systems. What would you choose for that kind of system? I also want good structure, and an interesting project, so I am leaning in the microkernel direction currently.

The main problem with microkernels is performance. However, they don't scale any worse than traditional kernels; so I think they would work well for your purposes.

Obviously I’m not an expert, but when I read the papers regarding poor performance of the micro kernel design vs a monolithic design, they focus on problems caused by having a single CPU. When you can have when servers and tasks can genuinely run concurrently (i.e. on an SMP Machine), these performance issues become negligible, and more than this, the design of the micro kernel lends itself to multiple threads of concurrent execution.

Aren't performance "problems" with early micro-kernels basically down to the fact they operated synchronously? That way, each message is guaranteed to result in a context switch, and all that that implies.

Even with asynchronous messaging, the problem then becomes one of implementing compatible APIs on top that are inherently synchronous, such as POSIX. But this translation can also be a benefit, as it gives scope to provide a more flexible user thread model if the OS interface is inherently asynchronous, so making N:M threading models not only easier to implement, but also natural.

And of course, when operating in messages, it doesn't matter where the message consumer lives, so it can be a kernel thread in the case of a hybrid kernel, or a user thread in the case of a purer micro-kernel. But in either case, you can scale across multiple CPUs to achieve performance through concurrency.

Doesn't this basically describe managarm?

[nexos]

The main problem with microkernels is performance. However, they don't scale any worse than traditional kernels; so I think they would work well for your purposes.

Obviously I’m not an expert, but when I read the papers regarding poor performance of the micro kernel design vs a monolithic design, they focus on problems caused by having a single CPU. When you can have when servers and tasks can genuinely run concurrently (i.e. on an SMP Machine), these performance issues become negligible, and more than this, the design of the micro kernel lends itself to multiple threads of concurrent execution.

Aren't performance "problems" with early micro-kernels basically down to the fact they operated synchronously? That way, each message is guaranteed to result in a context switch, and all that that implies.

Even with asynchronous messaging, the problem then becomes one of implementing compatible APIs on top that are inherently synchronous, such as POSIX. But this translation can also be a benefit, as it gives scope to provide a more flexible user thread model if the OS interface is inherently asynchronous, so making N:M threading models not only easier to implement, but also natural.

And of course, when operating in messages, it doesn't matter where the message consumer lives, so it can be a kernel thread in the case of a hybrid kernel, or a user thread in the case of a purer micro-kernel. But in either case, you can scale across multiple CPUs to achieve performance through concurrency.

Doesn't this basically describe managarm?

Yeah, that makes sense. I read an article a couple days ago about the only thing prevent computers with 100000+ nodes is POSIX I/O. Basically, it was saying POSIX I/O scales horribly, and I have to agree. Anyway, I think I have an architectural plan for my kernel, tell me any issues you see in it
So, my idea is an "everything is a message port idea". I'm going to revive the old Mach port idea, and message ports won't necessarily represent a message queue. It could represent something like a semaphore, a process control block, any resource inside the kernel is going to be a message port. A port representing one of these objects doesn't context switch, so it contains hardcoded function pointers called message handlers. The message manager takes the message ID, and uses that to find this messages messages handler, passing the message packet as a parameter. Now the object we sent the message to acts upon it, and returns, all without a context switch. Those are user mode to kernel mode messages. Then there are user mode to user mode messages, which work in the normal way. What do you say about that?

[PeterX]

You probably mean this article:
https://www.nextplatform.com/2017/09/11 ... -posix-io/
It's interesting read.

I like the general idea of Hurd, but I must admit that Mach makes me dizzy.

Greetings
Peter