Who wants to help me create an operating system?

[pippofranco]

It must be a microkernel operating system written completely in rust

As a userland it must have a Cocoa reimplementation like gnustep

And it needs to be optimized from the start for mobile and desktop devices

And then it has to be written for a risc-v architecture with extensions to have fast IPCs

So it will exceed the speeds of a monolithic kernel and you can make debugging quite easy

And it must be unix-like.

This way there shouldn't be any hardware components with their own firmware because each specialized chip interfaces directly with the CPU: for example, instead of having a single WiFi component, we have an RF and separately an ASIC that manages the WiFi protocol and they communicate with each other them through the CPU, so as to have a computer where everything is managed by the kernel and you can control up to the single status LED of the ethernet connector directly from the main operating system. The same can be extended to all the smart devices in the structure, which connect together forming a seamless cluster so as to have one large computer that does everything.

And a thousand other innovative things. All licensed under AGPL-3.

[Octocontrabass]

Sounds interesting. How much of it have you built so far?

[BigBuda]

What you describe in the last big paragraph is not building an OS but an entire computer architecture.

Besides, I don't think there is any advantage on having the CPU do everything and not letting the hardware manage itself. It'll be slow, it'll be computationally expensive, it won't be energy efficient, it will add a lot of latency.

[iansjack]

As a userland it must have a Cocoa reimplementation like gnustep

That's going to be incredibly slow if you aren't using the video card's firmware.

I'm not convinced that aiming at a single architecture, especially a custom one, is the best plan for a modern operating system.

A first implementation, or some detailed specs/roadmap, would increase your chances of getting others to join your project.

[kerravon]

Sounds interesting. How much of it have you built so far?

Brutal

[devc1]

How is it re-enovative to recreate an existing-like OS ? unix-like, I just want to understand why alot of people on these forums are trying to re-create linux/unix ?

How is it gonna be faster if it is the same OS : )

And who does use RISC-V anyway, I've never heard of this architecture until the day I entered this forum.

+ there are plenty of extensions to use in x64 for speeding up the IPC protocol, I recommand requiring a CC-NUMA System as most of the recent computers and mine (which is 13 years old) support that, it will remove the multiprocessor caching problem for you.

These questions are not meant to offend anyone, I just want an explanation, thanks,

And to mention, props to the cacawa system:)

Imagine turning on the computer and it says : Welcome to the CACAAWA userland

[BigBuda]

And who does use RISC-V anyway, I've never heard of this architecture until the day I entered this forum.

RISC-V is fairly recent in the computer ISA scene, but it's quickly gaining traction. In the last few 5-10 years has gained support from major companies like Nvidia and Western Digital, for instance. I think I even heard a rumor about MS launching their own RISC-V based CPU.

[kerravon]

How is it re-enovative to recreate an existing-like OS ? unix-like, I just want to understand why alot of people on these forums are trying to re-create linux/unix ?

I'm not one of them, so I can't answer authoritively. But as far as I can tell, the answer is NIH (not invented here). One person admitted that when I asked. I think most people don't want to admit that, so they come up with a line of bullshit.

[nullplan]

How is it re-enovative to recreate an existing-like OS ? unix-like, I just want to understand why alot of people on these forums are trying to re-create linux/unix ?

I'm not one of them, so I can't answer authoritively. But as far as I can tell, the answer is NIH (not invented here). One person admitted that when I asked. I think most people don't want to admit that, so they come up with a line of bullshit.

With that in mind, here's my line of BS: I want to see how little code is really needed to implement a UNIX-like kernel. I see the Linux source code has bloated to epic proportions, but most of that is in the drivers directory. But even the non-drivers stuff suffers from over-generalization from being oh so configurable, and of course from legacy cruft. Implementing a new kernel entirely allows me to get rid of legacy cruft like you wouldn't believe.

The thing is, the unixoid kernel has been criticized before as being a bad ideal to aspire to, but given that every mainstream OS implements the same idea (yes, Windows and Mac OS differ from Linux in some details, but for the most part they implement the same thing, with processes in one part of address space, and the kernel in another; with the MMU abstracting memory accesses; with hardware being abstracted behind kernel APIs of some sort, with a VFS all the processes communicate with through time and to each other, &c. pp.), the idea must have some merit. In particular, I never really heard about a better way. The Android security model for the most part just extends the idea of a user to mean an application running for some user (so the UID of this browser process is not just identifying "nullplan" but rather "firefox running on behalf of nullplan").

So, in any case, I really haven't heard anyone come up with a better way. People say the idea is half a century old now, but being old is not the same as being obsolete. The microkernel guys implement the same idea, but with some kernel calls as RPCs into other servers instead. RDOS really really likes his segmentation, but is, to my knowledge, not really implementing anything different from what was described above, only extended through one more layer of indirection. All the other OSes I've ever looked into try the same thing again, with different APIs, and sometimes the VFS isn't unified, like what you'd get under a UNIX with chroot and the proper bind mounts in place. The end result is always UNIX or something so similar that describing the differences means listing minutiae.

Or to put this yet another way, a couple of weeks ago, someone asked what the difference between a library and a kernel is, and iansjack said there really wasn't one. The kernel is a library for doing things you need elevated privilege for, and that is basically it. And I have not seen a single OS anywhere where the kernel-as-library idea would not fit. That is all I can see wherever I look. So given that everyone is copying the same fifty year old design, I might as well do it right and not hobble it with my misconceptions of security.

[Barry]

How is it re-enovative to recreate an existing-like OS ? unix-like, I just want to understand why alot of people on these forums are trying to re-create linux/unix ?

I'm not one of them, so I can't answer authoritively. But as far as I can tell, the answer is NIH (not invented here).

I think it's kind of like how most architects just copy that whole "four walls and a pointy roof" design. By "unix-like" most people usually refer to the interface the OS presents, which still gives you a huge space to implement it in. You could implement a unix-like microkernel based OS, or use a hybrid kernel. It'll be very different to the original Unix implementation, but it's still "unix-like".
Having said that, I do think part of the fun of developing your own operating system is adding your own ideas and concepts and having something that's truly yours. I like writing programs for my OS and actually having to do things, and be able to do things, differently to other operating systems. But a lot of the core concepts of Unix are pretty solid. So much so, in fact, that we no longer refer to them as "unix-like", because they were adopted by all remaining operating systems. Plus, it makes it easier to port programs

[rdos]

The thing is, the unixoid kernel has been criticized before as being a bad ideal to aspire to, but given that every mainstream OS implements the same idea (yes, Windows and Mac OS differ from Linux in some details, but for the most part they implement the same thing, with processes in one part of address space, and the kernel in another; with the MMU abstracting memory accesses; with hardware being abstracted behind kernel APIs of some sort, with a VFS all the processes communicate with through time and to each other, &c. pp.), the idea must have some merit. In particular, I never really heard about a better way. The Android security model for the most part just extends the idea of a user to mean an application running for some user (so the UID of this browser process is not just identifying "nullplan" but rather "firefox running on behalf of nullplan").

So, in any case, I really haven't heard anyone come up with a better way. People say the idea is half a century old now, but being old is not the same as being obsolete. The microkernel guys implement the same idea, but with some kernel calls as RPCs into other servers instead. RDOS really really likes his segmentation, but is, to my knowledge, not really implementing anything different from what was described above, only extended through one more layer of indirection. All the other OSes I've ever looked into try the same thing again, with different APIs, and sometimes the VFS isn't unified, like what you'd get under a UNIX with chroot and the proper bind mounts in place. The end result is always UNIX or something so similar that describing the differences means listing minutiae.

Or to put this yet another way, a couple of weeks ago, someone asked what the difference between a library and a kernel is, and iansjack said there really wasn't one. The kernel is a library for doing things you need elevated privilege for, and that is basically it. And I have not seen a single OS anywhere where the kernel-as-library idea would not fit. That is all I can see wherever I look. So given that everyone is copying the same fifty year old design, I might as well do it right and not hobble it with my misconceptions of security.

I think you are generalizing principles too far. UNIX has some very particular ways to deal with processes, VFS, libraries and syscalls so it's not a fact that UNIX once defined these and now everybody are just coping their concepts.

For one, UNIX has a very ugly fork concept, and Windows (and RDOS) does not normally create processes with fork. They both have a CreateProcess function. Both these OSes had a need to implement fork on top of this, but that's legacy cruft not copying. Fork in UNIX also have implicit requirements for what functions must exist in the paging environment, like reference counting and copy-on-write. UNIX also has the ugly concept of "exec" which is built-into compilers.

Threads were not even part of UNIX to start with, and for a long time were only user mode features.

In the file system area, UNIX abstracts everything as files, and so even devices that has nothing to do with files still are mapped to file handles. This is really ugly too, but if you want some UNIX compatibility, you need to cope with this one way or another. So, the existence of "everything is a file" concept in no way implies you are writing a UNIX clone, unless you really handle it that way at the core.

The VFS abstractions in UNIX are not very smart either. They won't allow effective file system optimizations. Neither do they have a good match with the micro-kernel way of implementing file system services. My new VFS has no resemblance to the UNIX VFS, so I can't say I'm copying it.

The signal concept is not worth copying either, but if you want Posix compatibility, you need to add some legacy stuff for this too.

Syscalls are typically implemented through a central service (interrupt or processor specific syscall instruction) in UNIX. This is inefficient because function numbers must be coded at the caller side and decoded at the kernel side. I use call gates instead, which avoid both coding & decoding. Unfortunately, Intel and AMD optimized for OSes that used central service points.

[nullplan]

See, this is what I mean by "Listing the differences means listing minutiae".

For one, UNIX has a very ugly fork concept, and Windows (and RDOS) does not normally create processes with fork.

You are currently arguing against the specific way in which processes are created, but not against the concept of processes. The only critiques of that concept I have ever heard came from the unikernel crowd, who don't want to make a general purpose OS. fork() or spawn() makes little difference in the end, although spawn() is less general. I have in the past argued against fork() myself, but I think I will have to implement it since some programs use fork() not to spawn a program, nor to background themselves, but to do something else entirely. I will however go for the more general clone() concept, which allows creating a new process in the same address space. Which is its own can of worms, but managed properly it can allow spawning of child processes without setting the pages to copy-on-write.

Fork in UNIX also have implicit requirements for what functions must exist in the paging environment, like reference counting and copy-on-write.

Reference counting on files is a must anyway, since an unlinked file can only be deleted once the last reference is removed. Windows does the same thing, ever since they let multiple processes open the same file. Difference is they don't let you remove an open file. How, pray tell, do you think they implement that requirement?

And copy-on-write is an implementation detail. You can get by without it, although your memory allocation will be much worse.

Threads were not even part of UNIX to start with, and for a long time were only user mode features.

Implementation detail. Threads are defined in POSIX, and as long as the implementation complies with the spec, whether threads are a kernel creation or a userspace idea should not matter except for performance. And I don't really care about performance.

In the file system area, UNIX abstracts everything as files, and so even devices that has nothing to do with files still are mapped to file handles.

Device files are apparently a contentious part of UNIX, but bespoke OS-internal handles for each service aren't that different, especially when the file handles in UNIX then are only used in ioctl() calls. But file handles are easy to use, have a defined life time, can be transmitted via UNIX domain socket, can be inherited through fork(), and whether they can be created or not depends solely on file permissions which are easy to check and adjust as needed. I can easily set the owning group of all sound devices as the group "sound", and deny access to the files to anyone not in that group, and then only users in the "sound" group can access the card. Or I can allow world access to the files and everyone gets a go. I cannot do the same with OS-internal APIs.

But device files and pseudo-files were not my point. I was talking about regular files and directories, which every OS has had since CP/M, and they continue to exist, because they are a useful abstraction. And we continue to interact with them through open(), read(), write(), and close(), or close approximations of these. I have yet to see an OS allow you to insert data into the middle of a file.

The signal concept is not worth copying either,

I disagree. Every OS I have seen so far that has not had signals has had to implement an inferior copy of signals to implement catchable exceptions e.g. on memory access violation. Inferior because there was not a general API that could be used, although the effect would have been the same. Instead a handler could be installed in some special way for just those exceptions. With no signals, no concept of signal-safety was defined in the OS, and so it was not clear what APIs the handler was allowed to call. In practise you just did as you would for a signal handler, avoiding obvious traps like printf(), but opening a crash log to write out the information.

And there is really nothing quite like a signal to tell a group of processes about an event that just happened. At work I deal with a software suite for an embedded device, and we use multiple processes for resilience. When the config is changed, all processes need to be notified to reload the config, and I would really rather they didn't have to poll a configuration date or something.

Syscalls are typically implemented through a central service (interrupt or processor specific syscall instruction) in UNIX. This is inefficient because function numbers must be coded at the caller side and decoded at the kernel side. I use call gates instead, which avoid both coding & decoding. Unfortunately, Intel and AMD optimized for OSes that used central service points.

Minutiae. I mean, look at it. You are currently arguing that whether the system call entry point is centralized or different for each one is a change in design philosophy.

[rdos]

You are currently arguing against the specific way in which processes are created, but not against the concept of processes.

Not only how they are created, but also how they are managed. By allowing fork and sharing pages between many processes with possibly a long inheritance chain will bloat the page management code. By allowing this you need reference counting on every page in user space, which is a major problem. I decided I could focus that uglyness only on forked processes, and so the normal process have no reference counting for pages.

The only critiques of that concept I have ever heard came from the unikernel crowd, who don't want to make a general purpose OS. fork() or spawn() makes little difference in the end, although spawn() is less general. I have in the past argued against fork() myself, but I think I will have to implement it since some programs use fork() not to spawn a program, nor to background themselves, but to do something else entirely. I will however go for the more general clone() concept, which allows creating a new process in the same address space. Which is its own can of worms, but managed properly it can allow spawning of child processes without setting the pages to copy-on-write.

None of that is really needed other than to support legacy code. It's perfecly possible to define a memory sharing mechanism between processes without needing to resort to fork.

Implementation detail. Threads are defined in POSIX, and as long as the implementation complies with the spec, whether threads are a kernel creation or a userspace idea should not matter except for performance. And I don't really care about performance.

Not naming threads is not an implementation detail. I could support POSIX threads, but I rather not since Posix lacks thread names. That would mean I would have to use thread names like "Thread 1233" or something. Without names on threads, I have some ID that I have no idea how it links to code. My process info include thread names, which makes the dumps much more informative. The thread names are listed in the application debugger and kernel debugger as well.

Device files are apparently a contentious part of UNIX, but bespoke OS-internal handles for each service aren't that different, especially when the file handles in UNIX then are only used in ioctl() calls.

I don't support ioctl, and never will. That's perhaps the ugliest legacy of UNIX.

But file handles are easy to use, have a defined life time, can be transmitted via UNIX domain socket, can be inherited through fork(), and whether they can be created or not depends solely on file permissions which are easy to check and adjust as needed. I can easily set the owning group of all sound devices as the group "sound", and deny access to the files to anyone not in that group, and then only users in the "sound" group can access the card. Or I can allow world access to the files and everyone gets a go. I cannot do the same with OS-internal APIs.

Single user systems have no need for the access control of ancient centralized solutions. I mostly see that as legacy too. The important issues today are more like who is allowed to use your camera, microphone and listen to your keyboard and recording touch screen activity. File permissions will not solve that.

But device files and pseudo-files were not my point. I was talking about regular files and directories, which every OS has had since CP/M, and they continue to exist, because they are a useful abstraction. And we continue to interact with them through open(), read(), write(), and close(), or close approximations of these. I have yet to see an OS allow you to insert data into the middle of a file.

Perhaps, but they forgot to name threads and syscalls.

I disagree. Every OS I have seen so far that has not had signals has had to implement an inferior copy of signals to implement catchable exceptions e.g. on memory access violation. Inferior because there was not a general API that could be used, although the effect would have been the same. Instead a handler could be installed in some special way for just those exceptions. With no signals, no concept of signal-safety was defined in the OS, and so it was not clear what APIs the handler was allowed to call. In practise you just did as you would for a signal handler, avoiding obvious traps like printf(), but opening a crash log to write out the information.

For exception handling, I use the Windows method that is implemented as part of PE exceptables. Exception handling is important, but it is not related to signals. Signals are best defined as part of the syscall API, either for general usage or for specific device functions.

Syscalls are typically implemented through a central service (interrupt or processor specific syscall instruction) in UNIX. This is inefficient because function numbers must be coded at the caller side and decoded at the kernel side. I use call gates instead, which avoid both coding & decoding. Unfortunately, Intel and AMD optimized for OSes that used central service points.

Minutiae. I mean, look at it. You are currently arguing that whether the system call entry point is centralized or different for each one is a change in design philosophy.

That's not the only difference. In the application code, syscalls are coded as far calls through the null-selector which causes a GPF. The kernel then typically will patch the code with a call gate, but it could also decide that the application is not allowed to access the camera API, and so will patch it to a null-call that fails. IOW, this allows the OS to control which syscalls an application can use, without any penalty in performance.

Another feature of the RDOS syscall interface is that the debugger can display the name of the syscall instead of a far call with a selector. Every syscall server must define the entry-point & name of syscalls it exports.

Device drivers in kernel can also use syscalls. They work both in user space and kernel space, because they are not libraries and do not rely on decoding logic. The kernel will patch syscalls in kernel either to far calls, or if in the same driver, as push cs and a near call.

The kernel also has a similar system called "os calls" which drivers or kernel export for the benefit of other drivers. It also works with registration, and patching the code to a far call or near call. This way I don't need link a huge kernel image, rather I can load selected drivers and build an image using the pre-linked driver files.

I wouldn't say this in any way resembles the syscall interface of UNIX.

[kerravon]

I don't support ioctl, and never will. That's perhaps the ugliest legacy of UNIX.

How do you suggest switching stdin from line mode to character mode, so that I can use a full-screen application like a text editor?

[kerravon]

I have yet to see an OS allow you to insert data into the middle of a file.

You are presumably aware of the utility "dd". I would like some of those things to be generic to all files, ie the first parameter of fopen() could take them. One situation is on the mainframe where you might wish to specify the recfm and lrecl. Someone else already modified my PDPCLIB to do that, and called it GCCLIB. On PDOS/386 I encountered a need to limit the size of a special file, so that is all I have implemented there. That was implemented in the OS, not the C library.

So syntax is:

mkdosfs --blocks 478208 --offset 128 -F 32 -n "PDOS" drv81:;size=489751552

It is the ";" (semicolon) that separates the normal filename from the modifiers. I was going to use "," (comma) but I remembered that CVS uses comma in the filename, so had to choose something else.

Currently the size= only works for certain special files, and was largely a proof of concept.

Do you have any suggestions to do what you want?