What parts of a kernel do you design for inter-OS compat?

[austanss]

My OS, called microNET, has reached a point where it requires some design thought.

At this point, I have implemented:

- Graphics
- 64-bitness
- BIOS+EFI support
- Keyboard
- Interrupts
- All the stops on memory management (GDT, paging, heap)
- Reasonably accurate chronology using the PIT
- PC speaker beeps
- A semi-functional text-based UI
- Serial console
- Userspace
- Context switching
- System calls

And once I reached system calls, I had to think more about my design vision.

I want my OS to have a reasonable level of compatibility with Windows.
No driver compatibility or anything, just moderately-complex userspace programs.

For example, I want to be able to write a Hello World program, in C, compiled for Windows, running on both Windows and my OS, hopefully by the end of the year.

I'm setting my goals reasonably, but I know compatibility is hard. It starts with the kernel interface. Other than system calls, what other parts of the OS need to be designed for compatibility?

[nexos]

The problem with NT is the extreme amount of the kernel that is exposed. In order to be NT compatible at the binary level, you would have to jump through hoops.
Note that one thing I learned the hard way is to think about design before even writing one line of code. If you don't, you'll problems spend some time rewriting large gobs of code.

[PeterX]

Other than system calls, what other parts of the OS need to be designed for compatibility?

Look at Windows architecture and then pick whatever you like. The kernel is not the only thing providing WIndows system calls. Or maybe they aren't called system calls, but you need them anyway.

Look here:
https://en.wikipedia.org/wiki/Architect ... ecture.svg
You could implement for example GDI and Win32 subsystem. And maybe some runtime libraries (for C and C++)

Of course you don't have to implement every aspect of Windows compatibility (You would get swamped).

Greetings
Peter.

[zaval]

A noble goal in the sea of UNIX clones. for Windows compatibility, you apparently need to implement the WinAPI subsystem. It's well documented, so go ahead, good luck. The complexity lies in the amount of functionality, one needs to implement. just take a look at a moderately complex program, written for Windows, its import list and you'll have the idea of how much you need to implement for those programs to run on your OS. you may try to borrow something from things like Whine or opensourced Microsoft things.

Other than system calls, what other parts of the OS need to be designed for compatibility?

In NT, so called system calls are called system services. Registry. PE image support. Asynchronous IO all the way. threads, the NT way. thread safe CRT, extensive list of synchronization objects. Security subsystem. and an endless list of frameworks and libraries for everything, from XML, HTTP to cryptography and sound, video and graphics.

[austanss]

*sigh*

A noble goal in the sea of UNIX clones. for Windows compatibility, you apparently need to implement the WinAPI subsystem. It's well documented, so go ahead, good luck. The complexity lies in the amount of functionality, one needs to implement. just take a look at a moderately complex program, written for Windows, its import list and you'll have the idea of how much you need to implement for those programs to run on your OS. you may try to borrow something from things like Whine or opensourced Microsoft things.

Other than system calls, what other parts of the OS need to be designed for compatibility?

In NT, so called system calls are called system services. Registry. PE image support. Asynchronous IO all the way. threads, the NT way. thread safe CRT, extensive list of synchronization objects. Security subsystem. and an endless list of frameworks and libraries for everything, from XML, HTTP to cryptography and sound, video and graphics.

A) My OS is not a Unix clone, I believe Unix is a bad design. My OS isn't modelled after any single OS. It's designed uniquely.
B) PE image support is no more difficult than any other executable format. I planned on implementing executables anyway.
C) I wish to be merely compatible for simple programs. I do not see myself implementing full-Windows compatibility. That being said, a lot of this you mentioned is an internal kernel requirement, which is not what I am aiming for.
D) Some of these things you mentioned are not system-level:
---a) HTTP is a protocol, but it uses TCP/IP like any other protocol. (You could also use UDP, but semantics). Thus, HTTP is a mere network program and should not and will not be implemented in the kernel.
---b) Cryptography is not a system level feature, at all. It is mere binary / mathematical operations. Cryptography has nothing to do with the system level.
---c) Video does require fast graphics, which is a system-level feature, but the player itself is not a system-level feature.

[xeyes]

For example, I want to be able to write a Hello World program, in C, compiled for Windows, running on both Windows and my OS, hopefully by the end of the year.

I'm setting my goals reasonably

You are likely underestimating yourself here

Once you can load and run any program, and you're willing to take the easy path, getting the same hello world binary to be able to print helloworld in both windows and yourOS shouldn't take more than a week.

Break down of the estimation:
1. Read about PE format and write a simple loader: 3 or 4 nights
2. Copy this (https://stackoverflow.com/a/1029093) or write something like it yourself, assemble and link it: 15 min
3. Wrap your native write() and _exit() to do what the program from step 2 would expect, debug issues if any: 1 or 2 nights

[xeyes]

A noble goal in the sea of UNIX clones. for Windows compatibility, you apparently need to implement the WinAPI subsystem. It's well documented, so go ahead, good luck. The complexity lies in the amount of functionality, one needs to implement. just take a look at a moderately complex program, written for Windows, its import list and you'll have the idea of how much you need to implement for those programs to run on your OS. you may try to borrow something from things like Whine or opensourced Microsoft things.

Other than system calls, what other parts of the OS need to be designed for compatibility?

In NT, so called system calls are called system services. Registry. PE image support. Asynchronous IO all the way. threads, the NT way. thread safe CRT, extensive list of synchronization objects. Security subsystem. and an endless list of frameworks and libraries for everything, from XML, HTTP to cryptography and sound, video and graphics.

Do you plan to port Whine or reinvent Whine or a combination of both on your noble path?

[zaval]

Do you plan to port Whine or reinvent Whine or a combination of both on your noble path?

no and no. my noble plan is just creating a minimalistic WinAPI inspired API set, read something similar, but not necessarily the same as what is in kernel32, advapi32 and friends. I am not hunting Windows compatibility, I just want my own lil WinAPI. with blackjack and ladies.

[austanss]

A noble goal in the sea of UNIX clones. for Windows compatibility, you apparently need to implement the WinAPI subsystem. It's well documented, so go ahead, good luck. The complexity lies in the amount of functionality, one needs to implement. just take a look at a moderately complex program, written for Windows, its import list and you'll have the idea of how much you need to implement for those programs to run on your OS. you may try to borrow something from things like Whine or opensourced Microsoft things.

Other than system calls, what other parts of the OS need to be designed for compatibility?

In NT, so called system calls are called system services. Registry. PE image support. Asynchronous IO all the way. threads, the NT way. thread safe CRT, extensive list of synchronization objects. Security subsystem. and an endless list of frameworks and libraries for everything, from XML, HTTP to cryptography and sound, video and graphics.

Do you plan to port Whine or reinvent Whine or a combination of both on your noble path?

No, I plan to implement compatibility at a lower level.

[bzt]

For example, I want to be able to write a Hello World program, in C, compiled for Windows, running on both Windows and my OS, hopefully by the end of the year.

Now that's completely different story. A Hello World program for Windows is not allowed to use system calls. So much, that NT system calls aren't really documented, plus they are about to change any time. What is documented, and what your Hello World program would be using, is the DLL interface (user32.dll, krnl32.dll, shell32.dll etc. so called WIN32 API). So if you aim at binary compatibility, you should create DLL files that programs can dynamically link with, and it doesn't matter what system calls the DLLs are actually using to communicate with your kernel under the hood.

Other than system calls, what other parts of the OS need to be designed for compatibility?

Depends on the compatibility level you want to achieve. For Windows compatbility, you'd need PE COFF, DLLs and not kernel system calls mostly. For POSIX compatibility, you can simply use the libc (same way as Windows uses DLLs), or you could implement a syscall wrapper for the Linux system calls (that's how FreeBSD provides binary compatibility with Linux apps, because BSD libc is quite different to glibc and musl, even though all are POSIX compatible in theory and both BSD and Linux are using ELF).

So you have:

• syscall compatibility level: use the same arguments and same responses, this is the most difficult as this interface might change any time
• system library level: implement the same functions as libc/krnl32.dll,user32.dll etc. reasonable and stable API, not straightforward, but simpler
• source level: binary doesn't matter plus you can also use ifdef guards, this is the easiest to achieve

Just for the records, wine implements the first two levels simultaneously: you can use winecfg to select if a specific DLL should be emulated as a wine library (builtin), or as a native DLL emulating the system calls (native).


Cheers,
bzt

[austanss]

For example, I want to be able to write a Hello World program, in C, compiled for Windows, running on both Windows and my OS, hopefully by the end of the year.

Now that's completely different story. A Hello World program for Windows is not allowed to use system calls. So much, that NT system calls aren't really documented, plus they are about to change any time. What is documented, and what your Hello World program would be using, is the DLL interface (user32.dll, krnl32.dll, shell32.dll etc. so called WIN32 API). So if you aim at binary compatibility, you should create DLL files that programs can dynamically link with, and it doesn't matter what system calls the DLLs are actually using to communicate with your kernel under the hood.

Other than system calls, what other parts of the OS need to be designed for compatibility?

Depends on the compatibility level you want to achieve. For Windows compatbility, you'd need PE COFF, DLLs and not kernel system calls mostly. For POSIX compatibility, you can simply use the libc (same way as Windows uses DLLs), or you could implement a syscall wrapper for the Linux system calls (that's how FreeBSD provides binary compatibility with Linux apps, because BSD libc is quite different to glibc and musl, even though all are POSIX compatible in theory and both BSD and Linux are using ELF).

So you have:

• syscall compatibility level: use the same arguments and same responses, this is the most difficult as this interface might change any time
• system library level: implement the same functions as libc/krnl32.dll,user32.dll etc. reasonable and stable API, not straightforward, but simpler
• source level: binary doesn't matter plus you can also use ifdef guards, this is the easiest to achieve

Just for the records, wine implements the first two levels simultaneously: you can use winecfg to select if a specific DLL should be emulated as a wine library (builtin), or as a native DLL emulating the system calls (native).


Cheers,
bzt

Thanks for the in-depth answer! This is rather insightful; I thought that Windows programs use syscalls, but apparently not. So, I can implement radically different syscalls, but in order to achieve Windows compatibility, I need to reimplement the Windows DLLs (just like Wine). I thought I was going somewhere unique with this syscall idea;

[xeyes]

Do you plan to port Whine or reinvent Whine or a combination of both on your noble path?

no and no. my noble plan is just creating a minimalistic WinAPI inspired API set, read something similar, but not necessarily the same as what is in kernel32, advapi32 and friends. I am not hunting Windows compatibility, I just want my own lil WinAPI. with blackjack and ladies.

That's a nice plan. WinAPI certainly has many "better thought out" aspects like wait for single object and its pals.

Not sure whether it can be called little though

[xeyes]

No, I plan to implement compatibility at a lower level.

I think you quoted the wrong box containing a question for zaval.

But regardless you have my admiration to take on the less traveled path and feel free to give the quick and dirty binary compatible helloworld idea a try when you are ready