practicality of a real-mode microkernel

[saltlamp]

Hi,

I'm doing some research, and I feel the need to ask some questions on here. But, I would like to make one thing extremely clear, before I begin the post; this is to avoid any possible confusion on the topic:

The goal for my operating system is to be a 16-bit DOS (PC-DOS, MS-DOS, etc) clone, but instead using a microkernel design; nothing more, nothing less. I only say this so that you all know my goals and can better answer my questions.

Now, I am curious: I noticed that most, if not all, of the DOS variants seem to use the monolithic design for their kernels: is it because of the fact that virtually everything is run in real-mode?

To be more specific, is there a reason that they needed to be monolithic? Excluding reasons for performance on systems that were exponentially slower than today's machines, was it not realistic to have some sort of 'executive' subsystem like NT, where there are separate managers for the microkernel, such as a memory manager, I/O manager, etc?

Here is sort of a really big question that I seriously need to ask: Is it because real-mode is non-asynchronous in nature? Because most DOS variants seem to only do one thing at a time. I am speculating that since you don't really need 'base' drivers for the hardware (unless you want to support more than what the PC-BIOS offers, of course), then you are stuck with the IVT, and then system-calls would use these interrupts instead of driver code, making message passing potentially slow. And then the only way to avoid the non-asynchronous IVT, you'd need need write you own 'base' drivers, Correct me if I'm wrong.

I know that you can do (most) protected-mode things in real-mode, I just imagine the non-asynchronous nature of real-mode might make it pretty dang slow, unless done hacks or something...

And I know it's a lot of questions, but thanks a lot for any help!

(PS: One more quick question that I thought of while writing this: can you load your own IDT while in real-mode? I have never found a wiki page taking about this subject, nor have I have any forum posts, or anything elsewhere discussing this. This question actually ties into the above.)

[alexfru]

DOS was an old and simple design. At the time there wasn't much RAM to waste for the various software layers (managers, wrappers, whatever). Further, in real mode they were all easily penetrable due to lack of protection features in the CPU (any program could access any memory location or device or mess up interrupts (yes, you obviously can change the contents of the IDT/IVT in real mode)). Developing and debugging complex and fragile things was not fun either. So, a lot of things in DOS were simple and minimal.

[Schol-R-LEA]

On the one hand, it is certainly possible to write a Microkernel for real mode x86 on the PC, in the sense that it could be done: back in the day there were several, including Minix 1.x/2.x, Coherent OS, PCix, and early versions of QnX. There may have been a PC port of TriPOS (the micro-kernel developed at Cambridge which went on to become Amiga Exec) but I am not sure. I am not sure if MultiDOS (or 4DOS, which I think might have been another name for it but might have been something separate, I am not sure) would be considered a microkernel or not, as I know next to nothing about them, but I doubt it.

You will note two things here. First, MS-DOS/PC-DOS isn't on that list, and I will explain why momentarily. Second, most of them are loosely based on Unix, or at least heavily influenced by it in terms of API, IPC, shell interface, etc., despite being (in terms of design approach) diametrically opposed to the Unix of the time, which was a genuine monolithic kernel which would need to be re-compiled even just to add a new driver. The latter was mostly for the sake of what some would call simplicity, and others would call laziness: by the mid-1980s, most new programmers would at least have used Unix at college, so it stuck with what people knew¹.

The former is more relevant here, as MS-DOS was neither a microkernel nor a monolithic kernel - in fact, it wasn't a kernel-based design at all, being what is technically known as an 'execution monitor'. An exec monitor is a type of monotasking system which basically has two jobs: launch programs, and provide an ABI that programs can call for basic system services. Exec monitors were common for minicomputers² in the early 1960s, when systems design was in its infancy and the systems were both too slow and too limited in memory to really do two things at a time, with some being basically just loaders for punched cards and/or paper tape. Some of the earliest interactive systems, such as the TX-0, Linc, PDP-1, and PDP-8, had somewhat more elaborate exec monitors, but they were still the same sort of thing. They often contained a machine code monitor as part of or even the whole of the user interface, both for coding and debugging. This was before the abstraction of a 'kernel' had even been invented yet (which was a few years later in 1965 or so, with the 'microkernel' design arising in 1967), and none of them would really be considered kernel systems today.

They vanished for a time, but saw a revival when the microcomputer boom hit, as the early micros were comparable - or a little below - the performance of the first-gen minis, so systems such as CP/M, TRS-DOS, and QDOS³ adopted the approach, while others such as Apple and Commodore went with a hybrid approach in which they used a BASIC interpreter in the role of the exec monitor (with some systems, including the Commodore PET and its successors, putting the BASIC in ROM so that they wouldn't require the use of a then-expensive floppy disk drive; Apple had a simpler BASIC, called Integer BASIC, in ROM, but the familiar Applesoft BASIC was part of the disk-based AppleDOS).

OK, so the point is, they were all Monotasking Systems which didn't have kernels at all. So what?

Well, the 'so what' here is that microkernels by definition are all about securely multi-tasking without hardware memory protection, because up until the late 1980s, hardware memory management was considered an expensive luxury for a small computer to have, and few microprocessor CPUs of the time supported it.⁴ The main goal of the micro-kernel design was to allow a multitasking kernel to remain stable even if a driver or other system facility went off the rails, so long as the kernel scheduler itself remained healthy. The 'micro' name is misleading, because it isn't really about size at all, but about the separation of core system functions from anything not absolutely necessary for the kernel to function.

But all of this is moot, at least with regards to newer systems, because not only is the Legacy BIOS going away at the end of this year, real mode itself (and 16-bit protected mode, and possibly even 32-bit protected mode eventually) is probably going away in the next five years. After 2020, you won't readily be able to go into real mode, even if the CPU still has it, as the UEFI BIOS⁵ won't support booting into it - you could drop down into real mode somehow, I suppose, but it would require you to have all the facilities of a protected mode or long mode OS in place in order to even do that.


Footnotes

1. Though older programmers would have probably preferred something more like Tenex, RTX, or maybe IBM's CMS - not to be confused with VAX VMS, which was still relatively new itself, but I digress.
2. 'Mini' in the sense that they were the size of a large refrigerator, rather than a small building as with most mainframes of the day.
3. Also called, 86-DOS but better known as SCP-DOS, which became PC-DOS/MS-DOS after Microsoft bought the failing Seattle Computer Products.
4. Those which did usually did so with a co-processor, as with the Motorola 68000 and the early MIPS and ARM microprocessors.
5. Don't give me grief over this - 'BIOS' ("Basic Input/Output System", that is to say, the set of firmware needed to boot the system and support a running OS) is a generic term, predating the PC by decades. UEFI is a BIOS, even if it isn't the Legacy BIOS.

[bzt]

Hi,

Your question is a bit vague, as a microkernel needs separated tasks communicating with messages, and dedicated service tasks. This implies that a microkernel architecture must be multitasking. The systems you've mentioned are monotasking, meaning only one task can run at any given time. If that task needs a service, it can't send a message to the server task as there's only one task running.

However monotasking does not mean there's no kernel. PC-DOS and MS-DOS also had a kernel, for the latter it was called MSDOS.SYS and IO.SYS. Separating the common kernel code from the drivers was a step forward the microkernel architecture, but since they were loaded to the same address space for the single task there was no need for messaging or task switching.

An interesting historical fact, that MS-DOS 4 was capable of multitasking, but to press Windoze M$ decided to criple DOS and removed multitasking from MS-DOS 5. RBIL is full of MS-DOS 4 only functions, check it out. If you can get your hands on the source of MS-DOS 4, you'll get exactly what you need, a real mode multitasking system, in which the common kernel code and device drivers are separated. There's only one step to create dedicated server tasks and you'll have a real mode microkernel.

To make it easier to understand, let's introduce a few phrases:
- thread: an execution environment for the CPU, usually includes register save area especially the instruction pointer
- process: separated address spaces, where one process cannot access other process' memory. A process must have one thread at least, but may have more (with the help of pthreads for example)
- task: a code that is executing in order to do a specific task. This is independent of processes, you can implement more tasks in one process, but all tasks must have their own threads for sure

MS-DOS 1,2,3,5: one thread, one process, one task at a time (not a microkernel, as you can't do server tasks)
MS-DOS 4: more threads, one process, more tasks (closer, but still not a microkernel as it has no dedicated server tasks and messaging)
AmigaOS: more threads, one process, more tasks (I think many would argue that AmigaOS is a real microkernel, let's just say it's pretty close)
Minix: more threads, more processes, more tasks (microkernel with all the requirements: dedicated server tasks, messaging etc.)

Hope this helps understanding,
bzt

ps: I haven't used, but maybe you should check out FreeDOS. It's Open Source, and if memory serves multitasking too, so very similar to MS-DOS 4.

[Schol-R-LEA]

BZT: Kernels, as an architectural principle, were very much the exception rather than the rule in 1980, even for minis and mainframes - Unix was one of the few at the time that did, IBM never used one on any of their OSes until AIX (a licensed SysV Unix port), and most of DEC's operating systems didn't prior to VMS.

MS-DOS never used that abstraction. It have a main executable, but a core executable is not the same as a kernel.

But perhaps I am playing too strictly with the semantics, so feel free to ignore me on this.

[bzt]

Hi,

BZT: Kernels, as an architectural principle, were very much the exception rather than the rule in 1980, even for minis and mainframes - Unix was one of the few at the time that did, IBM never used one on any of their OSes until AIX (a licensed SysV Unix port), and most of DEC's operating systems didn't prior to VMS.

MS-DOS never used that abstraction. It have a main executable, but a core executable is not the same as a kernel.

But perhaps I am playing too strictly with the semantics, so feel free to ignore me on this.

I don't want to ignore you, actually I like your posts very much.

The thing is, OSes haven't changed much since the '60s, still using the same abstractions as TSS and Multics did. But you're right that they were not called "kernel" back then. The phrase "core" was more common in the '80s. However having a main executable which is loaded by the bootloader and provides system calls (like mkdir, creat, open, read, write etc. device files like PRN, CON etc.) is something that we would call a kernel these days. So I think we can safely call MSDOS.SYS a kernel, even though that phrase was not in common use at the time MSDOS was introduced (IMHO).

(ps: by OS abstraction I mean there's a supervisor (core, kernel, exec whatever) responsible for managing user tasks that do the actual work for the users, providing system services for those user tasks using file and other high-level abstractions. This scheme hasn't changed a bit in the last 60 or so years.)

Cheers,
bzt

[Octocontrabass]

An interesting historical fact, that MS-DOS 4 was capable of multitasking, but to press Windoze M$ decided to criple DOS and removed multitasking from MS-DOS 5.

No one removed multitasking from anything. There are two operating systems called "MS-DOS 4" and they are very different from one another.

The first MS-DOS 4 was a multitasking MS-DOS, developed in parallel with and as the successor to MS-DOS 3. IBM refused to license it, since its multitasking didn't use protected mode, and most OEMs didn't want something that IBM didn't want. Development on MS-DOS 4 and the companion Windows version was canceled, and the MS-DOS 4 released to OEMs was missing planned features. As a replacement, Microsoft and IBM began a joint effort to develop MS-DOS 5. Shortly before announcing MS-DOS 5 to the public, IBM changed the name to OS/2.

The second MS-DOS 4 was an updated MS-DOS 3. This MS-DOS 4 was then updated and released as MS-DOS 5.

[linguofreak]

BZT: Kernels, as an architectural principle, were very much the exception rather than the rule in 1980, even for minis and mainframes - Unix was one of the few at the time that did, IBM never used one on any of their OSes until AIX (a licensed SysV Unix port), and most of DEC's operating systems didn't prior to VMS.

MS-DOS never used that abstraction. It have a main executable, but a core executable is not the same as a kernel.

But perhaps I am playing too strictly with the semantics, so feel free to ignore me on this.

I would argue that kernels don't strictly exist on any hardware that doesn't distinguish privilege levels, regardless of the architecture of the software side of the system, and that any executive/supervisor/monitor/"kernel" that does not make use of privilege levels that exist on a given piece of hardware (e.g, DOS proper, running only in real mode, on a machine with a protected-mode capable CPU) is not a kernel either. You can distinguish your executive from your application code in the same way that we'd distinguish libc from a compiled application on a modern privileged-hardware Unix system, in terms of the intended purpose of the code, but you can't make the distinction that is made between kernel and application on a modern system, in terms of the protection boundaries between the two software components.

You can consider everything on a privilege-agnostic system to be part of the kernel, or you can consider it to be all userspace with the executive being "libc", but you can't meaningfully talk about this component being the kernel and that component being application code, because once execution reaches any given module of code, it can, if it wants, scribble all over the IVT (or equivalent on the hardware in question), so that all interrupts are vectored to it and take over responsibility as the system executive.

In terms of the way that executive software on privilege agnostic-software tends to evolve when privilege separation is introduced on a given hardware architecture, the "everything is userspace" way of talking about privilege-agnostic systems actually makes a lot of sense: Generally, a new component is introduced (generally a VCPI or DPMI server on DOS) that administers the new privilege system provided by the hardware and virtualizes (to a greater or lesser extent) the way that the hardware environment looked pre-privilege for multiple different processes, and each process ends up running a copy of the pre-privilege executive as a runtime library for application code running in that process.

[linguofreak]

MS-DOS 1,2,3,5: one thread, one process, one task at a time (not a microkernel, as you can't do server tasks)
MS-DOS 4: more threads, one process, more tasks (closer, but still not a microkernel as it has no dedicated server tasks and messaging)
AmigaOS: more threads, one process, more tasks (I think many would argue that AmigaOS is a real microkernel, let's just say it's pretty close)
Minix: more threads, more processes, more tasks (microkernel with all the requirements: dedicated server tasks, messaging etc.)

Win16 on 386+ systems was very interesting here:

Multiple processes for DOS applications, but the userland components of Windows and all of its applications ran in a single, multithreaded process.

[bzt]

No one removed multitasking from anything.

You said it yourself, MS-DOS 4 was a multitasking OS, while MS-DOS 5 wasn't. Yes, there was two development in parallel, but I think that's circumventional, and does not change the fact that MS-DOS 5 lacked multitasking which at least one prior version had. We surely won't fight over this

I would argue that kernels don't strictly exist on any hardware that doesn't distinguish privilege levels

I give you that the definition is vague. There's no real standard for it, moreover changed a bit over time, so one can interpret it differently.

If we accept lingofreak's opinion, then strictly speaking there can't be any real-mode microkernel. However I did a little digging, and find that Minix1 (which is a microkernel for sure) can be compiled for 8088 real-mode only without privilege level protection (check out this source file). So at the end of the day it's just a matter of how you interpret the definition.

Cheers,
bzt

[Octocontrabass]

We surely won't fight over this

Well, since you insist...

The point of my post is that Microsoft didn't remove multitasking because MS-DOS never had multitasking in the first place. The "MS-DOS 4" with multitasking is a very different OS that happens to share a name and ABI compatibility with MS-DOS. It's the same way Windows 2000 is a very different OS that happens to share a name and ABI compatibility with Windows ME.

[linguofreak]

I would argue that kernels don't strictly exist on any hardware that doesn't distinguish privilege levels

I give you that the definition is vague. There's no real standard for it, moreover changed a bit over time, so one can interpret it differently.

If we accept lingofreak's opinion, then strictly speaking there can't be any real-mode microkernel. However I did a little digging, and find that Minix1 (which is a microkernel for sure) can be compiled for 8088 real-mode only without privilege level protection (check out this source file). So at the end of the day it's just a matter of how you interpret the definition.

Note that I said "don't strictly exist". For kernels that are designed to operate as privileged software on privilege-enforcing hardware, but that can be compiled for and run on privilege-agnostic hardware, there's reason to be less strict. But even in that case, when running privilege-agnostic, the kernel can't do anything to enforce its status as *the* kernel: Any application can steal the system out from under it.

Interestingly, Minix 1 and 2 had the option to steal the system out from under DOS. You could launch them as DOS applications, and they'd steal the IVT, launch into protected mode (or not), and pass control back to DOS on HDD interrupts (they used an image file on the DOS filesystem as their root filesystem in this mode).

[bzt]

MS-DOS never had multitasking in the first place.

The "MS-DOS 4" with multitasking is a very different OS

You contradicted yourself right away.
Both are products of MS, both are branded MS-DOS, both are capable of running the same applications. You too call it "MS-DOS 4", simply because it had no other name. The "MS-DOS 4" with multitasking is part of the MS-DOS family, there's absolutely no point in denying that.

It's the same way Windows 2000 is a very different OS that happens to share a name and ABI compatibility with Windows ME.

Both are products of MS, both are branded Windows, both capable of running the same applications. Believe me, the average end-user and the marketing department did not know nor care that one is built on top of the NT kernel and other is not.

Also this is a bad example, because Windows XP inherited the more advanced NT kernel, while MS-DOS 5 inherited the less advanced monotasking core (although we know it could have been built on top of the multitasking one, as MS had the code and all the rights).

For kernels that are designed to operate as privileged software on privilege-enforcing hardware, but that can be compiled for and run on privilege-agnostic hardware, there's reason to be less strict.

That's my point. Minix1 was never designed for a privilege-enforcing hardware. It was real-mode only. (Protected mode was introduced in Minix2.)

However I think you made a perfect argument for the OP. The practicality of a real-mode microkernel is zero, because of the lack of protection and privilege separation. This answers their question pretty well!

Cheers,
bzt

[linguofreak]

MS-DOS never had multitasking in the first place.

The "MS-DOS 4" with multitasking is a very different OS

You contradicted yourself right away.
Both are products of MS, both are branded MS-DOS, both are capable of running the same applications. You too call it "MS-DOS 4", simply because it had no other name. The "MS-DOS 4" with multitasking is part of the MS-DOS family, there's absolutely no point in denying that.

The sequence of events was:

Microsoft announced multitasking MS-DOS 4.0.

Multitasking MS-DOS 4.0 received lukewarm OEM reception.

Microsoft scaled back the project, and killed it when the last of the OEM contracts was fulfilled.

IBM announced PC-DOS 4.0, which they had developed internally from DOS 3 while Microsoft was working on multitasking MS-DOS 4.0.

PC-DOS 4.0 was buggy, so the project went back to Microsoft, which developed a bug-fixed version and released it as MS-DOS 4.01

So, yes, multitasking DOS 4.0 was part of the MS-DOS family, but it became a dead-end branch because OEMs didn't want it, not because Microsoft decided to kill it to push Windows. It was only licensed by a few European OEMs and saw hardly any use outside of Europe. American users hardly know about it.

For kernels that are designed to operate as privileged software on privilege-enforcing hardware, but that can be compiled for and run on privilege-agnostic hardware, there's reason to be less strict.

That's my point. Minix1 was never designed for a privilege-enforcing hardware. It was real-mode only. (Protected mode was introduced in Minix2.)

Minix1 didn't make use of Intel protected mode, but it was available from version 1.5 for M68k and SPARC.

"The core microkernel of MINIX 1.0 was under 1400 lines of C and assembler. To that you have to add the headers and device drivers, but the totality of everything that ran in kernel mode was under 5000 lines."

I'm not aware of 1.0 having been available for privilege-enforcing hardware, so it's likely that he's talking about Minix 1 in general, but this quote makes no sense if no pre-2.0 version of Minix ever ran on and made use of privilege-enforcing hardware.

However I think you made a perfect argument for the OP. The practicality of a real-mode microkernel is zero, because of the lack of protection and privilege separation. This answers their question pretty well!

Not only is the practicality of a real-mode microkernel zero, but whether you can even distinguish between a microkernel and a macrokernel is in question.

[eekee]

The goal for my operating system is to be a 16-bit DOS (PC-DOS, MS-DOS, etc) clone, but instead using a microkernel design; nothing more, nothing less. I only say this so that you all know my goals and can better answer my questions.

Hmm... My question is, why didn't the FreeDOS project do this, since they were in a prime position to benefit from the various components being developed by different people. Instead, they went in the opposite direction, the single file, kernel.sys replacing the two, msdos.sys and io.sys. Perhaps there's no reason, it may have been a mistake. Or perhaps the single file helps eliminate the possibility of the various parts getting out of sync with each other, but there are other ways to manage that.

Regardless, I think it'll be an interesting project, but it may not be little or easy. You speak of DOS compatibility, but even in the DOS world, compatibility is remarkably hard. It took 12 years for FreeDOS to reach 1.0! Granted, you could use components developed for their userspace, which may shorten the time somewhat, and I think you'll have access to better development tools than they had when they started, but I still think it'll be a long job. Fine for a hobby perhaps, but not so much if you're hoping for results. Of course, if you're some sort of DOS wizard, with a century of DOS experience under your belt, you may be able to do it in a few weeks. (Strange to think MS-DOS isn't half a century old yet. FreeDOS is more than half its age.)

But all of this is moot, at least with regards to newer systems, because not only is the Legacy BIOS going away at the end of this year, real mode itself (and 16-bit protected mode, and possibly even 32-bit protected mode eventually) is probably going away in the next five years. After 2020, you won't readily be able to go into real mode,

Just to note, the crucial words here are "newer systems". BIOS/DOS SoCs and SBCs will likely be available for many years. Emulation too, which is fine for hobby use and even some professional use... except, again, for incompatibilities. I found several fancy text editors which work well under FreeDOS on real hardware, but not Qemu. Perhaps OP might overcome that problem...