32-bit MSDOS

[kerravon]

Hi. About 30 years ago, someone made a comment on a group saying "until DOS is made 32-bit, DOS extenders are just a kludge".

I didn't know much about DOS-specifics back then to even understand the comment.

Microsoft abandoned the MSDOS API, but what would a 32-bit version of the MSDOS API look like if they hadn't abandoned it?

Since 1994, off and on, I have been working on PDOS/386, which is designed to be a 32-bit version of MSDOS. No more, no less.

I have recently just completed relocation of OS and app so that they can both run in a flat address space, meaning that applications can directly manipulate the hardware, e.g. by going *(char *)0xb8000 = 'X'; . With that change now in place, it is working to my satisfaction technically (I want to return to the days when blindingly fast applications could be written by manipulating the hardware), and I would like to solidify the API. What would a 32-bit MSDOS API ideally look like? My current thinking is that both int86() and some wrapper functions should be made available, which I have already done here:

https://sourceforge.net/p/pdos/gitcode/ ... /src/pos.h

Note that this is all working, but I am interested in tweaking the design/interface before lots of 16-bit MSDOS programs get ported to 32-bit, so I am open to suggestions. E.g. I was thinking that the PosAllocMem() should instead be put on a different AH code and take extra parameters, mainly a "location" which can be set to 20 to request memory in the first 1 MiB, or 32 for the full 4 GiB space, and in the future support 64.

PDOS/386 is explicit public domain (no license) and written mostly in C, with a bit of assembler. The way it works is that the OS and apps are pure 32-bit protected mode, but when a service like reading from disk is required, it converts the protected mode BIOS interrupt into a real mode interrupt. This conversion to real mode is transparent to the OS and apps.

I don't need any code written. I just need peer review of the design/API. Hopefully someone can offer some opinions. Thanks!

[Brendan]

Hi,

I just need peer review of the design/API. Hopefully someone can offer some opinions.

I'm not sure you're going to want my honest opinion, but...

Microsoft abandoned the MSDOS API, but what would a 32-bit version of the MSDOS API look like if they hadn't abandoned it?

It would still look extremely bad and inefficient.

All modern hardware is designed to work in parallel; so you can have (e.g.) a network card going flat out transferring packets while a hard disk controller is going flat out reading data from disk while a USB controller is going flat out writing to USB flash while the GPU renders a bunch of graphics while 8 CPUs are all going flat out processing data. MS-DOS is designed for "only one thing can happen at a time" (e.g. get a packet from network while all the rest of the hardware does nothing, then read some data from disk while all of the rest of the hardware does nothing, then ....). If you assume an average computer costs $1000 then this is like throwing $900 worth of performance in the trash.

What would a 32-bit MSDOS API ideally look like?

To make MS_DOS API better than crippleware, the first step would be to make the API asynchronous. For example, rather than writing to a file (and having all other hardware wasted until the write completes) you'd ask the OS to start writing a file and then continue doing other work, and then later either the OS would inform you when the write completes (e.g. maybe a callback?) or you'd have a polling loop (e.g. "while(running) { do a little more work; ask OS if write finished yet; }". The second step would be to add threads (including making sure all of the API functions handle threads correctly) so that an application/process can use more than one CPU.

Of course any attempt to make the DOS API tolerable would transform it into something that is not like DOS at all; and in that case you might be tempted to think about other major problems with MS-DOS (e.g. the complete lack of security, the broken device driver model that ruins any hope of applications being compatible with a wide range of hardware, etc).


Cheers,

Brendan

[kerravon]

To make MS_DOS API better than crippleware, the first step would be to make the API asynchronous. For example, rather than writing to a file (and having all other hardware wasted until the write completes) you'd ask the OS to start writing a file and then continue doing other work, and then later either the OS would inform you when the write completes (e.g. maybe a callback?)

The Posix API allows me to just write() to a file. Why would the existing MSDOS API not be a reasonable substitute for Posix?

in that case you might be tempted to think about other major problems with MS-DOS (e.g. the complete lack of security, the broken device driver model that ruins any hope of applications being compatible with a wide range of hardware, etc).

I am happy to live with restrictions for now. I'm wanting to get PDOS/386 sufficiently implemented that it can rebuild itself so that I can use a genuine public domain operating system to build something better in the future.

[Brendan]

Hi,

To make MS_DOS API better than crippleware, the first step would be to make the API asynchronous. For example, rather than writing to a file (and having all other hardware wasted until the write completes) you'd ask the OS to start writing a file and then continue doing other work, and then later either the OS would inform you when the write completes (e.g. maybe a callback?)

The Posix API allows me to just write() to a file. Why would the existing MSDOS API not be a reasonable substitute for Posix?

POSIX supports asynchronous IO, and also supports threads (which allows "poor man's asynchronous IO", where you create thread/s to do IO in parallel with your main thread).

Also don't forget that most operating systems are multi-tasking, so if a (single-threaded) process waits for a "write()" to complete then all of the other hardware (devices and CPUs) can still be kept doing useful work for other processes.

MS-DOS was "special" because it didn't provide any way to keep hardware (devices and CPUs) doing useful work in any situation.


Cheers,

Brendan

[kerravon]

Also don't forget that most operating systems are multi-tasking, so if a (single-threaded) process waits for a "write()" to complete then all of the other hardware (devices and CPUs) can still be kept doing useful work for other processes.

MS-DOS was "special" because it didn't provide any way to keep hardware (devices and CPUs) doing useful work in any situation.

The lack of multi-tasking in the actual implementation of 16-bit MS-DOS does not mean that there is a problem with the API itself that forever prevents a new version of MS-DOS (ie PDOS/386) from supporting multi-tasking, does it?

Regardless, I'm not trying to replace the MS-DOS API with Posix or something else, I'm taking the MS-DOS API *as a given*. What I'm looking for is a minimal set of changes, that still revolve around the MS-DOS API, but allow for 32-bit and maybe one day 64-bit programming. The only thing I would change about the MS-DOS write() API is to allow the 32-bit version to take the number of bytes to write in ECX rather than CX. And to avoid application programmers having to code two different int86x calls, one 16-bit populating cx, and one 32-bit populating ecx, I would instead have a wrapper function called PosWriteFile(int handle, void *data, size_t len);. Or something like that. I'm fairly happy with that one already. It's the allocation of memory that has me going round and round in circles trying to decide whether I need a new AH service and have a flag (register) to say whether I want memory in 20-bit space or 32-bit space, similar to how MVS (mainframe) does things. I'm expecting this one API call will be a necessary divergence from the rest of the API.

[Brendan]

Hi,

Also don't forget that most operating systems are multi-tasking, so if a (single-threaded) process waits for a "write()" to complete then all of the other hardware (devices and CPUs) can still be kept doing useful work for other processes.

MS-DOS was "special" because it didn't provide any way to keep hardware (devices and CPUs) doing useful work in any situation.

The lack of multi-tasking in the actual implementation of 16-bit MS-DOS does not mean that there is a problem with the API itself that forever prevents a new version of MS-DOS (ie PDOS/386) from supporting multi-tasking, does it?

The problem I'm talking about ("90% of hardware wasted most of the time due to no way to have any parallelism") has at least 3 solutions (asynchronous IO, threads, multi-tasking) where any one solution could make a huge difference; and where you could have 2 of them (asychronous IO and threads) while still not having any multi-tasking.

For multi-tasking, that's not a restriction of the API itself. There were compatible implementations of DOS that did support multi-tasking (e.g. DR-DOS) and various operating systems (OS/2, Windows 95, ...) that supported multi-tasking while still providing the same API for legacy DOS software.

The main problem (for multi-tasking) is not the API, it's the lack of API. More specifically, because the API was incomplete (memory, time, file IO and almost nothing else) applications had to use raw device access for a lot of things (video, sound, ...) which becomes a huge disaster for multi-tasking (because the applications can't coordinate their raw device access and end up trashing each other) and this requires a "device emulation" approach to work around.

Regardless, I'm not trying to replace the MS-DOS API with Posix or something else, I'm taking the MS-DOS API *as a given*. What I'm looking for is a minimal set of changes, that still revolve around the MS-DOS API, but allow for 32-bit and maybe one day 64-bit programming.

I understand this; but you've asked for opinions, and my opinion (which you're free to ignore) is that there are only 2 sane cases - either you try to provide "100% MS-DOS compatible" (real mode, like FreeDOS), or it's not compatible (and therefore you have no valid reason to make the API suck the same as the original MS-DOS API).


Cheers,

Brendan

[linguofreak]

Also don't forget that most operating systems are multi-tasking, so if a (single-threaded) process waits for a "write()" to complete then all of the other hardware (devices and CPUs) can still be kept doing useful work for other processes.

MS-DOS was "special" because it didn't provide any way to keep hardware (devices and CPUs) doing useful work in any situation.

The lack of multi-tasking in the actual implementation of 16-bit MS-DOS does not mean that there is a problem with the API itself that forever prevents a new version of MS-DOS (ie PDOS/386) from supporting multi-tasking, does it?

It's not that you can't support multitasking, it's that to do multitasking while retaining compatibility with existing DOS programs, you have to either give up stability or emulate device I/O, in which case you've given up on direct hardware access. For new programs, having (for example) a sound API that abstracts away the details of the sound hardware is easier for you, easier for application developers, and more performant than trying to emulate every single little detail of every sound card that a DOS program might ever have tried direct hardware access to.

Microsoft has already done what you're attempting. The "Allow direct hardware access, at the cost of stability" branch was called Windows 3.x and Windows 9x, and is responsible for the reputation of Windows as a crashy system. The "emulate hardware access" branch is called NTVDM. On 32-bit Windows systems to this day, it encapsulates a DOS instance as a Windows process, but even Microsoft doesn't have the resources to write emulation for every single hardware device that any DOS program ever might try to access directly, so there are a lot of DOS programs that won't run on it, and fewer and fewer with every release of Windows.

Regardless, I'm not trying to replace the MS-DOS API with Posix or something else, I'm taking the MS-DOS API *as a given*. What I'm looking for is a minimal set of changes, that still revolve around the MS-DOS API, but allow for 32-bit and maybe one day 64-bit programming. The only thing I would change about the MS-DOS write() API is to allow the 32-bit version to take the number of bytes to write in ECX rather than CX. And to avoid application programmers having to code two different int86x calls, one 16-bit populating cx, and one 32-bit populating ecx, I would instead have a wrapper function called PosWriteFile(int handle, void *data, size_t len);. Or something like that. I'm fairly happy with that one already. It's the allocation of memory that has me going round and round in circles trying to decide whether I need a new AH service and have a flag (register) to say whether I want memory in 20-bit space or 32-bit space, similar to how MVS (mainframe) does things. I'm expecting this one API call will be a necessary divergence from the rest of the API.

As I said, "for new programs, having (for example) a sound API that abstracts away the details of the sound hardware is easier for you, easier for application developers, and more performant than trying to emulate every single little detail of every sound card that a DOS program might ever have tried direct hardware access to".

For existing programs, you don't need any more than the existing DOS APIs, DOS itself, DPMI, Win16, etc.

For new programs, you don't want an API that allows direct hardware access, and you want an API that is defined in terms of a high-level-language, not x86 assembly, so you really don't want to try to extend the DOS API to 64-bits (DPMI already did that for 32 bits).

[kerravon]

I understand this; but you've asked for opinions, and my opinion (which you're free to ignore) is that there are only 2 sane cases - either you try to provide "100% MS-DOS compatible" (real mode, like FreeDOS), or it's not compatible (and therefore you have no valid reason to make the API suck the same as the original MS-DOS API).

Thanks for your opinion, but I don't see why it needs to be 100% MS-DOS compatible and run in real mode. My experience with porting Micro-Emacs, which writes directly to 0xb8000, is that very little needs to change. The recompile with a 32-bit compiler is enough to make Micro-Emacs no longer have a 640k memory limitation. I did have to make some changes, like they were writing to 0xb8000 using an "int *" instead of "short *". The existing int86 calls to both MS-DOS and BIOS worked unchanged. Isn't lifting the restriction on Micro-Emacs from 640k to 3.5 GB (or whatever) a worthwhile advance for a MS-DOS application?

Note that many people pointlessly do things like climb Mount Everest. I would like to do something similar with MS-DOS. It's there! Or rather, it isn't there! A 32-bit version of the MS-DOS API, for better or for worse, has never been implemented so we can never see it, warts and all.

[kerravon]

For existing programs, you don't need any more than the existing DOS APIs, DOS itself, DPMI, Win16, etc.

I don't see why it should be valid to create 32-bit DOS extenders using DPMI, but not valid to create a 32-bit DOS.

[Tom]

I don't see why it should be valid to create 32-bit DOS extenders using DPMI, but not valid to create a 32-bit DOS.

It *was* valid to create DOS extenders. In 1990.

DOS + (HIMEM or EMM386) was already some kind of 32 Bit DOS, even in it was somewhat unfriendly to Programmers.

DOS extenders added a nicer Interface(for programmers) and the ability to run with or without HIMEM, EMM386,EMM386.

Cool work in 1990. Not so much in 2018.

BTW: DRDOS EMM386 had a multitasking API in ~1993. Look it up for suggestions.

Tom

[SpyderTL]

I wouldn't worry too much about the negative feedback about your approach. That is normal for this site.

If I can rephrase your question as I understand it... If you wanted to create a new 32-bit version of the 16-bit DOS API, what changes would you make? (Other than the obvious CX to ECX type stuff.)

I assume the end goal is to make it as simple as possible to port a 16-bit application to use this new 32-bit API.

For instance, if you have the C/ASM source for a DOS mod file player, and you wanted to recompile it to use this new 32-bit API, what changes would you make to the API (or not make) to make that port as simple as possible.

Obviously, the fewer changes you make, the easier the port will be, but I can't think of any specific changes that I would make off of the top of my head. But I'll think about it a bit and respond later if anything pops up.

[linguofreak]

For existing programs, you don't need any more than the existing DOS APIs, DOS itself, DPMI, Win16, etc.

I don't see why it should be valid to create 32-bit DOS extenders using DPMI, but not valid to create a 32-bit DOS.

Purely as a retrocomputing exercise, you can do whatever you want. But 32-bit DPMI basically *is* a 32-bit DOS. DPMI had its start within Microsoft.

Basically what I'm saying is:

1) You seem to think it hasn't already been done, but it has.

2) Many of your stated goals are incompatible with each other. You want to write the OS that Microsoft would have written if they hadn't abandoned DOS, but you don't want to give up direct hardware access, even though they certainly would have done so, and were already starting to do so (and, with NTVDM, *did* do so. If you count NTVDM, MS-DOS is dying, but it isn't entirely dead).

[kerravon]

I wouldn't worry too much about the negative feedback about your approach. That is normal for this site.

If I can rephrase your question as I understand it... If you wanted to create a new 32-bit version of the 16-bit DOS API, what changes would you make? (Other than the obvious CX to ECX type stuff.)

Yes, but I'm not talking about what cool changes could be done with a brand new 32-bit operating system. I want to focus on the existing 16-bit API and in a sense I am asking what changes to make to the 16-bit API to future-proof it for an eventual 32-bit API. Minimizing the changes required to the 16-bit programs, but ensure that they are following a bit-neutral API.

I assume the end goal is to make it as simple as possible to port a 16-bit application to use this new 32-bit API.

Yes, and just applications. I'm not talking about device drivers. They can be totally redesigned as required.

For instance, if you have the C/ASM source for a DOS mod file player, and you wanted to recompile it to use this new 32-bit API, what changes would you make to the API (or not make) to make that port as simple as possible.

Yes, I'm only interested in applications that have source code so can be recompiled.

Obviously, the fewer changes you make, the easier the port will be, but I can't think of any specific changes that I would make off of the top of my head. But I'll think about it a bit and respond later if anything pops up.

Ok, thanks. I think we should consider having functions to wrap the API, so that programmers can code DosOpenFile() or similar, rather than coding the actual interrupt number. But my experience with porting micro-emacs is that so long as the app is using int86(), it will work with the 32-bit version of int86(). There wasn't very much I needed to change in micro-emacs. I had to change it to use "short *" instead of "int *" when writing to 0xb8000 though. But that should have been thought of in advance as part of 32-bit future-proofing.

[kerravon]

But 32-bit DPMI basically *is* a 32-bit DOS. DPMI had its start within Microsoft.

From the 32-bit application's point of view, it looks like a 32-bit DOS, but it isn't. It's a kludge, still running a 16-bit DOS. I want DOS itself to be 32-bit.

2) Many of your stated goals are incompatible with each other. You want to write the OS that Microsoft would have written if they hadn't abandoned DOS, but you don't want to give up direct hardware access, even though they certainly would have done so

I can clarify my goal as required. I am purely after recompiling DOS using a 32-bit compiler and hey presto, no need for DPMI, applications communicate with a 32-bit operating system instead of a 16-bit one. No other technical change to DOS. Of course, Microsoft couldn't recompile DOS with a 32-bit compiler because it was written in assembler instead of C. But I have written PDOS in C, and indeed, PDOS/86 runs 16-bit DOS programs. But I have recompiled it with a 32-bit compiler and hey presto, I now have a 32-bit version of DOS, ie PDOS/386. It's a little more involved than just a simple recompile. Some conditional compilation was necessary. But that's the principle. If Microsoft had written in C instead of assembler, they could have just recompiled. And MSDOS applications could similarly have been written in C and just need a recompilation. How close can we come to that ideal? ie vendors distributing both 16-bit and 32-bit versions of their MSDOS applications, with a near-identical code base.

[Schol-R-LEA]

Has it been four months already? Well, time for this discussion again.

Kerravon, are you familiar with the (now-defunct, AFAICT) 64-bit branch of FreeDOS? It was based on the FreeDOS code base, and had similar goals to your project. I am not aware of why it was discontinued, but I am pretty sure that the code repos are still publicly available.

OTOH, if you are concerned primarily with direct hardware access, rather than specifically running legacy MS-DOS software, I would recommend looking into the Exokernel model of OS design.

/me waits for Tilde to chime in, as this conversation is up his slightly demented alley