DOS memory managers?

[mattrix]

I'm confused.
I'm not sure if this is a question about memory or CPU modes or just how Dos works.

As far as I can tell Himem puts dos into "unreal" mode and allows dos code to be loaded into HMA.

EMM386 runs dos in VM86 mode and allows devices to be loaded into UMB using 386 paging. EMM386 requires Himem to be loaded first.
Presumably EMM386 itself runs in ring0? And is effectively managing DOS, ie DOS runs on top of EMM386?

Now my problem,
Can you access more than 1 MB in VM86 mode? If not youve just lost the kernel.
If DOS switches back to "unreal" mode, won't that break the paging, and devices will disappear?

How do these play with each other?
And how does DPMI fit into all this? I assume DPMI also wants to run in ring0 and control the GDT and TSS etc

[Brendan]

Hi,

EMM386 runs dos in VM86 mode and allows devices to be loaded into UMB using 386 paging. EMM386 requires Himem to be loaded first.
Presumably EMM386 itself runs in ring0? And is effectively managing DOS, ie DOS runs on top of EMM386?

Like most of the world I haven't touched DOS for nearly 20 years, so I may be "mis-remembering" some things. However...

When ancient machines first hit the 1 MiB barrier someone went and invented "enhanced memory". This worked by having additional RAM where the CPU couldn't access most of it; and then using bank switching in hardware to make selected 64 KiB chunks of this additional RAM appear in the first 1 MiB of the CPU's address space. Of course there was an API applications used to control which 64 KiB chunk of "enhanced memory" were made accessible to the CPU when.

When CPUs improved and could handle more RAM (called "extended memory"), some people continued to support the older API so that the old software (designed for enhanced memory) could take advantage of extended memory. The simplest way to do this was to switch to protected mode and copy the 64 KiB chunks to/from memory above 1 MiB. An alternative way is to use virtual8086 mode and paging to emulate bank switching (and avoid copying 64 KiB chunks); but this is a lot more complicated and I don't think anyone actually did that unless they were using virtual8086 mode for other reasons anyway (e.g. to run ancient DOS software inside Windows95).

Can you access more than 1 MB in VM86 mode? If not youve just lost the kernel.

You can only access 1 MiB of RAM in virtual8086 mode at any point in time (but can use 1234 GiB of RAM in virtual8086 mode if you don't need to access all of it at the same time).

And how does DPMI fit into all this? I assume DPMI also wants to run in ring0 and control the GDT and TSS etc

DPMI was another API that allowed software to access the additional memory more directly (e.g. using protected mode, without the "only 1 MiB at any point in time" restriction of real mode and virtual8086 mode).


Cheers,

Brendan

[linguofreak]

When CPUs improved and could handle more RAM (called "extended memory"), some people continued to support the older API so that the old software (designed for enhanced memory) could take advantage of extended memory. The simplest way to do this was to switch to protected mode and copy the 64 KiB chunks to/from memory above 1 MiB. An alternative way is to use virtual8086 mode and paging to emulate bank switching (and avoid copying 64 KiB chunks); but this is a lot more complicated and I don't think anyone actually did that unless they were using virtual8086 mode for other reasons anyway (e.g. to run ancient DOS software inside Windows95).

From what I've read, bank switching using paging was used almost exclusively by EMMs developed after the 386 came out.

[Owen]

Memory standards:

• EMS, Enhanced Memory System, was the oldest API. Originally, as Brendan said, designed for bank-switching expansion cards. Supports multiple but small windows into this expansion card memory. With the 386, EMS was normally implemented using paging and v8086 mode
• XMS, Extended Memory System, was invented with the 286. Based around copying, you would ask the XMS server to copy blocks back and forth between extended and real memory
• Himem enabled use of the (64kb-16b) of extra memory available when the A20 gate is disabled. Specifically designed applications could ask for memory here; and Himem provided APIs to turn A20 on and off as necessary. If DOS was loaded high, A20 would be enabled all the time, and a special bit of the upper memory block would be allocated to simulate the wraparound DOS itself required (In particular, the segment wrap around was required for the very clever mechanism by which DOS emulated the CP/M CALL 5 API)

On 386s and above, EMS and XMS would normally be implemented by turning on protected mode and then running DOS inside v8086 mode. You might immediately spot that this would prevent DOS apps from entering protected mode, and you'd be right. Hence

• VCPI, Virtual Control Program Interface, which enabled an application to ask whoever enabled protected mode "Please put me in ring 0". Normally VCPI would be implemented by your EMS/XMS server
• DPMI, DOS Protected Mode interface, which provides a protected mode ring 3 runtime environment for DOS. DPMI abstracts away slightly more than just the mode switching, and was devised by Microsoft while working on Windows 3.x, to enable such DOS apps to run inside Windows (an important compatibility feature to support, say, games which, at the time, were all DOS apps)

Normally an app would come with its' own DPMI server, which would be used if one wasn't already running (e.g. you were running under Windows). This DPMI server would use VCPI if necessary, else manually enter protected mode. The DPMI server would normally use XMS to allocate high memory (because it doesn't need the paging feature of EMS)

Windows 3.x provides an interesting case study of this: It was a VCPI client, permitting it to load on top of EMM386 and such, and a DPMI server, for apps inside a Windows 3.x DOS box.

[linguofreak]

I'm confused.
I'm not sure if this is a question about memory or CPU modes or just how Dos works.

As far as I can tell Himem puts dos into "unreal" mode and allows dos code to be loaded into HMA.

No. Himem does manage the HMA, but it uses straight real mode or v86 mode to do it. Unreal mode is a kludge that some DOS application programs used, but was generally not used by system software (although I have heard that many BIOSes used it quite extensively).

EMM386 runs dos in VM86 mode and allows devices to be loaded into UMB using 386 paging. EMM386 requires Himem to be loaded first.
Presumably EMM386 itself runs in ring0? And is effectively managing DOS, ie DOS runs on top of EMM386?

Yes, more or less, though the relationship between DOS and an EMM (such as EMM386) or DPMI host was rather incestuous, as your EMM would make calls to DOS for system services as well.

Now my problem,
Can you access more than 1 MB in VM86 mode? If not youve just lost the kernel.

You can access up to 1 MB + 64kB - 16B (look up the A20 line for more information). That 64kB - 16B above the 1 MB mark is the HMA.

If DOS switches back to "unreal" mode, won't that break the paging, and devices will disappear?

As I've said, unreal mode was generally not used by system software. In any case, unreal mode involved fiddling around with segmentation, so I don't think it would have screwed up paging.

How do these play with each other?
And how does DPMI fit into all this? I assume DPMI also wants to run in ring0 and control the GDT and TSS etc

DPMI was an API to allow protected mode programs to make calls back to DOS (which has to run in real or v86 mode). A program called a DPMI server would serve as the go-between bridging DOS and the application. EMM386 contained a DPMI server, but many DOS applications contained a built-in DPMI server so that they could run if there wasn't already a DPMI server running.

[alexfru]

No. Himem does manage the HMA, but it uses straight real mode or v86 mode to do it. Unreal mode is a kludge that some DOS application programs used, but was generally not used by system software (although I have heard that many BIOSes used it quite extensively).

No. Himem.sys' primary function is to allocate memory above the 1MB mark (above 1MB+64KB, probably) and copy to that memory from the memory below the 1MB mark and the other way around. To gain access to memory at, say, physical address of 2MB one would need to switch into protected mode. v86 has nothing to do with this per se and is available only on the 80386, while himem.sys works perfectly on the 80286. Himem.sys would either call a dedicated BIOS function to copy the memory in protected mode (int 15h, fxn 87h) or do it directly. I had to intercept and emulate this BIOS function in my v86 monitor in order for himem.sys, smartdrv and the rest of DOS to work in v86 as if nothing has happened.

[linguofreak]

No. Himem does manage the HMA, but it uses straight real mode or v86 mode to do it. Unreal mode is a kludge that some DOS application programs used, but was generally not used by system software (although I have heard that many BIOSes used it quite extensively).

No. Himem.sys' primary function is to allocate memory above the 1MB mark (above 1MB+64KB, probably) and copy to that memory from the memory below the 1MB mark and the other way around. To gain access to memory at, say, physical address of 2MB one would need to switch into protected mode. v86 has nothing to do with this per se and is available only on the 80386, while himem.sys works perfectly on the 80286. Himem.sys would either call a dedicated BIOS function to copy the memory in protected mode (int 15h, fxn 87h) or do it directly. I had to intercept and emulate this BIOS function in my v86 monitor in order for himem.sys, smartdrv and the rest of DOS to work in v86 as if nothing has happened.

OK, I'd missed that it's primary functionality was for XMS, but it still did manage the HMA, and probably did that via straight real/v86 mode, as the HMA is accessible with real-mode addresses with A20 enabled

[mattrix]

Thanks Guys,
A lot to think about.
Sorry for my delay in responding, its a pain when you don't know what your talking about, you have to double check everything. My mind is spinning in ever decreasing spirals...

As I've said, unreal mode was generally not used by system software. In any case, unreal mode involved fiddling around with segmentation, so I don't think it would have screwed up paging.

Are you saying that the TLB and paging registers are not changed going to v86 or real mode?

Are you sure, that in v86 mode you can access 00000h - FFFEFh?

Can/does himem use EMM386 services to do its memory copying?

[alexfru]

Are you saying that the TLB and paging registers are not changed going to v86 or real mode?

Page translation is not available in real mode (and must be disabled when switching to it from protected mode). No TLB, no nothing. Simple as that.

Page translation is optional in protected and v86 modes.

Are you sure, that in v86 mode you can access 00000h - FFFEFh?

I don't see a reason why not.
UPD: You probably meant 0-10FFEFh, right? If there's no page translation, the linear/virtual address goes out as a physical address and the A20 mask may affect it, irrespective of the CPU mode. If there's page translation, you can map these last ~64KB to wherever you want and the A20 mask may affect the resultant physical address (e.g. you could still have A20 enabled and have the address wrap around at the 1MB mark, all thanks to page translation). IOW, the A20 mask knows nothing of what's happening to addresses before they reach the address bus on the CPU pins. You can have A20 disabled/masked out and access more than 1MB memory, just use every other megabyte: 0-1MB, 2MB-3MB, 4MB-5MB, etc.

Can/does himem use EMM386 services to do its memory copying?

If I remember correctly, of all memory drivers himem.sys is always loaded first, before EMM386, QEMM and such. EMM386 would need to get some memory for its own purposes, so, where would it get it from in the first place? It would ask himem.sys. But, of course, if EMM386 simply enables v86, BIOS' int 15h fxn 87h wouldn't work if himem.sys tried to use it because BIOS would fail to switch into protected mode. So, EMM386 would need to intercept this BIOS call and emulate it, just like I said earlier.

[mattrix]

Thankyou,
that makes more sense.

I'd under-valued EMM, (after all it only provides UMB), but now I see it is doing a whole lot more.

I don't see a reason why not.

I've just seen 1 MB written so often, that I thought it was a hard limit.

UPD: You probably meant 0-10FFEFh, right?

Oops, yes.

Can EMM386 and DPMI both be loaded at the same time?

They both want control of the gdt etc. Who wins? and what does the other one do?

[alexfru]

I've just seen 1 MB written so often, that I thought it was a hard limit.

Try proofreading Intel manuals (on x86 CPUs (crosscheck with earlier versions all the way back to 1986's i80386 and with AMD's) and IGBE/XGBE network cards (especially, while crosschecking with their BSD/Linux driver code)). You'll see a lot of inconsistency, typos and omissions.

UPD: You probably meant 0-10FFEFh, right?

Oops, yes.

Can EMM386 and DPMI both be loaded at the same time?

They both want control of the gdt etc. Who wins? and what does the other one do?

See above:

VCPI, Virtual Control Program Interface, which enabled an application to ask whoever enabled protected mode "Please put me in ring 0". Normally VCPI would be implemented by your EMS/XMS server
...
This DPMI server would use VCPI if necessary, else manually enter protected mode.

[mallard]

BTW, there's no such thing as "Enhanced" memory.

EMS is the Expanded Memory Specification. (The bank-switching arrangement formalised by LIM - Lotus, Intel, Microsoft.)

XMS is the eXtended Memory Specification. This term was commonly used to mean all of the >1MB memory available in protected mode, but strictly refers to the API provided by memory managers like HIMEM.SYS that manage this memory under DOS. XMS allows, for instance, a TSR to use memory above 1MB without clobbering a running application that's also using the "high" memory.

HIMEM.SYS provides an API that manages all of the >1MB RAM (including the 64k UMB, which is directly accessible from real mode) which is why it needs to be loaded before EMM386 (so that EMM386 can use some of the >1MB RAM to emulate LIM EMS). HIMEM.SYS also manages the A20 line.

[linguofreak]

I don't see a reason why not.

I've just seen 1 MB written so often, that I thought it was a hard limit.

It is a hard limit. But it's a different hard limit depending on your processor and on whether A20 is enabled or disabled. On an 8086, or with A20 disabled, it's exactly 1 MB, and when 0x10 * seg + offset is greater than 0xFFFFF, the address wraps back around to 0 (so FF00:2000, for example, would be equivalent to 0000:1000, and both have a linear address of 0x1000). On post-286 processors with A20 enabled, the hard limit is 1 MB + 64kB - 16 B, and there's no wraparound (so FF00:2000 is 0x101000 rather than 0x1000) . But it's cumbersome to write "1 MB + 64kB - 16 B" every time you want to refer to the real mode address limit, so most people just write "1 MB" and figure it's close enough.

As to your question "Are you sure, that in v86 mode you can access 00000h - 10FFEFh?", the answer is, "if you are on a 286 or later with A20 enabled, you absolutely can".

Can EMM386 and DPMI both be loaded at the same time?

EMM386 can be loaded. DPMI, per se, cannot.

EMM386 is a memory manager.

DPMI is a standardized set of services that a memory manager can provide.

EMM386 provides DPMI.

Think of DPMI as a language that a program can use to talk to a memory manager, and EMM386 as a memory manager that understands DPMI. So any time EMM386 is loaded, DPMI is "loaded" as well.

[Owen]

EMM386 can be loaded. DPMI, per se, cannot.

EMM386 is a memory manager.

DPMI is a standardized set of services that a memory manager can provide.

EMM386 provides DPMI.

Think of DPMI as a language that a program can use to talk to a memory manager, and EMM386 as a memory manager that understands DPMI. So any time EMM386 is loaded, DPMI is "loaded" as well.

Nope, see my list above. EMM386 provides VCPI, but not DPMI. Pretty much all DPMI servers (e.g. CWSDPMI, HDPMI, the ones included in DOS/4GW etc) will use VCPI to activate protected mode.

You may not realise this because most apps which need DPMI ship with their own DPMI server in case you don't already have one.

[alexfru]

As to your question "Are you sure, that in v86 mode you can access 00000h - 10FFEFh?", the answer is, "if you are on a 286 or later with A20 enabled, you absolutely can".

You mean 80386, of course, as 80286 had no v86 mode.
Off by 100 error. Close enough.