OSDev.org

I used to set a MTRR for LFB to write-combining mode to speed-up my gfx (about 20x). It worked fine on my system with 2GB RAM (C2D, E8400, P31+ICH7). But when I upgraded to 4GB (2x 2GB DIMM) the MTRR setting lost effect. I started to investigate whats wrong. Here's a MTRR dump before:

VESA 3.0 NVIDIA [262144 kB]
LFB address: E0000000h
base mode mask
MTRR #0 = 000000000, 06; 000000000, used
MTRR #1 = 0E0000000, 00; 0E0000000, used
MTRR #2 = 000000000, 06; 0E0000000, used
MTRR #3 = 0DFF00000, 00; 0FFF00000, used
MTRR #4 = 000000000, 00; 000000000, unused
MTRR #5 = 000000000, 00; 000000000, unused
MTRR #6 = 000000000, 00; 000000000, unused
MTRR #7 = 000000000, 00; 000000000, unused
MTRR area E0000000-EFFFFFFFh was set to mode: WC

and after LFB set by my tool:
VESA 3.0 NVIDIA [262144 kB]
LFB address: E0000000h
base mode mask
MTRR #0 = 000000000, 06; 000000000, used
MTRR #1 = 0E0000000, 01; 0F0000000, used
MTRR #2 = 000000000, 06; 0E0000000, used
MTRR #3 = 0DFF00000, 00; 0FFF00000, used
MTRR #4 = 000000000, 00; 000000000, unused
MTRR #5 = 000000000, 00; 000000000, unused
MTRR #6 = 000000000, 00; 000000000, unused
MTRR #7 = 000000000, 00; 000000000, unused
MTRR area E0000000-EFFFFFFFh was set to mode: WC

This is dumped from DOS so it's not affected by other software.
It's BIOS default. Then I looked how WinNT set the MTRRs because
they run fast gfx, dump:

VESA 3.0 NVIDIA [262144 kB]
LFB address: E0000000h
base mode mask
MTRR #0 = 000000000, 06; 080000000, used
MTRR #1 = 080000000, 06; 0C0000000, used
MTRR #2 = 0C0000000, 06; 0E0000000, used
MTRR #3 = 0DFF00000, 00; 0FFF00000, used
MTRR #4 = 0E0000000, 01; 0F0000000, used
MTRR #5 = 0F0000000, 06; 0F0000000, used
MTRR #6 = 000000000, 06; 0E0000000, used
MTRR #7 = 000000000, 00; 000000000, unused
MTRR area E0000000-EFFFFFFFh was set to mode: WC

If I set MTRRs same way in DOS it also starts work fast.
I reduced the list and found that most important is MTRR0.
If I set
base mode mask
MTRR #0 = 000000000, 06; 080000000, used
it works fast, if I set
base mode mask
MTRR #0 = 000000000, 06; 000000000, used
it works slow regardless on other settings.

So the question is what's wrong with this setting and how to make best fix.
1) is mask 000000000 valid?
2) I'm not sure, if it covers entire 4GB area then it casue overlaping
with MTRR for LFB area. If 2 areas overlaps I guess that less caching
is used, in this case write-back for LFB instead write-combining,
am I right? If so, this means that I'll need to split RAM area in 2
(or more parts) and program 2 MTRRs. It should be job for BIOS to
initialize it properly but what can I do with it than fix it myself...

Well, I reply myself, maybe someone else will be interested...
First I found a bug in printing MTRRs that caused it overflow after <<12 and the top nibble was displayed always zero. So it should be:

before:

VESA 3.0 NVIDIA [262144 kB]
LFB address: E0000000h
MTRR #0: base = 000000000h ( 0MB), mask = F00000000h (4096MB), WB, used
MTRR #1: base = 0E0000000h (3584MB), mask = FE0000000h ( 512MB), UC, used
MTRR #2: base = 100000000h (4096MB), mask = FE0000000h ( 512MB), WB, used
MTRR #3: base = 0DFF00000h (3583MB), mask = FFFF00000h ( 1MB), UC, used
MTRR #4: base = 000000000h ( 0MB), mask = 000000000h ( 0MB), UC, unused
MTRR #5: base = 000000000h ( 0MB), mask = 000000000h ( 0MB), UC, unused
MTRR #6: base = 000000000h ( 0MB), mask = 000000000h ( 0MB), UC, unused
MTRR #7: base = 000000000h ( 0MB), mask = 000000000h ( 0MB), UC, unused
Setting MTRR #1...
MTRR area E0000000-EFFFFFFFh was set to mode: WC

after:

VESA 3.0 NVIDIA [262144 kB]
LFB address: E0000000h
MTRR #0: base = 000000000h ( 0MB), mask = F80000000h (2048MB), WB, used - my modification
MTRR #1: base = 0E0000000h (3584MB), mask = FF0000000h ( 256MB), WC, used - my modification
MTRR #2: base = 100000000h (4096MB), mask = FE0000000h ( 512MB), WB, used
MTRR #3: base = 0DFF00000h (3583MB), mask = FFFF00000h ( 1MB), UC, used
MTRR #4: base = 000000000h ( 0MB), mask = 000000000h ( 0MB), UC, unused
MTRR #5: base = 000000000h ( 0MB), mask = 000000000h ( 0MB), UC, unused
MTRR #6: base = 000000000h ( 0MB), mask = 000000000h ( 0MB), UC, unused
MTRR #7: base = 000000000h ( 0MB), mask = 000000000h ( 0MB), UC, unused
Setting MTRR #1...
MTRR area E0000000-EFFFFFFFh was set to mode: WC

It seems that my BIOS has permanently enabled some memory remapping that takes
the last 0.5GB and maps it above 4GB boundary - MTRR2. The problem is caused by
MTRR0 that set entire 4GB to WB and it overlaps LFB so LFB cannot work in WC mode
as lower caching has higher priority. I had to patch this MTRR with base 0, mask F000000
(searching for it) and change the area to 2GB. I assume that LFB will be higher and it
will not overlap then. It works now fine on my system.

But I think more clean way would be to do it via Page Attribute Table. I wonder if someone has a tutorial how to set PAT via MSRs. I'm reading intel SWDM it's not clear to me yet...

Hi,



That makes more sense!



Except now the 1535 MiB of RAM from 0x80000000 to 0xDFF00000 is set to "uncached" (instead of "write-back"), which will cause a serious performance problem if/when you start using that RAM.

To fix that you'd want something like:


Of course you'd need to write code to generate the best MTRR setup (you can't just hard-code these values), which won't be easy.

The alternative would be to leave the video RAM as "uncached" (the same as the firmware left it), and then use the PAT feature in page table entries for video RAM to change it to "write-combining". This is probably a lot easier (and may be necessary on some systems if you run out of MTRRs).


Cheers,

Brendan

Thanks for reply.

Yes, you are right that I need more complex algo to set MTRRs. I collected dumps from other machines so now I know what can I expect. Various BIOSes do it different way. A lot of years I was happy with code that only setup one MTRR for LFB. Nowdays when 4GB and more become common this approach fails. It seems that many BIOSes setup 4GB range that covers all and then define some uncached holes and even some use another overlaping area like this i5 machine:

VESA 3.0 NVIDIA [14336 kB]
LFB address: CF000000h
MTRR #0: base = 000000000h ( 0MB), mask = F80000000h (2048MB), WB, used - patched to not overlap LFB
MTRR #1: base = 200000000h (8192MB), mask = FE0000000h ( 512MB), WB, used
MTRR #2: base = 220000000h (8704MB), mask = FF0000000h ( 256MB), WB, used
MTRR #3: base = 230000000h (8960MB), mask = FFC000000h ( 64MB), WB, used
MTRR #4: base = 0CC000000h (3264MB), mask = FFC000000h ( 64MB), UC, used - but this bastard overlaps LFB too. It covers some MMIO? that should be uncached but I really don't know what area is really used. If I can shrink it to 32MB. This is BIOS specific issue that will complicate MTRR setting routine.
MTRR #5: base = 0D0000000h (3328MB), mask = FF0000000h ( 256MB), UC, used
MTRR #6: base = 0E0000000h (3584MB), mask = FE0000000h ( 512MB), UC, used
MTRR #7: base = 0CF000000h (3312MB), mask = FFF800000h ( 8MB), WC, used - my LFB setting, now with correct continuous mask
Setting MTRR #7...
MTRR area CF000000-CFDFFFFFh was set to mode: WC

After I cleared MTRR4 LFB transfer rate rocket jumped from ~160MB/s to 2000MB/s

Well so I need some loop with multiple checks that sort MTRRs:
1) base+size < LFB_base : keep
2) base > LFB_base+LFB_size : keep
3) LFB_base > base > LFB_base+LFB_size : clear it
4) (base < LFB_base) && (base+zise > LFB_base) : try to shrink it's mask by 2^n until base+zise will be under LFB_base
then cover as much as possible uncached RAM below LFB by setting MTRRs for 2GB+1GB+0.5GB+256MB...


I have general question to Page Attribute Table. Is it working even when paging not enabled? I have some old DOS SW that runs in protected mode to work with LFB but it doesn't use paging. And when I set PAT by one program, will the PAT setting remain preserved if the program returns to real or v86 mode and then I start another program that enables paging?
I only know that MTRR settings will persist until cold reboot so other programs or operating systems that can't set MTRR itself can gain the performace when it was preconfigured by my tool.

Hi,



I don't think that'd cover all the possibilities (e.g. consider inserting a write-combining area into the middle of an uncached area, such that the uncached area needs to be split into 2 smaller uncached areas separated by the new write-combining area).

Instead, I'd create an array of entries/structures (where each entry/structure contains the start address, end address and type for each area) to keep track of what you want; and then have a routine to convert what you want into MTRR register settings. During boot you'd extract current info from the MTRRs to generate your array of structures, but after that you'd modify your array of structures and use it to regenerate all MTRR registers. To make multi-CPU support easier, I'd generate a list of "new MTRR contents" in RAM. Think of it like this:

• Acquire lock
• Insert/remove area into the "what you want" array of entries/structures
• Convert the "what you want" array of entries/structures into a "MTRR settings" buffer in RAM
• Follow Intel's advice (in the section called "MTRR Considerations in SMP Systems" in their manual) to load the MTRR settings from your buffer in RAM into all CPUs at the same time
• Release lock

To determine what MTRR entries you need for each area, there's 2 general methods - "additive" and "subtractive". Additive is the normal case where you don't create any entries for "uncached"; and where you split an area into the largest blocks that conform to MTRR alignment and size rules. For "additive" one or more MTRRs are added together to describe an area. Subtractive is where you use an oversized area and "subtract" one or more (uncached) pieces from it.

The general idea would be; for each area in the list, work out how many entries "additive" would cost, then determine if "subtractive" is possible (e.g. the additive method costs more than one entry, the missing pieces are uncached and not something else, etc) and then (if subtractive is possible) work out how many entries "subtractive" would cost; then use the method that costs the least entries.

However, I wouldn't do this in order from lowest to highest address. I'd start by doing all the "write back" areas first, then do the "write through" areas, then the "write protected" areas, and only worry about the "write combining" areas last. The reason for this is that you will have to stop if/when you run out of MTRR registers. If there aren't enough MTRR registers for everything then you end up with "uncached" where you don't want it. For "write-combining" this isn't much of a problem because you can override the default ("uncached") from the MTRR using the PAT; which is why I'd do "write-combining" areas last. For "write protected" areas, you can't override it with the PAT (without any performance loss), but if you have to use "uncached" instead then it's probably not going to hurt performance much anyway; which is why I'd to "write protected" second to last.



You must be using paging to use the PAT; and ancient CPUs don't support it (it's P6 or later I think - check the corresponding feature flag in CPUID). For 80486 and Pentium there's only a "write-through" flag and an "uncached" flag in the page table entries that can be used instead. For 80386 there's nothing (no flags to control caching in page table entries and no MTRRs either). There's also no MTRRs on 80486. For some CPUs from other manufacturers there's something different instead of MTRRs. For example, Cyrix CPUs use "Address Range Registers" and "Region Control Registers" that are a little bit like Intel's MTRRs but are accessed via. IO ports and not MSRs. Ironically, Cyrix had this (and "write gathering") before Intel "invented" MTRRs and write combining.



If you want to use PAT then you have to use paging. It is possible to run most DOS applications under virtual8086 mode with "identity mapped" paging and use the PAT; but I don't know if there's any DOS extenders that actually do use paging (I doubt it). Fortunately, crusty old DOS applications are designed for crusty old slow computers, so you're more likely to have a "computer too fast" problem than a "computer too slow" problem and could probably forget about write combining completely.


Cheers,

Brendan

Well I think about this approach. But it grows in complexity that I never expected it would reach
I know that MTTRs are limited (usually only and some "badly" aligned areas cannot be fully covered even with 8 MTRRs so I can end up with all MTTRs full and the original purpose of configuring LFB get lost. But for my case I can expect that DOS progs usually don't access too high RAM physical address and also I can set memory manager to limit maximum available memory so then I would be happy with 2GB writeback area. I have higher priority to set LFB WC than utilize the last magabite somewhere beyond 3,5MB so I would use max up to 3 MTRRs for WB regins... I don't care about SMP setting in my case.

Under DOS I usually run with memory manager that switch CPU to V86 mode at boot and enable paging - Jemm386 (http://www.japheth.de/Jemm.html) is very good and recent. The DPMI servers CWSDPMI and HDPMI and DOS4GW and PharLap 386 extenders also use paging. Will then PAT work when I configure it by one program and then run another pmode program that doesn't turn off paging? There are some PM/V86 mode switches during running programs and between.
But not all extenders use paging. e.g. PMODE/W use PM without paging and for some operations it swithches to RM and back. In this case it will not work.

BTW not all DOS sw is obsolete, there are also projects uder development, various ports from other platforms, etc.
And old existing sw like CAD programs and games will utilize fast LFB too (if you get poor uncached 20MB/s it's still worse than good PCI VGA 10 years ago)

Hi,



No. When you exit one program everything is "destroyed" and returned to the state DOS expects (real mode, no paging), so if one program sets up PAT then it'd be gone when that program exits and the next program would have to set it up again.

Are any of these DOS extenders open source (I think the one that comes with DJGPP is)? Your best option would be to find an open source DOS extender (that already uses paging) and modify it so that it sets up the MTRRs and/or PAT during startup; and then use that DOS extender everywhere you can.



You're confusing "dead" with "obsolete".

DOS is both dead and obsolete. Some software/applications for DOS may be alive, but even though they're alive they're also obsolete (because DOS is dead). For example, FreeDOS is still alive (and still being maintained), but it was obsolete when people started the project in 1994, and it will remain "alive and obsolete" until it becomes "dead and obsolete".


Cheers,

Brendan

Even if program returns to V86 mode where paging is enabled all the time?



Yes, HDPMI and CWSDPMI servers are opensource and are still maintained but not very often. Simply beacuse they are mature by years of development and there was nothing much to fix/add. I contact authors first what they do thing about implementing this service. Also it could be implemented in JEMMEX memory manager that is resident during all time and may be usefull for other apps...



I didn't meant DOS itself, it is obsolete of course by design. But software for DOS may not. Would you say that e.g. recent mplayer SVN build is obsolete just because it's ported to DOS? Yes it is slightly limited in some features but some other ports. e.g. for ARM or other mobile/embedded devices has some limitations too.