Questions on boot-time memory detection

[linguofreak]

I've got a few questions about detecting memory:

1) First off, can I generally assume that the high memory area (FFFF:10 - FFFF:FFFF in real mode, 100000 - 10FFF0 physical in protected mode) will be available if the machine in question has any extended memory at all? My bootloader currently assumes that the HMA exists (it loads the payload, as well as the Int 0x15 eax=E820 memory map, into the HMA), and this works on my two test machines and in Bochs, but is this a safe assumption? Are machines that violate it rare/old enough that I can simply decide not to support them, or should I keep my working area entirely under the 640k mark until I've processed the memory map?

2)When cleaning up the E820 memory map, the Wiki recommends changing overlapping areas to the most restrictive type. I'm not entirely clear, though, on which types are most restrictive. Type-1 (usable memory) is certainly the least restrictive, and type-3 (ACPI reclaimable) seems to be next. But what order should I place types 2, 4, and 5 in?

3)I've been looking through the E820 map sanitization function in the Linux kernel (sanitize_e820_map in arch/x86/kernel), and it returns non-success on being passed a map with only one entry (the specific case is "if (*pnr_map < 2)"). Is this based on an assumption that a map with only one entry is certain to be invalid (Presumably because the BIOS is certain to account for at least one type-2 entry and there has to be at least one type-1 entry to have any space to execute non-BIOS code in)?

[Brendan]

Hi,

1) First off, can I generally assume that the high memory area (FFFF:10 - FFFF:FFFF in real mode, 100000 - 10FFF0 physical in protected mode) will be available if the machine in question has any extended memory at all? My bootloader currently assumes that the HMA exists (it loads the payload, as well as the Int 0x15 eax=E820 memory map, into the HMA), and this works on my two test machines and in Bochs, but is this a safe assumption? Are machines that violate it rare/old enough that I can simply decide not to support them, or should I keep my working area entirely under the 640k mark until I've processed the memory map?

It's "safe in general" for PC BIOS systems. There are no standards that require this area to be usable RAM or prevent firmware from using it for things like the "ACPI reclaimable" area or anything else; but (due to "de facto standards" and backward compatibility) it's still safe to use.

However, it is a bad idea to assume that the BIOS can/will safely access the high memory area in real mode. For example, for "Int 0x15 eax=E820" you should ask the BIOS to store an entry in a buffer below the address of the EBDA, and then copy the entry from the buffer to the high memory area yourself.

It's also a bad idea to assume that "below 640 KiB" is safe. For example, the EBDA could be 12 KiB from 0x9D000 to 0x0009FFFF and you may only have 628 KiB of RAM to use.

For UEFI systems there is no guarantees - your boot loader has to be position independent, and it has to use UEFI functions to allocate/free RAM during boot, and use the memory map provided by UEFI to determine what is usable RAM after "exitBootServices()" is called.

This mostly means that you need to worry about extremely ancient computers (with less than 1 MiB of RAM) and new computers (which have UEFI and not BIOS).

2)When cleaning up the E820 memory map, the Wiki recommends changing overlapping areas to the most restrictive type. I'm not entirely clear, though, on which types are most restrictive. Type-1 (usable memory) is certainly the least restrictive, and type-3 (ACPI reclaimable) seems to be next. But what order should I place types 2, 4, and 5 in?

Convert the BIOS types into your own types; such that types 0 to 5 have the same meaning as the BIOS and any type above 5 is changed to 0. Then define a new type 6 as "mixed" and a new type 7 as "refuse to boot". Then, create a lookup table like "int lookupTable[8][8] = {...." so you can do "newType = lookupTable[oldType1][oldType2];".

Note: If/when you start worrying about UEFI you'll need to define a lot more types (and worry about attributes for everything).

My suggestions would be:

• Type 7 mixed with any other type = type "refuse to boot"
• Type 6 mixed with type 5 or 4 = type "refuse to boot"
• Type 6 mixed with type 0, 1, 2 or 3 = type "mixed"
• Type 5 mixed with any other type = type "refuse to boot"
• Type 4 mixed with any other type = type "refuse to boot"
• Type 3 mixed with type 0, 1 or 2 = type "mixed"
• Type 2 mixed with type 0 or 1 = type "mixed"
• Type 1 mixed with type 0 = type 0

The "mixed type" would be treated the same as "reserved" (e.g. OS doesn't touch it).

After you've finished processing everything, display it (including any "refuse to boot" areas) so the user can see what you ended up with. After that, if there are any "refuse to boot" areas display an error message and lock up (wait for the user to read the screen and reset the computer).

3)I've been looking through the E820 map sanitization function in the Linux kernel (sanitize_e820_map in arch/x86/kernel), and it returns non-success on being passed a map with only one entry (the specific case is "if (*pnr_map < 2)"). Is this based on an assumption that a map with only one entry is certain to be invalid (Presumably because the BIOS is certain to account for at least one type-2 entry and there has to be at least one type-1 entry to have any space to execute non-BIOS code in)?

I'd assume it's based on an assumption.


Cheers,

Brendan

[linguofreak]

Hi,

1) First off, can I generally assume that the high memory area (FFFF:10 - FFFF:FFFF in real mode, 100000 - 10FFF0 physical in protected mode) will be available if the machine in question has any extended memory at all? My bootloader currently assumes that the HMA exists (it loads the payload, as well as the Int 0x15 eax=E820 memory map, into the HMA), and this works on my two test machines and in Bochs, but is this a safe assumption? Are machines that violate it rare/old enough that I can simply decide not to support them, or should I keep my working area entirely under the 640k mark until I've processed the memory map?

It's "safe in general" for PC BIOS systems. There are no standards that require this area to be usable RAM or prevent firmware from using it for things like the "ACPI reclaimable" area or anything else; but (due to "de facto standards" and backward compatibility) it's still safe to use.

However, it is a bad idea to assume that the BIOS can/will safely access the high memory area in real mode. For example, for "Int 0x15 eax=E820" you should ask the BIOS to store an entry in a buffer below the address of the EBDA, and then copy the entry from the buffer to the high memory area yourself.

Ooh. I hadn't thought of that. "Will" would certainly be a concern, though I'm having trouble imagining scenarios where "can" would be a concern (assuming that your "safe in general" comment above holds for the system in question). I can't imagine how anything but a programmatic restriction in BIOS (i.e. "won't" rather than "can't") would prevent BIOS from accessing the HMA if non-BIOS code is able to access it safely.

That said, are you aware of any specific systems where the BIOS either won't or actually can't reach the HMA? (Aside, of course, from systems where it is unreachable in general, such as systems with no extended memory, A20 permanently disabled or nonexistent, etc).

It's also a bad idea to assume that "below 640 KiB" is safe. For example, the EBDA could be 12 KiB from 0x9D000 to 0x0009FFFF and you may only have 628 KiB of RAM to use.

I am aware of this. I actually used the phrasing "below the EBDA" in a draft of my OP. (That said, I've just realized that my bootloader does put data above the 512KiB mark, and the Wiki says that the EBDA may be as big as 128KiB, though nothing beyond 8KiB has been observed).

For UEFI systems there is no guarantees - your boot loader has to be position independent, and it has to use UEFI functions to allocate/free RAM during boot, and use the memory map provided by UEFI to determine what is usable RAM after "exitBootServices()" is called.

UEFI is not a big concern for me at this point, as I don't have a test machine that uses it.

[Brendan]

Hi,

That said, are you aware of any specific systems where the BIOS either won't or actually can't reach the HMA? (Aside, of course, from systems where it is unreachable in general, such as systems with no extended memory, A20 permanently disabled or nonexistent, etc).

I'm not aware of any computers where the BIOS can't reach the HMA, but I've never tried either (e.g. I tested zero computers and 100% of them had trouble and 100% didn't).

I'd start by shifting boot code down to 0x00001000, then use "int 0x12" to find out how much usable RAM there is below the EBDA, then use RAM starting at "end of boot code" to create/process the memory map and initialise a temporary physical memory manager (while being careful not to exceed anything above "start of EBDA"), then use the temporary physical memory manager to allocate any RAM used after that.

I'm probably one of the most cautious developers though. For example, the most recent version of my boot code used paging to avoid the "payload is larger than the largest single area of usable RAM" problem (and could even load a 123 MiB payload and boot normally after failing to disable A20).

EFI is not a big concern for me at this point, as I don't have a test machine that uses it.

I don't have a machine that actually uses UEFI at the moment either (but I have one computer that does support it, and 2 virtual machines). I mostly want to avoid design flaws that make it hard to support UEFI in future (e.g. "payload" that expects to be loaded at a specific physical address).


Cheers,

Brendan