Detecting memory above 4G

[rdos]

The wiki and various discussions say that "use GRUB or BIOS" for memory map. I tend to agree with this for memory below 4G, but wouldn't it be relativey safe to simply detect memory above 4G? In a 32-bit setup this could be done with PAE-paging, as it is impossible to do with a unity-mapping.

[thepowersgang]

Devices can still be mapped >4GiB for certain reasons, and probing can cause other undefined behavior (including complete bus lock on some processors).

It is always best to use the firmware memory map if it's avaliable.

[Brendan]

Hi,

The wiki and various discussions say that "use GRUB or BIOS" for memory map. I tend to agree with this for memory below 4G, but wouldn't it be relativey safe to simply detect memory above 4G? In a 32-bit setup this could be done with PAE-paging, as it is impossible to do with a unity-mapping.

It is relatively safe to simply detect memory above 4G using GRUB's memory map or the BIOS memory map.

It is *not* safe to detect memory above 4 GiB using manual probing. There is no guarantee that the end of RAM and the end of usable RAM is the same thing (e.g. firmware may be using some space at the end of RAM for it's own purposes, especially in modern "UEFI and BIOS support" systems). There is also no guarantee that the firmware didn't put a memory mapped PCI device immediately after then end of usable RAM; and if I remember correctly, due to some issue with the way memory controllers in opterons work, AMD were recommending (to firmware manufacturers) that PCI devices are mapped immediately after the RAM.

Why are you asking? Surely you're already using GRUB's memory map or the BIOS memory map, and continuing to use the memory map you've already obtained is the easiest option.


Cheers,

Brendan

[rdos]

Why are you asking? Surely you're already using GRUB's memory map or the BIOS memory map, and continuing to use the memory map you've already obtained is the easiest option.

Erm, yes, and no. I left-shift the reported memory blocks by 10 and then save the number of bytes as a 32-bit int. The structure passed between boot-loader and kernel only uses 32-bit memory fields. I thought a "quick-and-dirty" solution would be to just manually probe, but I suppose I need to redo the grub-bootloader. In addition to that, I also need extra fields passed to kernel, which is more tricky to achieve on already installed systems.

Looking at the structure, it is evident there are a few unused fields that could be used for memory above 4G. To fix the issue with existing systems, I could detect that end of RAM is at 4G (which no installed system has), and use an extra field for memory above 4G.

This is really an interface I would have liked to have redone from scratch, but I'm unable to because of existing installed boot-loaders.

[Brendan]

Hi,

Why are you asking? Surely you're already using GRUB's memory map or the BIOS memory map, and continuing to use the memory map you've already obtained is the easiest option.

Erm, yes, and no. I left-shift the reported memory blocks by 10 and then save the number of bytes as a 32-bit int. The structure passed between boot-loader and kernel only uses 32-bit memory fields.

I think you mean "no".

If you left shift by 10 to get the number of bytes; then you must've started with the number of KiB. The memory map provided by GRUB and/or BIOS "int 0x15, eax = 0xE820" doesn't give you "number of KiB". You're not using the memory map, you're using Multi-boot's "mem_lower" and "mem_upper" fields.

This means that:

• Because the "mem_upper" frequently reports less RAM than it could/should (due to holes, limits, etc), you'd often ignore some RAM below 4 GiB that could've been used (without PAE/long mode).
• You never reclaim/free the "ACPI reclaimable" area.
• You can't support things like "hibernate" where the OS is meant to save and restore "ACPI non-volatile" areas because you don't know where these areas are.
• You can't safely reconfigure PCI device BARs because you don't know which areas are used/reserved by ROM, etc and which areas are safe for memory mapped devices.

Now you're looking at RAM above 4 GiB (for PAE and long mode). Maybe one day you'll also be looking at UEFI (which provides a memory map similar to the memory map you get on BIOS systems, that may describe a physical address space that looks nothing like what you'd get on PC BIOS systems).

Looking at the structure, it is evident there are a few unused fields that could be used for memory above 4G.

It would take four (32-bit) fields just to store one 64-bit "start address" and one 64-bit "length". If a system has 6 different "usable RAM" areas above 4 GiB, then you'd need 24 of these (32-bit) fields. How many is "a few"?

To fix the issue with existing systems, I could detect that end of RAM is at 4G (which no installed system has), and use an extra field for memory above 4G.

If you set the "end of ram" field to 4 GiB; would that mean you'd be unable to determine how much RAM is at 0x00100000 and have no choice but to waste up to about 3 GiB of RAM?

This is really an interface I would have liked to have redone from scratch, but I'm unable to because of existing installed boot-loaders.

Why?

Surely when someone installs a new version of the OS they also install a new version of the OS's boot code (even if they don't reinstall GRUB)?

Surely your OS has a version number, and you could tell customers that "RDOS version 4.0" isn't compatible with the boot loader used by "RDOS version 3.123 and earlier"?

Surely you've got some sort of "structure version number" field (so your kernel knows if the boot code is new or not and can act accordingly), or perhaps a "structure length" field (so you can append new fields for any reason whenever you like)?


Cheers,

Brendan

[gravaera]

Surely! Surely!! He hath borne our griefs and carried our sorrows

[rdos]

If you left shift by 10 to get the number of bytes; then you must've started with the number of KiB. The memory map provided by GRUB and/or BIOS "int 0x15, eax = 0xE820" doesn't give you "number of KiB". You're not using the memory map, you're using Multi-boot's "mem_lower" and "mem_upper" fields.

Yep, I use the mem_lower and mem_upper fields.

Because the "mem_upper" frequently reports less RAM than it could/should (due to holes, limits, etc), you'd often ignore some RAM below 4 GiB that could've been used (without PAE/long mode).

Not really that important. On modern systems there usually are hundreds of MB free physical memory. Some even have several GBs. We no longer live in the time of severe memory limits. If the reported amount is a few MB lower, nobody would care.

You never reclaim/free the "ACPI reclaimable" area.

No, why should I? It is relatively small, and insignificant on modern systems.

You can't support things like "hibernate" where the OS is meant to save and restore "ACPI non-volatile" areas because you don't know where these areas are.

No, I don't support hibernate, and I find it unlikely I will ever need to.

You can't safely reconfigure PCI device BARs because you don't know which areas are used/reserved by ROM, etc and which areas are safe for memory mapped devices.

There is no reason to reconfigure PCI BARs. BIOS does this good enough, so why reinvent the wheel?

Now you're looking at RAM above 4 GiB (for PAE and long mode). Maybe one day you'll also be looking at UEFI (which provides a memory map similar to the memory map you get on BIOS systems, that may describe a physical address space that looks nothing like what you'd get on PC BIOS systems).

Possible. I'll deal with that as I encounter it.

It would take four (32-bit) fields just to store one 64-bit "start address" and one 64-bit "length". If a system has 6 different "usable RAM" areas above 4 GiB, then you'd need 24 of these (32-bit) fields. How many is "a few"?

I simply will not support such a horrible memory-setup!

OTOH, an alternative approach would be to save the pointer that GRUB passes (only requires one 32-bit word), and parse it in the kernel startup that creates the free memory pool. But that would make the kernel bound to Multiboot.

[Owen]

I simply will not support such a horrible memory-setup!

Not that far from uncommon. Multi-socket systems often have discontiguous RAM and holes, because they have multiple RAM controllers.

OTOH, an alternative approach would be to save the pointer that GRUB passes (only requires one 32-bit word), and parse it in the kernel startup that creates the free memory pool. But that would make the kernel bound to Multiboot.

Not really bound to Multiboot. The GRUB memory map is essentially just the BIOS E820 one (defined by ACPI) that any boot loader can use (and is encouraged to do so: it's most reliable).

It's not an ideal format (the EFI format would be better: it confers more information), but its a workable one (e.g. Linux uses it to get its memory map from the bootloader. Yes, on EFI systems, they down convert to the BIOS format, and then some EFI specific portions go back to the original format to extract data *shudder*)

[Brendan]

Hi,

OTOH, an alternative approach would be to save the pointer that GRUB passes (only requires one 32-bit word), and parse it in the kernel startup that creates the free memory pool. But that would make the kernel bound to Multiboot.

Passing a pointer to the memory map sounds like the best possible alternative to me!

It's not an ideal format (the EFI format would be better: it confers more information), but its a workable one (e.g. Linux uses it to get its memory map from the bootloader. Yes, on EFI systems, they down convert to the BIOS format, and then some EFI specific portions go back to the original format to extract data *shudder*)

Heh - I never really liked that idea either.

My approach is for the boot loader to construct it's own "meta map", that combines all info from BIOS (with the "flags") or EFI (with the caching info), with NUMA information (ACPI SRAT), plus it's own information about what is "currently in use" by the boot loader (so that if the memory map says something is free the kernel can allocate/use it and not have to worry that it's used by something like multi-boot information, itself or the boot loader). Then there's a few special area types (e.g. "type = mixed" for when firmware can't make up its mind).

There's also the "holes" that BIOS and EFI don't mention. I categorise them as "safe for memory mapped PCI" and "unsafe for memory mapped PCI" and add them to my memory map too, so that every single address is covered by something in the memory map.


Cheers,

Brendan

[Owen]

Heh - I never really liked that idea either.

My approach is for the boot loader to construct it's own "meta map", that combines all info from BIOS (with the "flags") or EFI (with the caching info), with NUMA information (ACPI SRAT), plus it's own information about what is "currently in use" by the boot loader (so that if the memory map says something is free the kernel can allocate/use it and not have to worry that it's used by something like multi-boot information, itself or the boot loader). Then there's a few special area types (e.g. "type = mixed" for when firmware can't make up its mind).

There's also the "holes" that BIOS and EFI don't mention. I categorise them as "safe for memory mapped PCI" and "unsafe for memory mapped PCI" and add them to my memory map too, so that every single address is covered by something in the memory map.


Cheers,

Brendan

UEFI reserves all ID types above 0x80000000 (IIRC) for the OS to use for its own purposes. I suggest making use of them (that said - you can't pass them to the EFI allocation functions, because up until about a year ago Intel's EFI implementation would go off and dereference invalid pointers if you did that)

[Brendan]

Hi,

UEFI reserves all ID types above 0x80000000 (IIRC) for the OS to use for its own purposes. I suggest making use of them (that said - you can't pass them to the EFI allocation functions, because up until about a year ago Intel's EFI implementation would go off and dereference invalid pointers if you did that)

Ironically...


Cheers,

Brendan

[rdos]

OK, so I've updated the GRUB logic now to use the memmap fields (wasn't that hard to do). An unexpected side-effect of this is that one of my PCs, which have 4G installed, but only 3G available in RDOS, now have 4G available, and 1G is mapped above 4G. That also means I have a PC that have some memory above 4G which I could test further with various devices that only allow memory below 4G to be used.

Although, most important, is that the new design boots much faster as it only has to turn on/off paging one time per GRUB memory map entry, and not for each page, which was really slow.