Gaps in physical memory address space.. Normal?

[Tyyrus]

Hello all,

Sorry if this has been asked before.. I tried searching but didn't find a result. I'm working on my new OS and UEFI bootloader. I was starting work on my PMM, when I realized something a little funky about my memory map. There seem to be around half a Gb of gaps in the physical memory address space. Here's my memory map returned from GetMemoryMap() to be a bit more clear:

Code: Select all

Type Start Addr (Low bits)   Num Pages
7     0            160
7     1048576      768
2     4194304      2
7     4202496      7172
4     33579008     1
7     33583104     25
4     33685504     2272
7     42991616     9984
2     83886080     1
7     83890176     389055
1     1677459456   10
7     1677500416   491574
4     3690987520   32
7     3691118592   11827
2     3739561984   3
7     3739574272   775
4     3742748672   202
7     3743576064   27
4     3743686656   2836
7     3755302912   18
3     3755376640   366
5     3756875776   48
6     3757072384   36
0     3757219840   4
9     3757236224   8
10    3757268992   4
4     3757285376   134
6     3757834240   32
7     3757965312   16
7     4294967296  1179648

This system was set up in VirtualBox to have 8Gb of RAM. Notice at the 4Gb address there are 1179648 available 4KiB frames (4.5 Gb worth), making the final memory address at the 8.5Gb mark even though there's only 8Gb of RAM. If you add up the total number of frames, it adds up to right around 8Gb (about half a Mb shy if I did the calculations correctly, which would make sense if some memory is being reserved by the hardware). So my main question is, is this normal? It would make the memory bitmap more complicated, but workable - I could just map a bit directly to a physical page if I treated the system like it had 8.5Gb of RAM, and marked the "gaps" as already allocated. Something about this seems off to me though. I did notice the UEFI documentation stated

The firmware does not return a range description for address space regions that are not backed by physical hardware.

so I'm not sure if that's related. I'll likely have follow-up questions depending on the responses. Sorry if this is a silly question, I'm new to OSdev.

Edit: Messed up the final table entry, fixed

[Octocontrabass]

Yes, it's normal.

The gaps can be quite large, too - on some hardware I've seen several multi-gigabyte gaps.

[Tyyrus]

Thanks for the response! As a followup, let's assume we're using a simple bitmap to represent the available physical memory as described in https://wiki.osdev.org/Page_Frame_Allocation. Since we have these gaps, what would be the standard way to handle those gaps in the bitmap? I see two ways.

First, as I mentioned in my post above, you could include the gaps in the bitmap as "allocated" space. This would make translation from page frame to physical address easy, but you'd be storing a bunch of data for physical addresses that don't exist. If there were multi-gigabyte gaps as you mentioned is possible, this seems pretty space-inefficient.

As an alternative, you could have the bitmap location be the "page frame number" rather than physical address, and have a separate table (similar to the memory map above) which stores the actual physical addresses available. You'd then use that table to translate the "page frame number" to the actual physical address. This would save space in the bitmap, but would add complexity and time every time you wanted to calculate a physical address.

Am I missing some strategy here? I thought the bitmap was the "simple, direct" way to store physical memory availability, but if you have to translate that bitmap location through a separate table the "simplicity" advantage seems a lot less pronounced. Plus I believe the complexity to set the state of a page frame would no longer be O(1), but O(log(N)) at best, where N is the number of distinct memory "chunks" between the gaps.

[Octocontrabass]

First, as I mentioned in my post above, you could include the gaps in the bitmap as "allocated" space. This would make translation from page frame to physical address easy, but you'd be storing a bunch of data for physical addresses that don't exist. If there were multi-gigabyte gaps as you mentioned is possible, this seems pretty space-inefficient.

I can think of two easy ways you can deal with this.

One way is just to ignore it! Even though the gaps may be several gigabytes in size, the space they'll take in your bitmap is quite small relative to both the amount of memory and the amount of virtual address space available to a typical 64-bit kernel.

If you're only concerned about physical memory usage and not virtual address space, you can use some page table trickery to turn your bitmap into a sparse bitmap by remapping all of the parts of the bitmap that will never indicate usable memory to the same read-only page.

[Tyyrus]

First, as I mentioned in my post above, you could include the gaps in the bitmap as "allocated" space. This would make translation from page frame to physical address easy, but you'd be storing a bunch of data for physical addresses that don't exist. If there were multi-gigabyte gaps as you mentioned is possible, this seems pretty space-inefficient.

I can think of two easy ways you can deal with this.

One way is just to ignore it! Even though the gaps may be several gigabytes in size, the space they'll take in your bitmap is quite small relative to both the amount of memory and the amount of virtual address space available to a typical 64-bit kernel.

If you're only concerned about physical memory usage and not virtual address space, you can use some page table trickery to turn your bitmap into a sparse bitmap by remapping all of the parts of the bitmap that will never indicate usable memory to the same read-only page.

That makes sense. My primary concern with the "ignore it" philosophy is that I don't know if there are any rules about how big these gaps can be. In a 64-bit memory space, what if the system decided one page frame was going to be randomly set at physical address 0xf000000000000000? All of the sudden there's an Exabyte-sized gap, which I obviously couldn't store in my bitmap. I expect this probably wouldn't happen, but I don't know why these gaps exist, so why not? Is there some limitation to how high this physical address could be, maybe the maximum memory supported by the motherboard?

[Octocontrabass]

Is there some limitation to how high this physical address could be, maybe the maximum memory supported by the motherboard?

Usually it's some combination of the memory controller and the firmware that configures it during boot. The x64 architecture itself allows no more than 52 physical address bits, so the upper bound on the bitmap is 128GiB. As far as I know, there aren't any x64 CPUs that support more than 48 physical address bits, which further limits the bitmap to 8GiB.

As far as I know, the gaps are related in some way to the maximum supported memory configuration, so you should see large gaps only when the installed memory is much smaller than the maximum supported.

[Tyyrus]

Usually it's some combination of the memory controller and the firmware that configures it during boot. The x64 architecture itself allows no more than 52 physical address bits, so the upper bound on the bitmap is 128GiB. As far as I know, there aren't any x64 CPUs that support more than 48 physical address bits, which further limits the bitmap to 8GiB.

As far as I know, the gaps are related in some way to the maximum supported memory configuration, so you should see large gaps only when the installed memory is much smaller than the maximum supported.

Okay, thanks for all of the help! In that case I'm pretty comfortable mapping the whole address space and ignoring the parts of the bitmap that map to the gaps. I'll make sure there's enough available memory to hold the bitmap, but until the day someone manages to install a 512Mb chip in a board that supports a Petabyte of RAM I don't expect the kernel will need to error out there lol. I really appreciate your time and explanations