Page 1 of 1

Grub2 incomplete memory maps

Posted: Sun Apr 01, 2012 8:26 am
by AlfaOmega08
I'm writing my os from scratch again, using grub2 as bootloader for elf64 binaries. After entering long mode i'm building a bitmap based frame allocator. So I parse the memory maps given from Grub2 to set some zones as used in the frame allocator. It works correctly on Bochs & VMWare, and it "seems" to work correctly even on real PCs. On my PC however (and probably on others too) I have a strange output. I have 8 Gb of RAM. Here are my memory maps:

Code: Select all

0x0000000000000000 - 0x000000000009FBFF - kind 1
0x000000000009FC00 - 0x000000000009FFFF - kind 2
0x00000000000E4000 - 0x00000000000FFFFF - kind 2
0x0000000000100000 - 0x00000000BF77FFFF - kind 1
0x00000000BF780000 - 0x00000000BF797FFF - kind 3
0x00000000BF798000 - 0x00000000BF7DBFFF - kind 4
0x00000000BF7DC000 - 0x00000000BFFFFFFF - kind 2
0x00000000FEE00000 - 0x00000000FEE00FFF - kind 2
0x00000000FFE00000 - 0x00000000FFFFFFFF - kind 2
0x0000000100000000 - 0x000000023FFFFFFF - kind 1
My problem is that there are three zones of memory, that are totally unreferenced:

Code: Select all

0xA0000 - 0xE3FFF
0xF0000000 - 0xFEDFFFFF
0xFEE01000 - 0xFFDFFFFF
Well, the first one, is probably the video memory and I know I should protect it (I actually set the whole first MB as reserved). But what about the other zones? Should I mark those as reserved, as usable or what? Maybe some PCI device are mapped there?

Thanks

Re: Grub2 incomplete memory maps

Posted: Sun Apr 01, 2012 8:31 am
by bluemoon
I think it's better to not allocate any memory address not on the map.

By the way, 0xF0000000 looks like frame buffer to me but you need to detect hardwares to make sure, if pci report such address you may override the memory map.

Re: Grub2 incomplete memory maps

Posted: Sun Apr 01, 2012 9:24 am
by AlfaOmega08
Well just noticed that on VMWare is quite the same, there are holes there too. Just wasn't paying enaugh attention... Thanks for your reply tought.

Re: Grub2 incomplete memory maps

Posted: Mon Apr 02, 2012 5:54 am
by brain
my understanding of the memory map is that if its not in the map at all, then there is 'nothing there'. The memory is neither mapped to devices/bios nor is it ram so reads/writes to it will be undefined behaviour. These are spaces you cannot allocate out to user programs or kernel, but you can for example map pci devices mmio into.

Is my understanding correct?

Re: Grub2 incomplete memory maps

Posted: Mon Apr 02, 2012 6:29 am
by bluemoon
I only have a draft pci driver but I do not see any way you can remap PCI BARs.

And to my understanding, if a PCI report it use "physical" address that do not exist on the BIOS map, it should override the map.

Re: Grub2 incomplete memory maps

Posted: Tue Apr 03, 2012 7:19 am
by brain
bluemoon wrote:I only have a draft pci driver but I do not see any way you can remap PCI BARs.

And to my understanding, if a PCI report it use "physical" address that do not exist on the BIOS map, it should override the map.
As i understood it, you can just write a new page-aligned address to the BAR and it will remap them, as i did not see anywhere in the spec i read that these registers were readonly?
I have not needed to do this though as the spec also says that the bios should set them to sane values, along with the irq mappings etc and then leave the devices in a disabled state, so in theory all that needs doing is to read those current values and enable the devices.

Re: Grub2 incomplete memory maps

Posted: Tue Apr 03, 2012 1:48 pm
by Brendan
Hi,
brain wrote:As i understood it, you can just write a new page-aligned address to the BAR and it will remap them, as i did not see anywhere in the spec i read that these registers were readonly?
First determine if the BAR is for physical memory or IO ports, and (for MMIO) if it's a 64-bit BAR or not and if the area is prefetchable.

The lowest address bits of the BAR are hard-wired to zero to indicate both the alignment restrictions and the size of the area. You should write 0xFFFFFFFF (or 0xFFFFFFFFFFFFFFFF for 64-bit) to the BAR and then read it back to determine the minimum alignment required. The size of the area is the same as the minimum alignment required. For example, if you can only set the highest 16 address bits of a 32-bit BAR, then the area has a 64 KiB minimum alignment and a 64 KiB size. Once you know the alignment and size, you can allocate a suitable area in the physical address space and then set the BAR properly.

Also note that you also have to setup any PCI bridges - there's no point telling a device to accept accesses within a certain area if the bridge/s above it aren't setup to forward accesses in that area towards the device. For each bridge (starting from the host controller), determine how much space you need for all of the bridge's children (for IO ports, for prefetchable MMIO, and for non-prefetchable MMIO); then allocate a range of IO ports and 2 different areas of the physical address space for the bridge; then split those larger areas into pieces for the bridge's children.

For example, if there's one bridge and 2 devices, like this;

Code: Select all

bridge 2
  |__device 3 (one BAR for a 64 KiB area of non-prefetchable, one BAR for a 256 MiB area of prefetchable)
  |__device 4 (one BAR for a 64 KiB area of non-prefetchable, one BAR for a 16 IO ports)
  |__device 5 (one BAR for a 512 MiB area of prefetchable, one BAR for a 8 IO ports)
Then you'd check the children to determine you need:
  • a 128 KiB area for non-prefetchable MMIO (64 KiB for device 3 and 64 KiB for device 4)
  • a 1 GiB area for prefetchable MMIO (256 MiB for device 3, 512 MiB for device 5, and 256 MiB unused due to rounding up)
  • 32 IO ports (16 IO ports for device 4, 8 IO ports for device 5, 8 IO ports unused due to rounding up)
Now imagine that bridge2 is underneath bridge1:

Code: Select all

bridge 1
  |__device 1 (one BAR for a 4 KiB area of non-prefetchable)
  |__device 2 (one BAR for a 128 KiB area of non-prefetchable, one BAR for a 16 IO ports)
  |__bridge 2 (128 KiB of non-prefetchable, 1 GiB of prefetchable and 32 IO ports)
       |__device 3 (one BAR for a 64 KiB area of non-prefetchable, one BAR for a 256 MiB area of prefetchable)
       |__device 4 (one BAR for a 64 KiB area of non-prefetchable, one BAR for a 16 IO ports)
       |__device 5 (one BAR for a 512 MiB area of prefetchable, one BAR for a 8 IO ports)
For bridge 1 you need:
  • a 512 KiB area for non-prefetchable MMIO (4 KiB for device 1, 128 KiB for device 2, 128 KiB for bridge 2 and 252 KiB unused due to rounding up)
  • a 1 GiB area for prefetchable MMIO (1 GiB for bridge 2)
  • 64 IO ports (16 IO ports for device 2, 32 IO ports for bridge 2 and 16 IO ports unused due to rounding up)
Now that you know what you need, you allocate the IO space and 2 pieces of the physical address space. Lets pretend that the 64 IO ports you allocated are IO ports 0xD000 to 0xD03F, the first area (for non-prefetchable MMIO) is from 0xC0000000 to 0xC007FFFF
and the second area (for prefetchable MMIO) is from 0x80000000 to 0xBFFFFFFF. You might end up with IO space looking like this:

Code: Select all

0xD000 to 0xD03F accepted by bridge 1 and forwarded to its secondary bus
    0xD000 to 0xD00F accepted by device 1
    0xD010 to 0xD01F unused
    0xD020 to 0xD03F accepted by bridge 2 and forwarded to its secondary bus
        0xD020 to 0xD02F accepted by device 4
        0xD030 to 0xD037 accepted by device 5
        0xD038 to 0xD03f unused
The physical address space might look like this (showing areas used by PCI only):

Code: Select all

0x80000000 to 0xBFFFFFFF accepted by bridge 1 and forwarded to its secondary bus
    0x80000000 to 0xBFFFFFFF accepted by bridge 2 and forwarded to its secondary bus
        0x80000000 to 0x8FFFFFFF accepted by device 3
        0x90000000 to 0x9FFFFFFF unused
        0xA0000000 to 0xBFFFFFFF accepted by device 4
0xC0000000 to 0xC007FFFF accepted by bridge 1 and forwarded to its secondary bus
    0xC0000000 to 0xC0000FFF accepted by device 1
    0xC0001000 to 0xC003FFFF unused
    0xC0040000 to 0xC005FFFF accepted by device 2
    0xC0060000 to 0xC007FFFF accepted by bridge 2 and forwarded to its secondary bus
        0xC0060000 to 0xC006FFFF accepted by device 3
        0xC0070000 to 0xC007FFFF accepted by device 4

Cheers,

Brendan