OSDev.org

Hiya everyone,

The UEFI specification says that pages of certain types shouldn't be accessed by an OS (after calling ExitBootServices), such as RT Code, RT Data, Reserved and ACPI NVS. Does this also mean that we can't move these pages to another location to make the physical space less cluttered/fragmented? For example, if the original physical memory layout looked like this...

* RT Data (16 pages)
* BS Data (4 pages)
* RT Code (12 pages)
* BS Code (1 page)
* RT Code (9 pages)
* Conventional (268 pages)
* RT Data (12 pages)

...is there a way to change the physical layout to something like this...

Group 1 (not to be accessed by OS)
* RT Code - 12 pages
* RT Code - 9 pages
* RT Data - 16 pages
* RT Data - 12 pages
Group 2 (freely accessible by OS)
* BS Code - 1 page
* BS Data - 4 pages
* Conventional - 268 pages

Kind regards,

papabearx wrote: ↑Fri Oct 25, 2024 3:43 pm Hiya everyone,

The UEFI specification says that pages of certain types shouldn't be accessed by an OS (after calling ExitBootServices), such as RT Code, RT Data, Reserved and ACPI NVS. Does this also mean that we can't move these pages to another location to make the physical space less cluttered/fragmented? For example, if the original physical memory layout looked like this...

it does

* RT Data (16 pages)
* BS Data (4 pages)
* RT Code (12 pages)
* BS Code (1 page)
* RT Code (9 pages)
* Conventional (268 pages)
* RT Data (12 pages)

...is there a way to change the physical layout to something like this...

Group 1 (not to be accessed by OS)
* RT Code - 12 pages
* RT Code - 9 pages
* RT Data - 16 pages
* RT Data - 12 pages
Group 2 (freely accessible by OS)
* BS Code - 1 page
* BS Data - 4 pages
* Conventional - 268 pages

Kind regards,

there isn't.

Ok, thanks Zaval!

There is no reason to charge the physical memory layout. You can map your physical memory pages to any linear page, and reserved physical memory should not be mapped at all.

If Direct Memory Access (DMA) could benefit from contiguous memory space, wouldn't that be a valid reason? I haven't studied disk controllers yet, so I could be completely wrong about this. But if we wanted to load - let's say - a 409,6 KB file into memory, and there are a hundred unused 4k pages scattered in RAM, wouldn't an OS have to issue a hundred read commands (one for each 4 KB page that needs to be filled) as opposed to a single read command if there was a contiguous block of memory available that could fit the entire 409,6 KB file?

Unless DMA accepts a linear memory address of a data buffer, it would seem to me that reorganizing physical memory to create larger contiguous blocks could help optimize data transfer. But again, haven't studied disk I/O yet, so forgive me if I'm saying things that sound silly to the game dev experts os dev experts out here.

Edit: fixed a little oopsie

papabearx wrote: ↑Sat Oct 26, 2024 5:13 pmIf Direct Memory Access (DMA) could benefit from contiguous memory space, wouldn't that be a valid reason?

Modern hardware supports scatter/gather DMA, so you usually don't need physically contiguous memory.

papabearx wrote: ↑Sat Oct 26, 2024 5:13 pmgame dev experts

Huh?

Octocontrabass wrote: ↑Sat Oct 26, 2024 6:34 pm Modern hardware supports scatter/gather DMA, so you usually don't need physically contiguous memory.

Ah, I see! I was not previously familiar with the Scatter Gather DMA mode, but after a brief search, I now understand why physically contiguous memory is not really needed anymore. Thank you for this information!

Octocontrabass wrote: ↑Sat Oct 26, 2024 6:34 pm

papabearx wrote: ↑Sat Oct 26, 2024 5:13 pmgame dev experts

Huh?

Oops, my apologies! I need to switch my coffee blend from 'Game Dev Espresso' to 'OS Dev Brew'!

Modern hardware, like disc controllers, uses in-memory lists of blocks. That's a consequence of OSes using random linear to physical mappings at the 4k level that typically are not continuous. There are also scatter-gather requirements, but even without those, modern paging means that hardware devices cannot rely on larger blocks than 4k of continuous memory. There can also alignment requirements, like dword alignment, since buses are wider than a byte.

rdos wrote: ↑Sat Oct 26, 2024 11:39 am and reserved physical memory should not be mapped at all.

Just to be thorough, does this apply only to pages marked as 'reserved' or does it also apply to pages marked as 'unusable', 'MMIO', 'MMIO port space', and 'unaccepted'? Initially, I was planning to identity map all the regions that shouldn't be accessed by an OS. I thought that if I wouldn't, it would break things because certain services might expect certain code to be present at specific memory addresses, such as EFI runtime or MMIO for example.

The UEFI manual says this about the different page types:

Reserved - do not use
RTCode - preserve
RTData - preserve
Unusable - do not use
ACPI NVS - preserve
MMIO - do not use
MMIO Port Space - do not use
PalCode - preserve
Unaccepted - treat as not-present

So 'do not use' in this case means don't map them at all? And 'preserve' means identity map them so that they can still be accessed by services that require them such as EFI runtime? I must admit, I'm a little confused though. How would an OS be able to access MMIO registers if it's not mapped into linear address space?

papabearx wrote: ↑Mon Oct 28, 2024 8:03 amJust to be thorough, does this apply only to pages marked as 'reserved' or does it also apply to pages marked as 'unusable', 'MMIO', 'MMIO port space', and 'unaccepted'?

You don't need to map anything you're not using. That includes ordinary usable memory. You're free to map them anyway, if you really want to.

Unlike ordinary usable memory, you shouldn't access any of those regions unless the firmware tells you to do so. For example, the ACPI tables might tell you one of those regions contains the local APIC MMIO.

papabearx wrote: ↑Mon Oct 28, 2024 8:03 amI thought that if I wouldn't, it would break things because certain services might expect certain code to be present at specific memory addresses, such as EFI runtime or MMIO for example.

You get to choose which addresses it expects, though. You need to call SetVirtualAddressMap() before you can use any other EFI runtime services anyway.

papabearx wrote: ↑Mon Oct 28, 2024 8:03 amSo 'do not use' in this case means don't map them at all? And 'preserve' means identity map them so that they can still be accessed by services that require them such as EFI runtime?

None of those things need to be mapped when you're not using them. When you want to call EFI runtime services, the EFI_MEMORY_RUNTIME attribute tells you which things you need to map before you make the call, and you can unmap them after the runtime service returns.

papabearx wrote: ↑Mon Oct 28, 2024 8:03 amI must admit, I'm a little confused though. How would an OS be able to access MMIO registers if it's not mapped into linear address space?

Easy: it maps them when it needs to access them. You'll need to do this anyway, since most MMIO will be located at addresses not listed in the EFI memory map.

I see. It looks like I've got some more reading to do. I find this whole memory management subject a bit challenging to grasp, and the more I ask, the more questions emerge. But for now I think I'll leave it at that, there's enough information for me to explore further on my own. Thanks for the help everyone.

papabearx wrote: ↑Mon Oct 28, 2024 4:56 pm I see. It looks like I've got some more reading to do. I find this whole memory management subject a bit challenging to grasp, and the more I ask, the more questions emerge. But for now I think I'll leave it at that, there's enough information for me to explore further on my own. Thanks for the help everyone.

It might be easier to think of and manage only ordinary memory the kernel will be using for mapping normal memory mappings. This is what you dole out to satisfy memory requests.

Other memory, such as MMIO, you're directed to by the hardware registers (such as PCI BAR registers).

ACPI and UEFI memory is managed by and for the use of their respective runtimes, so if you don't use those in your kernel, then you have no interest in them.

Basically, my recommendation is to use ordinary memory, then worry about ACPI and UEFI memory as and when you add support for those.

You can do an entirely functional basic kernel without touching either.

thewrongchristian wrote: ↑Tue Oct 29, 2024 12:09 pm

papabearx wrote: ↑Mon Oct 28, 2024 4:56 pm I see. It looks like I've got some more reading to do. I find this whole memory management subject a bit challenging to grasp, and the more I ask, the more questions emerge. But for now I think I'll leave it at that, there's enough information for me to explore further on my own. Thanks for the help everyone.

It might be easier to think of and manage only ordinary memory the kernel will be using for mapping normal memory mappings. This is what you dole out to satisfy memory requests.

Other memory, such as MMIO, you're directed to by the hardware registers (such as PCI BAR registers).

ACPI and UEFI memory is managed by and for the use of their respective runtimes, so if you don't use those in your kernel, then you have no interest in them.

Basically, my recommendation is to use ordinary memory, then worry about ACPI and UEFI memory as and when you add support for those.

You can do an entirely functional basic kernel without touching either.

Thank you for your advice, I appreciate it. For the time being I shall focus on working with ordinary memory only and treat all pages that UEFI says shouldn't be accessed by an OS as if they weren't there. Not mapping them at all, at least for now.

It makes sense to focus on ordinary memory first before delving into ACPI and UEFI memory support. Especially since I'm only trying to get a computer to boot, present a blue screen to the user and have it shut down by either a simple mouse click or by pressing a random key on the keyboard. This may sound like a very simple thing to do, but instead of doing it the fast way, I would like to do it the right way (for example, the blue screen not just being a blue screen but actually being a process running in user space, if that makes sense).

I'll follow your advice and shall focus on ACPI and UEFI memory when it's needed, which will probably somewhere around the time when the shutdown functionality needs to be added. Or maybe even sooner, who knows, time will tell.

Thanks thewrongchristian.