x86-64: Mapping memory without holes

[IanSeyler]

A little background... I have a mono-tasking, SMP-enabled, OS written in x86-64 Assembly. I would like to have a contiguous area of memory available at address 0xFFFF800000000000 for the running application. Memory is currently mapped with 2MiB pages.

Code: Select all

Example memory map for a 256MB QEMU instance:

Base Address       | Length             | Type
0x0000000000000000 | 0x000000000009FC00 | Free Memory (1)
0x000000000009FC00 | 0x0000000000000400 | Reserved Memory (2)
0x00000000000F0000 | 0x0000000000010000 | Reserved Memory (2)
0x0000000000100000 | 0x000000000FEDF000 | Free Memory (1)
0x000000000FFDF000 | 0x0000000000021000 | Reserved Memory (2)
0x00000000FFFC0000 | 0x0000000000040000 | Reserved Memory (2)

The kernel would be in the first 2MB of RAM starting at 0x0. The GDT, IDT, PML4, etc would also be there.

All other free memory would be mapped starting at address 0xFFFF800000000000. This is where I want the application to run in.

Ideally what I am expecting on a system with 256MB of RAM is a memory map of 2MB at 0x0 and 254MB (or close to it) at 0xFFFF800000000000.

A few questions:
1) Is this possible?
2) Do I still need to map physical memory to things like the APIC/IOAPIC at 0xFEE00000/0xFEC00000?

Thanks,
-Ian

[iansjack]

Paging allows you to map memory in any way you wish to, so your layout is certainly possible. And you need to map all memory that you wish to access, including memory-mapped i/o.

[Brendan]

Hi,

A little background... I have a mono-tasking, SMP-enabled, OS written in x86-64 Assembly. I would like to have a contiguous area of memory available at address 0xFFFF800000000000 for the running application

For 64-bit code, most instructions don't handle 64-bit immediate operands and only handle "sign/zero extended 32-bit immediate operands". The result of this is worse performance when addresses of data that were known at compile time don't fit in 32-bits. For example, you can't do "lea rax,[rbx*8 + 0xFFFF800000000000]" and would have to break it into 2 instructions (like "mov rcx,0xFFFF800000000000" then "lea rax,[rbx*8 + rcx]").

Basically; for performance reasons you want the addresses of everything in a program's ".rodata", ".data" and ".bss" sections to end up either in the range from 0x0000000000000000 to 0x000000007FFFFFFF or the range from 0xFFFFFFFF80000000 to 0xFFFFFFFFFFFFFFFF. The remaining space can be used for other things (stacks, dynamically allocated data, memory mapped files, etc) without any problem because the addresses aren't encoded into instructions at compile time.

If you have a contiguous area of memory from 0xFFFF800000000000 to "0xFFFF800000000000 + total_RAM" then its virtually impossible to avoid the performance problem (unless there's 128 TiB or more RAM).

If you have a contiguous area of memory from "0xFFFFFFFFFFFFFFFF + 1 - total_RAM" to 0xFFFFFFFFFFFFFFFF; then you can avoid the performance problem by loading the process at 0xFFFFFFFF80000000, but only if there's 2 GiB or more RAM, and this will leave you with the remaining space (used by heap, etc) that's fragmented (some at lower addresses if there's more than 2 GiB of RAM; and some at higher addresses). Alternatively; you can avoid the performance problem, avoid the "only if there's 2 GiB of RAM" problem, and avoid the "remaining space is split" problem; by doing "executable start address = "0xFFFFFFFFFFFFFFFF + 1 - sum of section sizes" in the linker script. Even if the linker doesn't support something that bizarre you could do it manually (subtract 2 MiB from the executable's start address when linking fails).

Which of these alternatives are you planning to use (slow, broken or bizarre)?

Now let's talk about file system performance. RAM is fast and storage devices (including SSD and network) are slow in comparison; so you want to use all "unused" RAM for caching file data (possibly including buffered writes). For example, if a computer has 16 GiB of RAM and the process and kernel only use 4 GiB, then you can cache 12 GiB of "most likely to be needed again" file data and avoid massive file IO performance problems. How are you planning to implement this (in a way that ensures the process can't/doesn't trash everything in your file data cache)?

Let's also think about memory mapped files. If a file that the process wants (or the process' executable file itself) is in the file data cache, it'd be nice if we could just map those pages into the process (with a "copy on write" scheme, to ensure the process can modify it without trashing the original) instead of duplicating the data and wasting lots of RAM for no sane reason. How are you going to implement this?

Finally, if a process only uses a fraction of space it reserved in its ".bss" section; it'd be nice to avoid wasting RAM for something that isn't being used, and use that RAM for something more important (like caching file data), and then put it back where it came from if it ever is used so that nobody ever notices that it was gone (apart from getting faster file IO).


Cheers,

Brendan