64-Bit Higher Half Kernel

[rpio]

Finally I got to full switch to 64-Bit mode, fixed linking issues and other stuff. Now I am stuck at the higher half kernel because most higher half kernel tutorials are for 32-Bit CPUs which I am not interested in. For 64-Bit CPUs I only found several forum posts and a wikipedia page.

My understanding of higher half kernel - it is a kernel that is remapped to a virtual memory region that leaves just enough space to fit the kernel before the end of RAM(I want it to be 1 GB(left till the end of RAM), in 32-Bit mode there is really a 32-Bit address space and all of it could be used so it is easy to choose and mostly people say to put it at 3GB.

But problem arises in 64-Bit mode where only 48-Bits could be used for addresses(I won't bother with PML5 which is only in the Ice Lake+ anyway and in any case it is still not 64-Bits), so the kernel should be located at (2^48 - 1) - 1GB, but on all forum posts and wikipedia article they talk about loading to FFFF8000'00000000 which is not an address on x86-64 with PML4 as it is larger then 48-Bit and does not exist even in the virtual address space.

So where should I load the kernel in 64-Bit mode and why do they talk about an unexistant address that can't be used?

Wikipedia page I am talking about - https://en.wikipedia.org/wiki/X86-64#Vi ... ce_details

[kzinti]

FFFF800000000000h is a perfectly valid virtual address.

Although you can only specify 48 bits, the msb one - bit 47 - is repeated in positions 63..48.

So the ranges of VMA you can use with a PML 4 setup are:

0000000000000000h - 00007FFF FFFFFFFFh (128 TB)
FFFF800000000000h - FFFFFFFF FFFFFFFFh (128 TB)

You typically want to put your kernel in the last 2 GB of memory an use the "-mcmodel=kernel" switch. This puts your kernel in the FFFFFFFF80000000h - FFFFFFFFFFFFFFFFh range.

[zhiayang]

If you look at the diagram in the wikipedia article you linked --- in fact the diagram is in the exact section you linked --- you'll notice that the virtual address space is split into two halves, due to the canonical address requirement. The 40-bit, 48-bit, and 57-bit address limitation applies in a different way than you think. Again, due to the canonical addresses, you effectively get a 39-bit lower half and a 39-bit higher half, or a 47-bit lower half and a 47-bit higher half, etc. etc.

Again, most of your questions come from your (lack of?) understanding of the way canonical addresses work on 64-bit.

[rpio]

FFFF800000000000h is a perfectly valid virtual address.

Although you can only specify 48 bits, the msb one - bit 47 - is repeated in positions 63..48.

So the ranges of VMA you can use with a PML 4 setup are:

0000000000000000h - 00007FFF FFFFFFFFh (256 TB)
FFFF800000000000h - FFFFFFFF FFFFFFFFh (256 TB)

You typically want to put your kernel in the last 2 GB of memory an use the "-mcmodel=kernel" switch. This puts your kernel in the FFFFFFFF80000000h - FFFFFFFFFFFFFFFFh range.

I can't really understand how the FFFFFFFF80000000h - FFFFFFFFFFFFFFFF can be addresses if the 48-64 should be copies of 47 that adds only two addresses to the address space(with 48(48-64 as they are all the same) off and with it on)? How can one extra bit that should be the same as 47 add access to full 64-Bit address space

And also when I implement VMM should it always set the bits 48-64 if bit 47 is set?

[rpio]

If you look at the diagram in the wikipedia article you linked --- in fact the diagram is in the exact section you linked --- you'll notice that the virtual address space is split into two halves, due to the canonical address requirement. The 40-bit, 48-bit, and 57-bit address limitation applies in a different way than you think. Again, due to the canonical addresses, you effectively get a 39-bit lower half and a 39-bit higher half, or a 47-bit lower half and a 47-bit higher half, etc. etc.

Again, most of your questions come from your (lack of?) understanding of the way canonical addresses work on 64-bit.

That is hard for me to understand:
1. What does it mean that it is divided in two halves, what are these halves?
2. How can halves be 39 and 39 or 47 and 47 if there are only 64 bits available?
3. What is are the canonical addresses?
4. Where am I wrong with my thoughts about 48/57 bit limitation?

[kzinti]

I can't really understand how the FFFFFFFF80000000h - FFFFFFFFFFFFFFFF can be addresses if the 48-64 should be copies of 47 that adds only two addresses to the address space(with 48(48-64 as they are all the same) off and with it on)? How can one extra bit that should be the same as 47 add access to full 64-Bit address space

That's the thing... You cannot access the full 64 bit address space. You can only access 48 bits of it, which means 256 TB (2^48).

Because bit 47 is copied, you effectively can access the first 128 TB and the last 128 TB of the address space. This is what we call the two halves because there is a big gap in between that cannot be used.

One way to think about it is to ignore the top 16 bits. Just pretend they don't exist (because really they don't). Then you are left with 48 bits.

And also when I implement VMM should it always set the bits 48-64 if bit 47 is set?

Yes, you have to provide valid addresses and anything between the 2 halves is not a valid address.

What is are the canonical addresses?

The act of copying bit 47 into positions 48..64 results in what is called a canonical address. Confusing name, I know. But this is the language picked up by Intel to say that the address is valid.

For example, address 0000800000000000h is invalid / non-canonical because bit 47 has not been copied to 48..64. But if you replicate bit 47, you get a canonical address - FFFF80000000000h

[bzt]

My understanding of higher half kernel - it is a kernel that is remapped to a virtual memory region that leaves just enough space to fit the kernel before the end of RAM

That's wrong. Higher half kernels are mapped at the top of the virtual address space, this has nothing to do with how much RAM you have. The fact that C0000000 virtual address just happens to be at 3G and 32 bit machines can only have 4G (without PAE) is just a coincidence.

1. What does it mean that it is divided in two halves, what are these halves?

Well, it's just a matter of perspective. If you think about address space with signed addresses, where the higher half are actually negative addresses, then you have one region only (-256Tb to +256Tb):

Code: Select all

FFFF8000 00000000 - 00007FFF FFFFFFFF

But if you think of it as unsigned addresses, then you got three regions, and the one in the middle is not usable (0 to +256Tb and -256Tb to -1):

Code: Select all

00000000 00000000 - 00007FFF FFFFFFFF (lower half)
00008000 00000000 - FFFF7FFF FFFFFFFF (not valid)
FFFF8000 00000000 - FFFFFFFF FFFFFFFF (higher half)

2. How can halves be 39 and 39 or 47 and 47 if there are only 64 bits available?

Because not all the 64 bits are used, only the least significant 39 or 47 bits. The remaining most significant bits must be the same; addresses where those top bits are not all 0s or 1s are in the (not valid) region. With the signed addresses example, those are either less than FFFF8000 00000000 (-256Tb) or bigger than 00007FFF FFFFFFFF (+256Tb).

3. What is are the canonical addresses?

An address with all the most significant bits all 0s or 1s. In other words, addresses that are either in the (lower half) or in the (higher half) region, but not in the (not valid) region.

4. Where am I wrong with my thoughts about 48/57 bit limitation?

You are mixing address space with memory bus size. An address space can be 48 bits wide (with PML4, not using PML5), and it might be expanded further in the future. This means your address space might grow, and the (not valid) region will shrink. If all 64 bits were used, there would be no (not valid) region at all. On the other hand, 56 bit memory bus size won't change, that's an architectural limit, which means you cannot use more physical RAM than that.

Cheers,
bzt

[rpio]

My understanding of higher half kernel - it is a kernel that is remapped to a virtual memory region that leaves just enough space to fit the kernel before the end of RAM

That's wrong. Higher half kernels are mapped at the top of the virtual address space, this has nothing to do with how much RAM you have. The fact that C0000000 virtual address just happens to be at 3G and 32 bit machines can only have 4G (without PAE) is just a coincidence.

1. What does it mean that it is divided in two halves, what are these halves?

Well, it's just a matter of perspective. If you think about address space with signed addresses, where the higher half are actually negative addresses, then you have one region only (-256Tb to +256Tb):

Code: Select all

FFFF8000 00000000 - 00007FFF FFFFFFFF

But if you think of it as unsigned addresses, then you got three regions, and the one in the middle is not usable (0 to +256Tb and -256Tb to -1):

Code: Select all

00000000 00000000 - 00007FFF FFFFFFFF (lower half)
00008000 00000000 - FFFF7FFF FFFFFFFF (not valid)
FFFF8000 00000000 - FFFFFFFF FFFFFFFF (higher half)

2. How can halves be 39 and 39 or 47 and 47 if there are only 64 bits available?

Because not all the 64 bits are used, only the least significant 39 or 47 bits. The remaining most significant bits must be the same; addresses where those top bits are not all 0s or 1s are in the (not valid) region. With the signed addresses example, those are either less than FFFF8000 00000000 (-256Tb) or bigger than 00007FFF FFFFFFFF (+256Tb).

3. What is are the canonical addresses?

An address with all the most significant bits all 0s or 1s. In other words, addresses that are either in the (lower half) or in the (higher half) region, but not in the (not valid) region.

4. Where am I wrong with my thoughts about 48/57 bit limitation?

You are mixing address space with memory bus size. An address space can be 48 bits wide (with PML4, not using PML5), and it might be expanded further in the future. This means your address space might grow, and the (not valid) region will shrink. If all 64 bits were used, there would be no (not valid) region at all. On the other hand, 56 bit memory bus size won't change, that's an architectural limit, which means you cannot use more physical RAM than that.

Cheers,
bzt

Thanks for your explanation, but I have several questions to better understand the higher half memory topic

1. Can the memory of the so called second half after the FFFF8000 00000000 be accessed freely(just map it to the physical RAM) or do I need to have some address calculation/page table hacks to access it?
2. How to I know which page(what entries to use in tables) I should map to be able to access 1 GB starting from this point(start of second half)?
3. Does the 2 half division mean that the applications would only be able to access the first half ?
4. Does the 2 half division mean that the higher half kernel will grow upwards, which I think is bad because on other platforms it would grow down?

[bzt]

1. Can the memory of the so called second half after the FFFF8000 00000000 be accessed freely(just map it to the physical RAM) or do I need to have some address calculation/page table hacks to access it?

Your question makes no sense from technical point of view. You cannot access anything freely, all memory access goes through the MMU where the virtual address is translated into physical address. And since paging is mandatory in long mode, then yes, you must set up paging tables, but those ain't hacks just normal operation.

2. How to I know which page(what entries to use in tables) I should map to be able to access 1 GB starting from this point(start of second half)?

You implement a PMM. You probably want to take a look at the big picture.

3. Does the 2 half division mean that the applications would only be able to access the first half ?

No. Applications can access any address that's mapped in the page table. It is usual to map lower half for user access, and higher half for kernel access only, but page security is completely independent to the addresses.

4. Does the 2 half division mean that the higher half kernel will grow upwards, which I think is bad because on other platforms it would grow down?

It does not grow anywhere. You have a virtual address region you must map, it's completely up to you how you set up the paging tables.

What I've said about the address size growing, that applies to the address size, meaning using more bits expands both the lower half and the higher half at the same time. The wiki page you linked explains this in detail.
With 48 and 56 bits:

Note that both the yellow and green regions became bigger (and the gray smaller) when more bits were used.

Cheers,
bzt

[rpio]

1. Can the memory of the so called second half after the FFFF8000 00000000 be accessed freely(just map it to the physical RAM) or do I need to have some address calculation/page table hacks to access it?

Your question makes no sense from technical point of view. You cannot access anything freely, all memory access goes through the MMU where the virtual address is translated into physical address. And since paging is mandatory in long mode, then yes, you must set up paging tables, but those ain't hacks just normal operation.

2. How to I know which page(what entries to use in tables) I should map to be able to access 1 GB starting from this point(start of second half)?

You implement a PMM.

3. Does the 2 half division mean that the applications would only be able to access the first half ?

No. Applications can access any address that's mapped in the page table. It is usual to map lower half for user access, and higher half for kernel access only, but page security is completely independent to the addresses.

4. Does the 2 half division mean that the higher half kernel will grow upwards, which I think is bad because on other platforms it would grow down?

It does not grow anywhere. You have a virtual address region you must map, it's completely up to you how you set up the paging tables.

What I've said about the address size, that applies to the address size, meaning using more bits expands both the lower half and the higher half at the same time. The wiki page you linked explains this in detail.
With 48 and 56 bits:

Note that both the yellow and green regions became bigger (and the gray smaller) when more bits used.

Cheers,
bzt

1. You have probably misunderstood me because I didn't call paging hacks, I just meant that maybe I should do some bit changing or something like that to access the address of the higher half
2. The PMM gives out the areas of physical memory that I could map the virtual memory to, but I want to know how do I calculate the entry in each table(PML4, PDPT, PDT...) to correspond to the virtual FFFF8000 00000000 address
3. Here I expressed myself in a wrong way - I meant that how should I handle applications accessing the area, should I somehow make the memory manager not touch this area for the heap, stack...?
4. Actually the kernel grows, as it needs memory for stuff like the PMM bitmap(in my case), maybe drivers... so maybe I should remap the kernel to right the last 2GB of VRAM so that I could put the things I described before under it?

Also I wanted to ask about some implementation details

In the tutorials about the higher half kernel I looked at(which were for i686) - they linked the kernel as if it is at the higher half address(0xC000...), but the LMA(load memory addresses) was set to 1MB so the kernel will be loaded there. And when calling/jumping to labels in assembly the assembler would substitute them as if they were to the addresses in the higher half when the kernel is still at 1MB so the 0xC000... was subtracted from every label for calls/jumps/access to the correct addresses, but there they were in 32-Bit mode and they stayed in it just with paging enabled so 32-Bit addresses were subtracted, here it is harder. First I am first in 32-Bit mode where probably the addresses should also be 32-Bit but the labels are linked as 64-Bit addresses as if they are in the higher half so is it okay if I take 64-Bit numbers away from labels in the 32-Bit code so label addresses are as if the kernel starts at 1Mb and not at 256Tb

[bzt]

1. You have probably misunderstood me because I didn't call paging hacks, I just meant that maybe I should do some bit changing or something like that to access the address of the higher half

In that case no special bits.

2. The PMM gives out the areas of physical memory that I could map the virtual memory to, but I want to know how do I calculate the entry in each table(PML4, PDPT, PDT...) to correspond to the virtual FFFF8000 00000000 address

The virtual address is splited into sign+9+9+9+9+offs bits, which are then used to index the paging tables. Read the Intel or AMD documentations.

3. Here I expressed myself in a wrong way - I meant that how should I handle applications accessing the area, should I somehow make the memory manager not touch this area for the heap, stack...?

That's one thing, but you should also set up the security bits in the paging tables to mark pages as supervisor or user page.

4. Actually the kernel grows, as it needs memory for stuff like the PMM bitmap(in my case), maybe drivers... so maybe I should remap the kernel to right the last 2GB of VRAM so that I could put the things I described before under it?

That's totally up to you. The MMU does not know about the kernel heap, you are in charge where you put that and to map pages in that region.

Also I wanted to ask about some implementation details

In the tutorials about the higher half kernel I looked at(which were for i686) - they linked the kernel as if it is at the higher half address(0xC000...), but the LMA(load memory addresses) was set to 1MB so the kernel will be loaded there. And when calling/jumping to labels in assembly the assembler would substitute them as if they were to the addresses in the higher half when the kernel is still at 1MB so the 0xC000... was subtracted from every label for calls/jumps/access to the correct addresses

That's not the best way to do it. I'd recommend to link your kernel at it's VMA address, and do not care what the LMA actually is. In long mode you'll use paging, so your code only needs to know about VMA addresses (with the exception of the init code, which maps your kernel in the higher half and jumps to the VMA start address).

Maybe take a look at BOOTBOOT, that's a boot loader that loads higher-half kernels in long mode. It sets up the initial paging tables, so your kernel does not need special trampoline code, it can be cleanly compiled at VMA address. You define some symbols to tell the loader where to map things (the framebuffer for example), but your kernel will be simply mapped at its linking address (anything from FFFF8000 00000000 to FFFFFFFF FFE00000 could be used). There's also a sample kernel you can use as a skeleton for your own kernel, with linker script examples.

Cheers,
bzt

[sj95126]

4. Actually the kernel grows, as it needs memory for stuff like the PMM bitmap(in my case), maybe drivers... so maybe I should remap the kernel to right the last 2GB of VRAM so that I could put the things I described before under it?

Just a suggestion on my part, but I wouldn't try to cram things into the last part of anything. The 64-bit linear space is huge. Feel free to spread things around.

The way I've implemented my OS, the kernel (text, data, etc.) is at 0xffff800000000000 and the kernel heap starts at 0xffff900000000000. Stack is way up near the top at something like 0xfffff70000000000. One of the advantages of such a distribution is that when I see an address (in, say an exception) I can tell right away which region it involves. ffff8 is kernel data, ffff9 is kernel heap. So I have a better idea where to start looking. And I try to avoid anything starting with 0xffffffff so I don't have to turn my head sideways to start counting f's.