Problems completely understanding memory management.

[AndrewBuckley]

Well yes… but it's kinda hard to imagine a stack that expands downwards… physically. Is it simply a matter of reversing the gravitational direction? :p

instead of picturing dishes being placed on top of one another, try picturing rooms 1-100 in a hallway with a stick in front of door 78. you need 3 rooms to put stuff in for a day or so so you walk down the hall to the stick, you put your stuff in rooms 78,77,76 and you put the stick in front of room 75. when you come back the stick should be in front of room 75, and you know that above that is the 3 rooms you put your junk. your take your stuff and you put the stick back in front of room 78. by using this stick you need not remember a room number because next month when you need another 3 rooms it might not be the same ones. the main requirement for this setup is that taking things out of the rooms happen in the exact opposite order so the stick positions are correct. if this is not done, you have a corruption of the stack.

[bluemoon]

1. Higher half. Is it simply a matter of:
- Enable paging
- Map first x MB where kernel resides in memory (at y)
- Map x amount of virtual memory somewhere high up (3GB?) to y.
- Far jump to virtual y and execute the kernel?

No. You should have prepared the proper paging structures before enable paging.
At the time you enable paging, the CPU execute the address following the enabling instruction, therefore you need the identify mapping so that the CPU can continue execute the next instruction; the next instruction is usually the jump to higher address, then, the identity map can be removed.

2. What are the differences in paging in higher half and normal?

Generally, there is no preferred address in CPU in protected mode, it's just a matter of design choice to partition the address space.

If I load the kernel at 3GB virtual, where in virtual memory should I allow for normal allocations? Anywhere (excluding things like BIOS of course) below the 3GB mark?

This depends on how you partition the address space, it's totally up to you, and even can be quite dynamic.
Note that the BIOS sit on physical address, you can just do not map it in logical address.

3. PAE Paging. It's similar to paging, just that there's another structure (PDPT) that is an array of page directories?

Yes, on top of that are the CPU capability detection and setup logic.


On 64 bit and PSE, check the wiki example, note that there are relatively few tutorial and resource on 64-bit compared to 32-bit, so I would suggest you to master the 32-bit paging and have a bare-bone running first.

[eino]

No, reference manuals and the wiki are not fantastic resources for explaining concepts. The encyclopaedia format isn't as good as a pointed answer (StackOverflow or this forum) or guides written in the tutorial style.

I totally agree.

Besides the heap is quite poorly explained in most articles. IMO.

[Combuster]

Besides the heap is quite poorly explained in most articles. IMO.

Yet, can you be a proper C programmer without knowing what allocations on the heap and allocations on the stack (and their essential differences) are?

[zhiayang]

On 64 bit and PSE, check the wiki example, note that there are relatively few tutorial and resource on 64-bit compared to 32-bit, so I would suggest you to master the 32-bit paging and have a bare-bone running first.

You have a point there. But just to clarify, when (if) I get to long mode,

What is the order of events? I would like a higher half 64-bit kernel, so…

- Set up 32-bit environment
- Set PAE bits, set up various things
- >> !! <<
- Higher Half 64-bit kernel.


The problem is clear: Circular dependency!

1. If I want higher half, I need to enable 64-bit PAE Paging to map the kernel (module) to higher memory.
2. To enable 64-bit PAE Paging, I need to be in 64-bit mode!

So: Is it a matter of having a 3rd stage, as an intermediate 64-bit lower half bootstrap to the higher half?

Unless… You don't need to be in 64-bit mode to use PAE? I'm pretty sure you do… right?

[bluemoon]

My boot sequence is:

Code: Select all

1. Boot Loader:
  a. Basic machine initialization (stack, flags, registers, A20)
  b. Detect CPU (assume 64-bit here)
  c. Memory Map
  d. Load 64-bit kernel.bin and initrd into 1M
  e. Create PML4 structure with 0-2M identity map
  f. CR3 = Phy address of PML4
  g. Enable PAE (CR4.bit[5] = 1)
  h. Enable Long mode (MSR[C0000080].bit[8]=1)
  i. LGDT with CODE64 entry
  j. CR0 |= (1<<31) | (1<<0) - CPU is in 32bit compatibility mode after this
  k. jmp CODE64:start64 - CPU in 64bit long mode after this

Boot Loader: start64
  a. Parse entry point of kernel
  b. jmp to kernel

Kernel: bootstrap
  a. Clear BSS
  b. Create PML4 structure with Addr(0-2M) => Phy(0-2M), Addr(High)=Phy(0-2M)
  c. Reload CR3
  d. Jmp to high address
  e. Reload GDTR (since remapped kernel to high address)
  f. Reload CS/SS by IRETQ
  g. Remove 0-2M entry (and invalidate)
  h. LTR
  i. Setup syscall MSR
  h. xor rbp, rbp and jmp kmain

Note that I enter long mode directly, which is a trick and may not officially supported.
Some procedure may be redundant but it works for me.

[Combuster]

Unless… You don't need to be in 64-bit mode to use PAE? I'm pretty sure you do… right?

You don't. PAE is enabled as long as PG, PE, and PAE are set together, and doesn't require the LME bit. PAE was after all supported by several 686s.

Also, neither the LME bit, nor the LMA bit implies 64-bit mode - after all, 32-bit and 16-bit code can run under a 64-bit kernel.

But actually learning to RTFM will save you from repeatedly making these kind of errors. Heck, even those very manuals have sample code on setting up long mode.

Note that I enter long mode directly, which is a trick and may not officially supported.

For pendancy, you can just set PE and PG in two consecutive writes, without needing any other changes to code. (That is, if I'm right - I haven't bothered with long mode since my more interesting testing hardware is rather old in comparison)

[JAAman]

as was said, you can use PAE in 32-bit PMode, on any CPU which supports PAE -- it was originally introduced with the P6 architecture (originally marketed as the "Pentium Pro" around 1995, and then later re-released as the Pentium II), which introduced a 36bit address bus (supporting 64GB of memory in PMode) and now currently supports access to the full address bus within PMode

however, you do not need to map the kernel to the top-of-memory before entering LMode, just load the kernel code after entering LMode (alternatively, if you don't have drivers for the disk systems in use, load it before and then re-map it into the higher half after entering LMode -- just like you would in PMode)