Higher Half, Paging, and Segmentation

[gsingh2011]

So I have a working bare OS thanks to this wiki and Bran's kernel development tutorial. Now I've been doing a lot of reading on memory management such as segmentation and paging, and I've taken a look at the higher half tutorials, and after reading them, I have a bunch of questions.

1. What's the point of putting the kernel in the upper half of virtual memory?
2. Is the limit of memory available with paging the amount of hard drive space available?
3. I've been working with code segments and data segments while setting up my GDT. Do all of those things disappear with paging? How do applications use those segments?
4. In the Higher Half with GDT tutorial, a trick GDT is first loaded, then paging is enabled, then the real GDT is loaded. When I wasn't using the higher half model I could load the GDT without all of this. I read through the wiki but I don't understand why all this is necessary. The tutorial says,
"Note that you must enable paging before loading the new GDT as the CPU, once loaded the GDT with 0x0 as base address, will be still using virtual addresses and if paging isn't enabled... triple fault!"
So paging is still enabled so isn't it not entirely segmentation? And it still doesn't explain the trick GDT.

Thanks.

[bluemoon]

1. What's the point of putting the kernel in the upper half of virtual memory?

I believe it's noted on wiki.
Some advantage is that you have a consistent address space for application, which would always starts from zero; without later worry if the kernel space getting bigger and bigger. Imaging if you do kernel at zero based instead, and application at 1G+, you are limiting kernel & stuff on 1G range without breaking into ugly chunks.
And there are only pros & cons as well, depend on your actual design.

2. Is the limit of memory available with paging the amount of hard drive space available?

The limit is on your imagination :p you could even swap memory to network.
In reality, you may also compress the swap file.

3. I've been working with code segments and data segments while setting up my GDT. Do all of those things disappear with paging? How do applications use those segments?

segments/selector are still there. On flat memory model it usually setup to be all addressable space.
you use segment/selector as usual (by moves, pop), the CPU provide additional validation in protected mode.

4. In the Higher Half with GDT tutorial, a trick GDT is first loaded, then paging is enabled, then the real GDT is loaded. When I wasn't using the higher half model I could load the GDT without all of this. I read through the wiki but I don't understand why all this is necessary.

IIRC the tutorial loads kernel on 1M, then map it on 3G.
So, the first GDT is loaded on 1M address; upon setup paging and mapped the kernel to 3G, and before you un-map 1M,
you need to tell the CPU the new location of GDT; otherwise the CPU cannot access it with the old address (1M) anymore.

[gsingh2011]

Thanks, I'm still a little confused on how segmentation and paging work at the same time, but I'll do some more reading before coming back to it.

In the Higher Half tutorial with paging, I don't see how the first four MB are mapped to 3GB+1MB. I see the creation of the page directory but I don't see the creation of any page tables. And the actual address of the page is stored in the page table. So how does that mapping work?

[Combuster]

It uses page size extensions (PSE) to create 4M pages. You should see a write to CR4 to enable it and some special bits set in the page directory that control size.

[berkeleyrc]

Thanks, I'm still a little confused on how segmentation and paging work at the same time, but I'll do some more reading before coming back to it.

It's really not too trippy. I believe it works like this: the segment selector is used to translate your original address, and then this address goes through the paging system, where sets of bits are used to locate page directory entries, page table entries and your offset. This gives you your final physical address.

It seems that most people today prefer to keep the segmentation end as simple as possible using offsets of 0 and maximum limits. This way the only thing that you need to think about when resolving virtual addresses into physical addresses is the paging. Of course you still need to keep your segment descriptors around for everything to work, and you still need to worry about privileges in the segment descriptors/selectors, etc.

Hope that helps.

[Owen]

x86 has three address types:

• Physical addresses. These are exactly what they say on the tin.
• Linear addresses. These are the addresses seen by the paging system
• Virtual addresses. These are the addresses seen by the segmentation system

Virtual addresses are transformed to logical ones using segmentation. The simple equation is LA = Segment.Base + VA
Linear addresses are transformed to physical ones using paging. There is no simple equation for this; it depends upon paging mode, page size, etc. It is a case of looking up the linear address in the page tables and generating an output address. For traditional protected mode 4kB paging, it is, in pseudo-C, PA = CR3->PDE[LA[31..22]]->PTE[LA[21..12]] + LA[11..0]
Physical addresses are what goes out to the processor's bus & cache system, with no further transformation.

If you set your segment base to 0 and limit to 0xFFFFFFFF, you effectively get LA = VA. If in long mode, then for CS, DS, ES and SS no base can be specified; GS and FS have bases but no limits (but only the lower 32-bits, zero extended, can be loaded from a descriptor table). In long-compatibility mode, segment translations are performed as above except the address is then masked by 0xFFFFFFFF, i.e. truncated to fit the lower 4GB. This, I believe, applies also when the GS_BASE and FS_BASE MSRs are used to set >4GB bases for GS and FS.

[gsingh2011]

It uses page size extensions (PSE) to create 4M pages. You should see a write to CR4 to enable it and some special bits set in the page directory that control size.

That makes more sense now. So because the PSE is bit is set and the PS bit is set, the entry in the page directory points to a 4MB page instead of a 4kB page table. And in this case, the first 10 bits of the entry point to the page, in this case that starts at the address 0x0, so this page includes the kernel, which starts 1 MB into the page (thus explaining the 3GB+1MB). But what I'm still confused about is how the page is mapped to 3GB. Is it because of the placement in the table? Does the fact that the kernel page entry is the 769th entry in the table map it to 3GB?

Segmentation makes more sense now too. I just want to clarify something. So the GDT is mostly for the kernel memory right? And the LDT for process memory? If I was writing a kernel with only segmentation and not paging (I'm not), then every time a program ran the kernel would have to create an LDT and specify a segment and tell the program where to find the LDT, correct? Is there anywhere I can see some code that does this?

[rdos]

If I was writing a kernel with only segmentation and not paging (I'm not), then every time a program ran the kernel would have to create an LDT and specify a segment and tell the program where to find the LDT, correct? Is there anywhere I can see some code that does this?

No. LDT is kept in the LDTR register, and is also part of the TSS. The idea is that the LDT would have different values in different processes, but would not vary between kernel and user. Any valid selector (GDT or LDT) can be loaded in kernel (ring 0), while in userspace (ring 3) only selectors that are defined for ring 3, and use a RPL field of 3, can be loaded. This means that any loaded selectors are maintained as a program goes from userspace into kernel space, while when returning from kernel space to userspace, selectors that are not valid in userspace are replaced with null-values.

[gsingh2011]

It uses page size extensions (PSE) to create 4M pages. You should see a write to CR4 to enable it and some special bits set in the page directory that control size.

That makes more sense now. So because the PSE is bit is set and the PS bit is set, the entry in the page directory points to a 4MB page instead of a 4kB page table. And in this case, the first 10 bits of the entry point to the page, in this case that starts at the address 0x0, so this page includes the kernel, which starts 1 MB into the page (thus explaining the 3GB+1MB). But what I'm still confused about is how the page is mapped to 3GB. Is it because of the placement in the table? Does the fact that the kernel page entry is the 769th entry in the table map it to 3GB?

Could someone answer that question too?

[bitshifter]

If page is 4096 bytes and we have 769 of them...
4096 * 769 = 3149824

[linguofreak]

2. Is the limit of memory available with paging the amount of hard drive space available?

The limit is on your imagination :p you could even swap memory to network.

If you want to get really devious (and accept huge speed and reliability penalties), you could swap to the user's brain:

"Out of memory: Please memorize the following numbers and type them back in when asked for page number 42".

[bluemoon]

Well, in fact, swap to network is not as weird as it sound.
With ultra fast network that is comparable to harddisk IO, there are many project on this, including some specially designed OS/software that group a cluster of computers to enable share physical memory to each other.

Swap to user's brain would require a relative high "Hardware Requirement" :p