I wrote a simple HigherHalf kernel...

[Pype.Clicker]

For relocatable binaries you have to use swap space instead of the original executable file because the binary was modified after it was loaded.

Well, if you can tweak the memory manager, you can also re-load the page from the executable binary and re-apply relocation on the fly (if you have the part of relocation list present in memory, the cost of re-relocating is miserably small compared to the cost of of loading the page from the disk)

You could also try hard to have the same binary loaded at the same location, even if you have the option to relocate it ... that means if you have the choice between fragmenting the address space or relocating a highly-used binary, go for fragmentation.

[JAAman]

the practice of loading the OS at top_of_mem, comes from the old era when the OS used to be burned into rom chips -- standard convention says that the CPU will expect top_of_mem to contain ROM and the bottom_of_mem to contain RAM therefore the OS must be in the top_of_mem (of course nowdays the OS is loaded after startup from disk so this is no longer required) notice how all rom chips are always at the top of memory and ram at the bottom, afaik, this is true for all CPUs (they either start, or expect a pointer to entry, to be at the top of the address space, and the bottom must be ram --
x86 starts at top-16 and places the default IDT at 0
6502 starts at vector at top(there were 4 vectors don't remember which one was boot), and places stack at 0+256

other chips are most likely very similar (though my experience with them is minimal)

[Brendan]

Hi,

the practice of loading the OS at top_of_mem, comes from the old era when the OS used to be burned into rom chips -- standard convention says that the CPU will expect top_of_mem to contain ROM and the bottom_of_mem to contain RAM therefore the OS must be in the top_of_mem (of course nowdays the OS is loaded after startup from disk so this is no longer required) notice how all rom chips are always at the top of memory and ram at the bottom, afaik, this is true for all CPUs (they either start, or expect a pointer to entry, to be at the top of the address space, and the bottom must be ram --
x86 starts at top-16 and places the default IDT at 0
6502 starts at vector at top(there were 4 vectors don't remember which one was boot), and places stack at 0+256

IIRC 6502 had a "zero page" which was the first 256 memory locations. It was important because you could access anything in this area with a 2 byte instruction (rather than the 3 byte instructions you'd normally use).

You can also save a few bytes on 80x86 by doing similar things (e.g. "movzx esi, byte address" or "mov eax,[word address]") but no-one worries about code size anymore - I've never even seen an optimizer that will do it this.


Cheers,

Brendan

[Colonel Kernel]

You could also try hard to have the same binary loaded at the same location, even if you have the option to relocate it ... that means if you have the choice between fragmenting the address space or relocating a highly-used binary, go for fragmentation.

This is exactly what NT does. DLLs have a preferred base address, which you can either leave as default when you build them or set via a linker switch. If the DLL can't load at its preferred address, it is "rebased" and can no longer be demand-paged, as Brendan pointed out.

The NT kernel probably sets CS base to <whatever> and possibly even uses a CS segment override prefix to access kernel data.

IIRC, NT uses a strictly flat model and mostly ignores segmentation, so I doubt it uses CS to re-locate. My guess is it just gets fixed up at load time.

[carbonBased]

Because virtual 8086 tasks must use the lowest 1 MB of linear memory, so if you plan on having several "DOS boxes" or something you'll be screwed if the lowest 1 MB is mapped the same into every address space (e.g. treated the same as kernel space). There are ways around this, but in general leaving the lowest N MB for user applications would make it easier to allow any user-level processes to contain virtual 8086 code.

Hmm... I may have a misconception, then. I was under the impression vm86 used the lowest 1MB of *physical* memory. In fact, I seem to recall someone saying it was actually 16MB, but I've never heard that again, nor have I consulted the Intel manual to prove/disprove it.

In any event, I'm narrowminded in my usage, thinking only of my OS, in which case this is actually a reason *for* having the OS at the bottom end. Your definition of "thunking" would be my only use for vm86 (as fallback drivers for video, etc).

2. More generically, user applications are not dependent of how many memory is kernel space (your app can be linked for 0x400000 regardless of whether kernel is at 0xC0000000, 0x80000000 or 0xE0000000 ...), which makes ABIs nicer.

So if the kernel is from 0x00000000 to 0x3FFFFFFFF you'd have applications code starting at 0x40000000 with application data above the application's code. Then, if the kernel happens to be from 0x00000000 to 0x1FFFFFFFF would you have the application at 0x40000000 with the application's data above this and wasted space from 0x20000000 to 0x3FFFFFFFF, or would you have a fragmented system with data above the applications code and below it? How about if the kernel grew and used from 0x00000000 to 0x7FFFFFFFF - would you still be able to run applications that are designed to be at 0x40000000?

All the kernel's I've seen that have their kernel mapped high map it to something similar to 0xC0000000, and occupy the top 1GB. Due to this, apps can only span 3GB.

If the kernel were loaded at the lower end, it may occupy up to 0x40000000. Thus, apps are still limited to 3GB.

Unless I'm missing something, I don't see the difference. If the kernel grows larger from the top, applications are then limited in size as well.

If PIC is, indeed, a hack on Intel, then I guess I can see that picking an arbitrary value to load apps in can be a problem... if your kernel grows above 1GB... this seems unlikely, esp. given a microkernel who's servers will exist in userspace, anyway.

So, yes, perhaps I'm wasting some address space in only allowing apps 3GB. All the examples of higherhalf kernels seem to do the same thing, however.

IMHO the 80x86 CPU is not designed for 32 bit position independant code (at least not without using segmentation), and therefore all attempts at making it run 32 bit position independant code are ugly hacks. I don't like ugly hacks, and I don't agree that all applications should be ugly hacks (even if your compiler hides these hacks, they still exist).

I must admit, I haven't much tried it, so I can't comment on this.

If the OS is designed well it's easy to remove all of the kernel from the 32 bit part of the address space. Problems come from a badly designed ABI - for e.g. allowing (or expecting) applications to access kernel data directly would cause problems (but then you've got problems with any design change that effect the structure or location of this data, and the kernel can't safely modify the data in a multi-CPU environment).

Indeed, I wasn't suggesting the app actually be able to access or modify the OS structure, but rather thinking of a setup whereby calling the OS would result in a pager context switch followed by a jump to a specific kernel function. Upon further though, however, this isn't feasable anyway...

so, yes, I'll conceed that a higher-half kernel has the advantage of allowing 32-bit apps on a 64-bit OS to have the full 32-bit memory space.

Let's consider this from the opposite perspective - is there any benefit from having the kernel in the lower part of the address space?

The only benefit I can think of is that it makes it easy for people to pretend they are writing an OS (e.g. slapping a "hello world" kernel on top of GRUB and then having a party to celebrate). Maybe I've missed something...

I'm not suggesting lower half kernels have an advantage over anything. I was suggesting neither have advantages over the other.

And I hope you're not hinting that I might be one of those "hello world" developers...

--Jeff

[carbonBased]

blah... my usage of quote sucks...

[Brendan]

Hi,

Hmm... I may have a misconception, then. I was under the impression vm86 used the lowest 1MB of *physical* memory. In fact, I seem to recall someone saying it was actually 16MB, but I've never heard that again, nor have I consulted the Intel manual to prove/disprove it.

In any event, I'm narrowminded in my usage, thinking only of my OS, in which case this is actually a reason *for* having the OS at the bottom end. Your definition of "thunking" would be my only use for vm86 (as fallback drivers for video, etc).

Virtual 8086 always uses linear memory (not physical memory, unless you disable paging or identity map) and (like real mode) can only access a little more than the first 1 MB (up to 0xFFFF:0xFFFF, or 0x0010FFEF).

If PIC is, indeed, a hack on Intel, then I guess I can see that picking an arbitrary value to load apps in can be a problem... if your kernel grows above 1GB... this seems unlikely, esp. given a microkernel who's servers will exist in userspace, anyway.

So, yes, perhaps I'm wasting some address space in only allowing apps 3GB. All the examples of higherhalf kernels seem to do the same thing, however.

Tutorials are just tutorials - compare them to Windows NT and Linux (which both have an adjustable kernel space" AFAIK).

Perhaps a better idea may be to compare the tutorials to my plans - I'm doing a 32 bit kernel with 32 bit paging, a 32 bit kernel with PAE and a 64 bit kernel. All of these kernels will run the same 32 bit processes, and all of them will have a different "top of memory" (either 3.5 GB, 3 GB or 4 GB) due to the way paging works in each case (and other factors).

Let's consider this from the opposite perspective - is there any benefit from having the kernel in the lower part of the address space?

The only benefit I can think of is that it makes it easy for people to pretend they are writing an OS (e.g. slapping a "hello world" kernel on top of GRUB and then having a party to celebrate). Maybe I've missed something...

I'm not suggesting lower half kernels have an advantage over anything. I was suggesting neither have advantages over the other.

And I hope you're not hinting that I might be one of those "hello world" developers...

I guess it'd be fair to say that either method has advantages which depend on the exact design of the OS (for e.g. if virtual 8086 is used for "thunking" only, and if the OS is using PIC or the size of kernel space will never change then it might be better to put the kernel at the bottom of each address space), but that putting the kernel at the top of the address space has more advantages for more OS designs.

For example, in your case it could go either way - I'd be tempted to make the virtual 8086 code more generic (so it can be used for video, SCSI BIOS functions, a virtualizer, etc) and make the video code run entirely as an independant service (rather than having special virtual 86 hooks into the kernel, which is going to make it harder to support computers with multiple video cards).

I wasn't hinting that you might be one of the "hello world" developers, it's just that I've seen too many people start with C code booted by GRUB (loaded at 1 MB) which is then identity mapped when paging is initialized. They tend to do it this way because it's easy, not because they've thought about it or planned it this way, and not because it's what they actually wanted.

As far as I can tell, you have thought about it, researched it and planned it. To be honest, when I first started I was a "hello world" developer, but no-one was using protected mode back then and GRUB didn't exist. For my first "OS", all applications where built into the same bootable binary, "files" were written directly into floppy disk sectors (no file system) and the BIOS was used as much as possible. It worked, in that it could chainload other OSs at any time and you could use the text editor and save/load your work, but it was a complete waste of time - it wasn't planned and wasn't what I wanted...


Cheers,

Brendan

[falconfx]

Finally the tutorial is into the OSFAQ. I'm very happy: it's the first tutorial of my life!

I'd like to know what you think about it and read some comments!

Cheers,

falconfx