Loading kernel low or high

[Kemp]

Hi. I've noticed there's a tendency in the more 'professional' looking/sounding hobby OSes and in a lot of commercial/well-known ones to load the kernel high up in memory and I was wondering about the reasons for this. After some thought I came up with the following possible reasons for chosing each way:

Loading High
Point: Allows applications to have a 0 base
Counter-point: Any half-decent executable format has relocation tables making fore-knowledge of the base unnecessary

Point: Though remapping has to be done before the jump to the 'real' code thus disallowing the final memory manager full control over the process, it can be passed details of what happened
Counter-point: Memory may be set up wrong, eg on a NUMA system if the remapping part doesn't know how to handle the full list of system types that the memory manager knows about


Loading Low
Point: Much simpler not having to handle loading low and then remapping up high with the tricks or multiple kernel versions inherent in this
Counter-point: Existing examples can be adapted and used

Point: Applications can be given a 0 base by starting their segment base after the kernel code
Counter-point: Segments are being phased out, eg on AMD64 systems in full 64-bit mode this will not work


There's a few more points and counter-points I thought of, but you get the idea. I was just wondering what particular advantage pushed you in either direction for your particular design. I'm personally thinking of keeping the kernel low because a 0 base for applications really isn't necessary given relocation tables.

[AR]

This has been discussed before, one of the other main points is expanding down from high is easier then expanding up (by default executables do not have relocation entries, you have to manually enable them), so if you use 0-1GB as kernel, then in the extremely unlikely event you need more than 1GB you'll be stuffed since every app will have to be recompiled (in situations like Windows with thousands of 3rd party vendors, it simply isn't reasonable to think people will rebuild for you).

A more advanced reasoning for the above point is that migrating from IA-32 to AMD64 means that you can take more virtual memory for the kernel but if the kernel is built low then it will be more difficult to support 32bit software.

Of course, if you want to force relocation entries for everything then you can circumvent all this.

[Pype.Clicker]

The main difficulty with code relocation is when to relocate. For instance, it may be interresting not to load the whole 32MB of HugeOffice or GiantFotochop binary straight when starting the program, but rather load a couple of "commonly needed" stuff, and then let the page faults tell you what to load next.

But that means when a page fault occur, you need to be able to handle relocations for that page before execution resumes. If you need later to free pages of code (e.g. swap them out, but as they're read-only, you don't actually swap them, just mark them unavailable) , you'll need to relocate them again when you'll load the stuff again.

The real question would rather be "why would your kernel require more than 2^x (x being 3..7) MB of storage for its own needs" ?

Replies usually are:
- all the kernel stacks are available in all processes (but you can easily keep kernel stacks local to their process, anyway)
- all the physical pages are available to the kernel in all process (e.g. out of your 1GB of kernel space, 512 MB are 1:N+1 mapping of the physical memory -- but you're in for trouble as soon as someone has more memory).
- i need plenty of room for disk caches, offscreen buffers, pixmaps, etc. (but do you really need them _mapped_ by the kernel ?)

[Solar]

Another question is how high to load the kernel. While AR already pointed out that expanding (or shrinking!) a high-loaded kernel is easier than a low-loaded one, it's still a design question how much address space you reserve for your kernel, and how much is open for apps to use.

Linux uses 3 GB app, 1 GB kernel by default. WinNT used 2 GB each.

And on the production server I'm working on / for, we're closely watching that caching proxy process that has grown to 3.18 GB in size and continues to grow, because even on our 64 GB machine a process is limited to 3.5 GB in size and we don't really know how to cope with that limitation. So don't tell me 2 / 3 GB ought to be enough for everybody.

[Kemp]

by default executables do not have relocation entries, you have to manually enable them

Oh, I thought it was standard (at least with the elf format), as I haven't seen people talk about them *without* the relocation table.

if you use 0-1GB as kernel, then in the extremely unlikely event you need more than 1GB you'll be stuffed since every app will have to be recompiled

That is a good point, I didn't notice this due to my assumption of the apps being relocatable.

The real question would rather be "why would your kernel require more than 2^x (x being 3..7) MB of storage for its own needs" ?

I don't follow why that would be the real question... ??? If the kernel ended up requiring 128MB of storage for its own stuff then that's easily handled by the 1GB allocation (on 32-bit systems) that it would have whether it's mapped up high or low.


Also, a point I didn't make in the first post: I'm talking about where it's located in virtual memory. Looks like the two of you assumed that anyway, but just in case someone else thinks I'm talking about where to put it in physical memory.


Edit (Solar's post wasn't there before I posted):

At the moment I'm thinking of a 1GB/3GB split on x86 systems and on systems that allow more probably expand to 4GB for the kernel and the rest for apps (though if I use that much for the kernel then I'm probably doing something wrong somewhere Even XP with SP2 only takes up that much if you have restore points switched on). Of course, only having a 4GB virtual space doesn't help that idea, but I'm sure I can pull a few tricks with it.

[Kemp]

Hmmm... Assume you only have a 4GB virtual address space and you can't pull tricks to make it appear like more. You're left with your allocation scheme the same as on a 32-bit machine (1GB/3GB for me). I'm thinking it might be a good idea to allow the application (via the header or something maybe) to specifiy that it requires as much as possible and performance is allowed to suffer to a small degree as long as it can have as much of the 4GB as it can scrounge. In this case, it would be possible to map none (or a very limited amount) of the kernel into its address space and to switch space to one containing the rest of the kernel at the same time the stack is switched when a system call is made. For something like a database server, the user probably won't notice a few tenths of a second added onto the time required to fetch results when compared to the time taken to actually generate said results normally, especially as said database server probably won't be making a lot of system calls. It would probably get along happily just having the filesystem/disk driver available to it mainly.

[/brain dump]