Higher Half Tutorial - Loading Kernel as GRUB module

[max]

Hey guys,

I've written a loader (that must be loaded by GRUB) to get my kernel to the higher memory. It basically does the following:
- sets up an initial GDT
- creates a bitmap of all free pages for a simple physical page allocator (which uses GRUBs memory map to check what is free)
- enables paging, recursively mapping the directory to itself in the last entry
- checks if there is a GRUB module with the path "/boot/kernel" ELF binary and creates the virtual image of that binary
- sets up and initial kernel stack and calls the entry method of the binary. the entry must then simply accept 1 parameter of the type "SetupInformation*", which points to a struct that contains all information the kernel later needs: its reserved area (for example 0xC0000000 - 0xC0030000), the location of the initial kernel stack (for example 0xC0031000), and the location of the bitmap (which is still in the lower half and must be copied) [and I will add the Multiboot data, too].

Then the only thing you must do to get a higher half kernel is compile your kernel as an ELF binary, which has a start address in the higher half, and add it with the loader binary into your image, and tell GRUB to boot the loader binary. Then use the setup information (copy needed multiboot data, copy the bitmap) and initialize like a bare bones kernel.

Now I have basically two questions:
1. Should I write a Wiki article (/edit the existing "Higher Half Kernel" article) about how to do this? Because I think that this is a little more advanced, I would not provide code examples, but only give instructions on what steps should (or at least could) be done to develop such a loader
2. Should I release the binary of the loader? This would allow anyone to create a higher half kernel by just compiling their kernel as an ELF binary, virtual linking to the higher half, with an entry point of the signature "void loadKernel(SetupInformation* info)" and voila, there is the higher half kernel, already with paging setup. From there on you can continue by loading a new GDT, setting up interrupts and so on. I don't think that releasing the source is a good idea, because if you know how stuff works, you can simply follow the tutorial (1.) and write your own loader.

What do you think?
Greets

[onlyonemac]

I didn't read what you were doing, but yeah, go for it! The wiki is here to share your knowledge about osdevving, and if you've figured something out, then share it with us!

[Bender]

IMO our wiki should be more manual-like than code/binary based. So yes; I go with option A. Although it'll be more work for the author, but it'll be more helpful to newbies in the long run. You should try explaining the concept of a higher-half kernel, advantages, why to do so, some disadvantages etc. and you may post the loader as a 'try-it-yourself'. If you give someone an understanding of the concept he'll be able to write the code but the reverse isnt true.

[max]

I didn't read what you were doing, but yeah, go for it! The wiki is here to share your knowledge about osdevving, and if you've figured something out, then share it with us!

Haha, okay I'll go for it.

IMO our wiki should be more manual-like than code/binary based. So yes; I go with option A. Although it'll be more work for the author, but it'll be more helpful to newbies in the long run. You should try explaining the concept of a higher-half kernel, advantages, why to do so, some disadvantages etc. and you may post the loader as a 'try-it-yourself'. If you give someone an understanding of the concept he'll be able to write the code but the reverse isnt true.

Yep, I think thats true. It's cool as a total newbie to have your first "Hello world" kernel running, but it's even cooler to know exactly what is going on. So the code examples have their right to exist, too, but more advanced topics should rather encourage the reader to fiddle through the Intel docs by themselves
As there is already the article Higher Half Kernel, I will more focus on what needs to be done than what is the goal

[sortie]

Hi max,

Don't prematurely make up your mind based on the first two posts. It is an unfortunate fact that whoever seems to reply first to a thread gets the most influence.

I actually considered doing something not entirely unlike what you are doing, but with a few important changes. Basically, what you are doing is dealing with the fact that multiboot is crap and wrapping your real kernel in a special loader that that recovers from the multiboot brain damage (such as loading the kernel into physical memory).

I don't think this should go into the wiki right now for the reason that this project seems involved and may need a considerable amount of peer reviewing. Prematurely publishing another unfinished or imperfect Higher Half tutorial to the wiki can do damage. Additionally, it is it large? Is its design sound? Could you share the project in a github repository or something so we can investigate it?

I truly welcome efforts to fix the existing higher half tutorials we have and merging them into a single article, or perhaps even marking them for deletion.

[max]

Hey sortie

Don't prematurely make up your mind based on the first two posts. It is an unfortunate fact that whoever seems to reply first to a thread gets the most influence.

I had basically already started writing a little of it, I just wanted someone to approve it before really putting effort in it

I actually considered doing something not entirely unlike what you are doing, but with a few important changes. Basically, what you are doing is dealing with the fact that multiboot is crap and wrapping your real kernel in a special loader that that recovers from the multiboot brain damage (such as loading the kernel into physical memory).

I actually don't think that multiboot is total crap, because it gives the necessary base to build a kernel upon it (but I must admit that I don't really use information except the memory map and the module positions), and I really wouldn't want it to set up things like paging for me - in that case I would much rather write my own bootloader.

I don't think this should go into the wiki right now for the reason that this project seems involved and may need a considerable amount of peer reviewing. Prematurely publishing another unfinished or imperfect Higher Half tutorial to the wiki can do damage. Additionally, it is it large? Is its design sound? Could you share the project in a github repository or something so we can investigate it?

In the article I don't want to guide people on how to build an exact copy of what I've done. I just want to give them a guideline/manual on what steps are involved to do it that way - sure there are many others. I think the architecture is sound, because it doesn't do anything specific, except enabling paging and loading the kernel. All further steps are left to do by the kernel, you have a clean startup of the kernel, the only thing thats left is using the information you get from the loader to finish the loading and there you go

You can take a look at the source of my implementation here (hope that hoster is okay, I don't have a public repo yet). Some headers are not included, but you can imagine their functionality. It might not be perfect yet, some things need a little refactoring/refining, but it's the basic implementation.

[jnc100]

Ideally, of course, the best way to do it would be to have a GRUB which supports loading your kernel to a particular virtual address and starts in paged protected mode or optionally long mode, then you wouldn't need either a protected mode stub or a separate loader containing an ELF interpreter. I'm sure GRUB could be patched to support this, the problem would be agreeing on the necessary extensions to the multiboot format.

For what its worth I had to also do what the OP did and write a small second stage loader which sets up paging and loads the ELF kernel to the appropriate address and then jumps to it.

Regards,
John.

[Combuster]

You can always link VMA != LMA and use some PIC to set up paging quickly. No separate loaders needed for that...

[jnc100]

You can always link VMA != LMA and use some PIC to set up paging quickly. No separate loaders needed for that...

Unfortunately its not quite that simple. Your PIC stub would need to identify some free pages to use as paging structures, and so would need to parse the multiboot header. Alternatively, you could save some space in your image itself, but then you'd need to know in advance how large your kernel will be when compiled, and allocate the appropriate amount of free space for the paging structures. Then, after enabling paging, you need to rewrite the pointers in the multiboot header (e.g. mmap, modules, elf headers etc) to point to valid virtual addresses, rather than physical ones. By the time you've done all that, you've essentially written a small loader anyway, with the constraint that it needs to be position-independent.

Regards,
John.

[Combuster]

You're making things way more complicated than necessary