Boot x86-64 kernel from GRUB

[quanganht]

AFAIK, Grub hasn't added the ability to boot an 64-bit ELF kernel. So this maybe a workaround: First, Grub is instructed to load a 32-bit "dummy kernel". This dummy then load our real kernel, enter Long mode, and pass control to the real kernel.
What's your opinion?
BTW, how does Linux's 64-bit kernel actually boot with GRUB?

[gerryg400]

I use this method. When I moved to 64bits, I used part of my old 32bit kernel to create my 32bit loader.

AFAIK, Linux is not loaded as an ELF file.

- gerryg400

[StephanvanSchaik]

Hi,

It generally depends on which version of GRUB you're using. GRUB Legacy doesn't support ELF64 natively and thus requires you to write a loader in the ELF32 format (or any other format that is supported by GRUB Legacy) that loads the ELF64 kernel as a module, you can in that case additionally set up long mode in the loader, however, doing this in the ELF64 instead allows you to use GRUB 2 as well, since GRUB 2, at the other hand, supports ELF64 natively and thus can load an ELF64-file and execute it, in which case you only have to set up long mode in the kernel. There's also a "third" method in which you can use a patched version of GRUB Legacy that supports ELF64, but basically this is the same scenario as for GRUB 2.

AFAIK, Linux is not loaded as an ELF file.

Neither does Linux support multiboot directly, since Linux has its very own boot protocol.


Regards,
Stephan J.R. van Schaik.

[Creature]

You can also set PAE (with PML4) up temporarily, map in some low regions (i.e. where your kernel is at), set up a GDT and such and jump to long mode from the multiboot entry point. This is one of the easiest ways of doing it, because you can write a 64-bit kernel which is plainly booted by GRUB, and it doesn't involve having more than one kernel or multiple executables loading one another (the method is described in the wiki). Of course, your way is one of the other potential options.

[quanganht]

You can also set PAE (with PML4) up temporarily, map in some low regions (i.e. where your kernel is at), set up a GDT and such and jump to long mode from the multiboot entry point. This is one of the easiest ways of doing it, because you can write a 64-bit kernel which is plainly booted by GRUB, and it doesn't involve having more than one kernel or multiple executables loading one another (the method is described in the wiki). Of course, your way is one of the other potential options.

Then what excutable format do you use? Because 64-bit code cannot be stored in elf32, and GRUB (legacy) doesn't support Binary or PE without some kind of "hacks". The aim here is to use stock GRUB, even legacy one, to boot 64-bit kernel correctly

[Creature]

You can also set PAE (with PML4) up temporarily, map in some low regions (i.e. where your kernel is at), set up a GDT and such and jump to long mode from the multiboot entry point. This is one of the easiest ways of doing it, because you can write a 64-bit kernel which is plainly booted by GRUB, and it doesn't involve having more than one kernel or multiple executables loading one another (the method is described in the wiki). Of course, your way is one of the other potential options.

Then what excutable format do you use? Because 64-bit code cannot be stored in elf32, and GRUB (legacy) doesn't support Binary or PE without some kind of "hacks". The aim here is to use stock GRUB, even legacy one, to boot 64-bit kernel correctly

Why ELF64 of course with a multiboot header which is "32-bit" (simply [BITS 32], not -elf but still uses -elf64). This way the multiboot header will receive command from GRUB (as any 32-bits kernel would), it then sets up a GDT (with 64-bit entries), sets up PAE paging, etc. etc. and jumps to long mode (and sets [BITS 64]), then it can do some additional setup (stack changing, reading the GRUB multiboot header pointer, which you obviously stored somewhere in the 32-bits code), and finally jump to the 64-bits main. All C/C++ files can this way be compiled as 64-bits without any linking issues. ArcticOS has a fairly good example of this AFAIK. Look for source/browse/trunk/Boot/grub_boot.asm in the SVN tree.

[Owen]

Because 64-bit code cannot be stored in elf32

Wrong. I have successfully linked a significant quantity of x86_64 code into an elf32-i386

[quanganht]

Because 64-bit code cannot be stored in elf32

Wrong. I have successfully linked a significant quantity of x86_64 code into an elf32-i386

So will you show how did you do that?

[xenos]

There is another possibility (which is the one used in my 64 bit kernel): You can use an ELF64 kernel and provide a multiboot header close to the start of this file. GRUB will then recognize it as a multiboot kernel and load it as if it was a flat binary. This puts some limitations on the sections in your kernel ELF64 file: They have to appear in the file exactly as they should appear in memory later, and this may require some output section tweaking in the linker script.

[quanganht]

Hacky?
I have to admit, IMO every piece of code that boots up 64-bit kernel that I have seen is ugly. Any exceptions?

[omnitek]

Because 64-bit code cannot be stored in elf32

Wrong. I have successfully linked a significant quantity of x86_64 code into an elf32-i386

So will you show how did you do that?

I do this to load my 64 bit kernel.
At the start of my linker script I have this:

Code: Select all

OUTPUT_FORMAT(elf32-i386)
OUTPUT_ARCH(i386:x86-64)
ENTRY (start)

I've never had any problems with grub loading this.

[brho]

I do this to load my 64 bit kernel.
At the start of my linker script I have this:

Code: Select all

OUTPUT_FORMAT(elf32-i386)
OUTPUT_ARCH(i386:x86-64)
ENTRY (start)

I've never had any problems with grub loading this.

EDIT: I had two issues with this. Here's the first one:
--------------------------------
I had an issue with this trick using gcc/ld 4.6.1. The issue was that the linker was using 1MB pages instead of 4KB. When looking at the objdump, the first section's file offset was 0x100000, which is beyond the two-page limit that multiboot will scan.

Linking with -z max-page-size=0x1000 (as suggested in http://wiki.osdev.org/Creating_a_64-bit_kernel), fixes that such that the offset is only 0x1000, which is within multiboot detection range. Now, grub legacy loads my "64 bit" kernel.

EDIT: here's the second issue:
--------------------------------
While that linker script hack was sufficient to trick grub legacy to load my kernel, there were some subtle problems with the linkage.

I do *not* recommend linking like that.

The problem I had was with string arguments not being linked properly into the code. Specifically, printk("some string") would not always work, depending on which file was linked in. Sometimes it would print gibberish, and once it printed part of another string in the kernel. I traced it down to differences in the assembly generated by the linker.

Here's the output of a test function in the kernel, with linking the kernel as a 32 bit with:

Code: Select all

OUTPUT_FORMAT(elf32-i386)
OUTPUT_ARCH(i386:x86-64)

Code: Select all

        c0132280 <myfunc>:
        c0132280:   55                      push   %ebp
        c0132281:   48                      dec    %eax
        c0132282:   c7 c7 3c f2 13 c0       mov    $0xc013f23c,%edi
        c0132288:   31 c0                   xor    %eax,%eax
        c013228a:   48                      dec    %eax
        c013228b:   89 e5                   mov    %esp,%ebp
        c013228d:   e8 6e 99 fe ff          call   c011bc00 <cprintf>
        c0132292:   e8 f9 3d 00 00          call   c0136090 <vm_init>
        c0132297:   eb fe                   jmp    c0132297 <myfunc+0x17>
        c0132299:   55                      push   %ebp
        c013229a:   48                      dec    %eax

Here's the output with the linker creating a proper x86-64, with:

Code: Select all

OUTPUT_FORMAT("elf64-x86-64", "elf64-x86-64", "elf64-x86-64")

Code: Select all

ffffffffc0132280 <myfunc>:
ffffffffc0132280:   55                      push   %rbp
ffffffffc0132281:   48 c7 c7 20 eb 13 c0    mov    $0xffffffffc013eb20,%rdi
ffffffffc0132288:   31 c0                   xor    %eax,%eax
ffffffffc013228a:   48 89 e5                mov    %rsp,%rbp
ffffffffc013228d:   e8 6e 99 fe ff          callq  ffffffffc011bc00 <cprintf>
ffffffffc0132292:   e8 f9 3d 00 00          callq  ffffffffc0136090 <vm_init>
ffffffffc0132297:   eb fe                   jmp    ffffffffc0132297 <myfunc+0x17>

ffffffffc0132299 <nr_sem_waiters.part.1>:
ffffffffc0132299:   55                      push   %rbp
ffffffffc013229a:   48 c7 c1 b0 c9 13 c0    mov    $0xffffffffc013c9b0,%rcx

And finally, here's the output if you take the 'linked as 64' and objcopy it to 32 bit with the command:

Code: Select all

$ objcopy -I elf64-x86-64 -O elf32-i386 KERNEL

Code: Select all

        c0132280 <myfunc>:
        c0132280:   55                      push   %ebp
        c0132281:   48                      dec    %eax
        c0132282:   c7 c7 20 eb 13 c0       mov    $0xc013eb20,%edi
        c0132288:   31 c0                   xor    %eax,%eax
        c013228a:   48                      dec    %eax
        c013228b:   89 e5                   mov    %esp,%ebp
        c013228d:   e8 6e 99 fe ff          call   c011bc00 <cprintf>
        c0132292:   e8 f9 3d 00 00          call   c0136090 <vm_init>
        c0132297:   eb fe                   jmp    c0132297 <myfunc+0x17>

        c0132299 <nr_sem_waiters.part.1>:
        c0132299:   55                      push   %ebp
        c013229a:   48                      dec    %eax

The only thing that really matters is the actual instruction stream, not necessarily the disassembled output on the far right. Note instruction c0132282 (mov ADDR, %rdi). The 32-bit-linked code thinks the address of the string is c013f23c, while the 64 bit code thinks it should be c013eb20. The objcopy'd version agrees with the 64 bit version. This is seen in both the instruction stream and the disassembly. Also note the 32-bit-linked version dropped the symbol at c0132299 for <nr_sem_waiters.part.1>. Not sure if that's a big deal or not.

After switching over to the "objcopy" method, the original bug was fixed.

Anyway, if you're trying to get your 64 bit kernel running from an unmodified grub legacy, I would put this in your linker script:

Code: Select all

OUTPUT_FORMAT("elf64-x86-64", "elf64-x86-64", "elf64-x86-64")

and link with this flag:

Code: Select all

-z max-page-size=0x1000

and after you build the kernel, objcopy it to its new format:

Code: Select all

$ objcopy -I elf64-x86-64 -O elf32-i386 KERNEL 

If you want to disassemble your kernel (for debugging), do it before objcopy. The way I ran objcopy (with no output file listed), your original KERNEL file will be replaced. Ideally, you want to objdump with the file in the correct format (or you could use -M x86-64, but there's really no need).

FWIW, I'm linking my kernel for 0xffffffffc0000000 and compiling with -mcmodel=kernel.

[bluemoon]

Hacky?
I have to admit, IMO every piece of code that boots up 64-bit kernel that I have seen is ugly. Any exceptions?

Since I have no idea how many piece of code you've seen, I can't prove otherwise.
However, I guess some forks in this board created some elegant boot loaders.

And this does not necessary be hacky.
Basically, you detect cpu architecture, branch to boot32 or boot64 and load elf32/elf64 accordingly, and optionally implement multi-boot alike interface, simple.