Another of those "can't enter long mode" post...

[kzinti]

First some background about my project setup...

- I have an EFI bootloader implementation that loads the kernel (an elf file), setups page mapping and then jump to the kernel. I support both ia32 and x86_64 targets. Everything is working fine here.
- I implemented a multiboot bootloader (loaded by Grub) that uses much of the same code because I want to be able to run my kernel in Bochs. This bootloader is always ia32 code but can load both an ia32 and x86_64 kernel, because why not.
- The multiboot loader shares the same code as the EFI bootloader for page mapping, elf parsing and so on.

So in theory, all I need is some custom code to jump from the ia32 bootloader to the x86_64 kernel. Lucky me I have implemented this in a previous OS project so I just copied the code over and it should just work right? Well it does, on Bochs at least. But on QEMU and real hardware, it fails miserably.

- Works with EFI bootloader
- Works with BIOS under Bochs (both ia32 and x86_64)
- Works with BIOS under QMU and hardware for ia32
- Does not work with BIOS under QEMU or hardware for x86_64

It fails when I enable paging. Boom. So two possibilities comes to mind:

1) My page tables are not setup properly. But they are exactly the same ones I use with x86_64 EFI. I went through the pain of printing them all out and comparing them. They look the same.
2) My code that jumps from 32 to 64 bits code has a bug. I've been checking and rechecking these ~20 lines of code over and over against my old OS project and documentation on the wiki, other places on the internets and Intel reference manuals. I can't find anything wrong with it.

Clearly I am missing something, but after days of banging my head on the wall, I am looking for a saviour to help me.

Here is the function I use to jump to long mode (it crashes on line 171 when I try to enable paging):
https://github.com/kiznit/rainbow-os/bl ... try.S#L156

The page table setup is here:
https://github.com/kiznit/rainbow-os/bl ... x86_64.hpp

The way it works is:
1) I call init() to identity map the first 4 GB - https://github.com/kiznit/rainbow-os/bl ... ot.cpp#L72
2) The elf loading code calls map() to map the kernel in high memory - https://github.com/kiznit/rainbow-os/bl ... r.cpp#L388
3) enable() is called to set cr3 before I call jumpToKernel64() - https://github.com/kiznit/rainbow-os/bl ... t.cpp#L150

Any help much appreciated, this is driving me nuts.

[MichaelPetch]

I haven't looked at any of the code, but I assume you enabled A20.

[kzinti]

I am using grub for my multiboot loader, so that's already taken care of for me. Or is it? I sure didn't do it in my previous OS project and I also don't do it for the 32 bits kernel.

edit: yeah I am covered by the multiboot specification, section 3.2: https://www.gnu.org/software/grub/manua ... iboot.html

[MichaelPetch]

It is taken care of if you are using an existing one. As I said I didn't look at your code, and I misinterpreted "I implemented a multiboot bootloader".I assumed you wrote a replacement multiboot loader from scratch as a replacement for GRUB (or other multiboot loader)

[Octocontrabass]

It fails when I enable paging. Boom.

What does QEMU have to say about the failure? Anything interesting in its diagnostic outputs?

[kzinti]

I am getting a page fault that ends up in a triple fault:

Code: Select all

(qemu) check_exception old: 0xffffffff new 0xe
     2: v=0e e=0019 i=0 cpl=0 IP=0010:0000000000105427 pc=0000000000105427 SP=0018:0000000000136f74 CR2=0000000000105427
EAX=80000011 EBX=ffffffff ECX=c0000080 EDX=00000000
ESI=00001000 EDI=00000000 EBP=00136f9c ESP=00136f74
EIP=00105427 EFL=00000016 [----AP-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0018 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
CS =0010 00000000 ffffffff 00cf9a00 DPL=0 CS32 [-R-]
SS =0018 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
DS =0018 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
FS =0018 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
GS =0018 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT=     0000000000106a40 00000020
IDT=     0000000000000000 00000000
CR0=80000011 CR2=0000000000105427 CR3=00000000bffcf000 CR4=00000020
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=00000014 CCD=00136f84 CCO=EFLAGS  
EFER=0000000000000d00

Here is the disassembly around 0x105427

Code: Select all

00105406 <jumpToKernel64>:
  105406:	0f 20 e0             	mov    %cr4,%eax
  105409:	0f ba e8 05          	bts    $0x5,%eax
  10540d:	0f 22 e0             	mov    %eax,%cr4
  105410:	b9 80 00 00 c0       	mov    $0xc0000080,%ecx
  105415:	0f 32                	rdmsr  
  105417:	0f ba e8 08          	bts    $0x8,%eax
  10541b:	0f 30                	wrmsr  
  10541d:	0f 20 c0             	mov    %cr0,%eax
  105420:	0f ba e8 1f          	bts    $0x1f,%eax
  105424:	0f 22 c0             	mov    %eax,%cr0
  105427:	0f 01 15 70 54 10 00 	lgdtl  0x105470
  10542e:	b8 10 00 00 00       	mov    $0x10,%eax
  105433:	8e d8                	mov    %eax,%ds
  105435:	8e d0                	mov    %eax,%ss
  105437:	ea 3e 54 10 00 08 00 	ljmp   $0x8,$0x10543e

It does seem to suggest something wrong with my page tables... Specifically the identity mapping of the first 4 GB.

[MichaelPetch]

I've tried to clone your bochs branch which seems to have the recent changes.I get an error when trying to make the image (yes I have even set the MACHINE and ARCH type). I have read your readme file but when I try to build the images it can't find the kernel. Is the bochs branch what we should be using? Do you have a set of instructions to retrieve and build the version that fails for you. I'd like to look at it myself but I can't get past the building stage.

[bzt]

Btw, it does not fail when you enable paging. It fails when you use lgdt, which is the first instruction that uses ds and a memory operation.

It does seem to suggest something wrong with my page tables... Specifically the identity mapping of the first 4 GB.

For one, I don't see where you set up cr3 (which is required for long mode as you know). You should start qemu with monitor, and try "info mem" and "info tbl" to list paging tables to see if they're correct. Also you could dump (with "x") the tables and compare them with the dump made in bochs. I think they should be the same bit by bit (unless you allocate the memory for the page tables dynamically). If the tables differ, then you should investigate why. Comparing the control registers' values could be useful too.

As a general rule, if your code works in one emulator but not the other, you should always look for any difference, then try to figure out what's causing that difference. Once you've found that, you'll know how to fix.

Cheers,
bzt

[MichaelPetch]

Btw, it does not fail when you enable paging. It fails when you use lgdt, which is the first instruction that uses ds and a memory operation

. It isn't failing at referencing the GDT and the memory address.The output is clear that CR2=0000000000105427 when that page fault was raised. That address corresponds to the instruction lgdtl 0x105470. That out says to me that when paging was enabled on the instruction before, the next instruction couldn't even be fetched at all. LGDTL wasn't executed at all. Had this been an issue with accessing memory location ds:0x105470 then CR2 would have been 0x105470.

My conclusion is that for whatever reason the lgdtl instruction at 0x105427 was never executed because trying to fetch that instruction failed which suggests some kind of paging problem. CR3 would have probably been set (or should have) somewhree in the multiboot_main code that sets everything up.

Things that come to mind are that when he ran his multi_boot code an entry in the paging structure wasn't invalidated when it had to be, possible the page table structures were not properly initialized (bogus data/bits - uninitialised variable or memory somewhere)? Did some inline assembly code not force data to be flushed to memory before being executed? (Missing "memory" clobbers or equivalent etc)? Is he accessing RAM that is unusable - ie page structure in a memory hole that isn't usable?

[bzt]

My conclusion is that for whatever reason the lgdtl instruction at 0x105427 was never executed because trying to fetch that instruction failed.

Probably you're right. Anyway, the problem lies with paging for sure. We should see if CR3=00000000bffcf000 is correct or not (that's around 3G, it's likely that the vm has less RAM).

Just a sidenote, the elf loader relies on ELF Section Headers. They may not exists, it would be better to use Program Headers.

Cheers,
bzt

[kzinti]

I've tried to clone your bochs branch which seems to have the recent changes.I get an error when trying to make the image (yes I have even set the MACHINE and ARCH type). I have read your readme file but when I try to build the images it can't find the kernel. Is the bochs branch what we should be using? Do you have a set of instructions to retrieve and build the version that fails for you. I'd like to look at it myself but I can't get past the building stage.

Yes the Bochs branch is the right one. I did add a commit or two since I created this thread, but they shouldn't matter.

To run it, all you should need to do (assuming a x86_64 Linux):

Code: Select all

make MACHINE=bios run-bochs

If this doesn't work, any chance of posting the build log? I'd be curious to see where it fails for you.

My conclusion is that for whatever reason the lgdtl instruction at 0x105427 was never executed because trying to fetch that instruction failed which suggests some kind of paging problem.

That's what I concluded as well.

Things that come to mind are that when he ran his multi_boot code an entry in the paging structure wasn't invalidated when it had to be, possible the page table structures were not properly initialized (bogus data/bits - uninitialised variable or memory somewhere)? Did some inline assembly code not force data to be flushed to memory before being executed? (Missing "memory" clobbers or equivalent etc)? Is he accessing RAM that is unusable - ie page structure in a memory hole that isn't usable?

These are the things I am looking into as well. But there might be something more obvious in my page setup code that I am not seeing. The problem has to be in how I identity map the first 4 GB (i.e. this problem is not in the map() function since I never reach the kernel).

Unless... Mapping the kernel overwrite the page table entries I setup in the first 4 GB... Something I'll look at later today.


Thanks for taking the time to look at this!

[kzinti]

Btw, it does not fail when you enable paging. It fails when you use lgdt, which is the first instruction that uses ds and a memory operation.

Enabling paging makes it fail when it tries to fetch the LGDT instruction. The LGDT is never executed. I verified this by putting an infinite loop right before the LGDT instruction. Sure enough, I get a page fault when trying to fetch the JMP instruction.

I am getting pretty convinced that my page tables have a problem. What's confusing me is that the same code (same page tables) run fine with the EFI version. Manual inspection also seems to show that everything is fine. Clearly I am missing something... In one case (BIOS) I compiled for ia32. In the EFI case I compile for x86_64. That's an obvious difference. So I've been trying to find anything in that paging code that would be different (type casts for example), but I don't see anything. I do a few explicit casts from 64 bits to 32 bits for physical addresses returned by the memory allocator, but this should be fine as all physical addresses are within the first 4 GB of memory (I have verified this as well).

For one, I don't see where you set up cr3 (which is required for long mode as you know).

Just before calling the assembly trampoline code, I call vmm_enable() here (which only sets CR3 in this case, it does not turn on paging):
https://github.com/kiznit/rainbow-os/bl ... t.cpp#L150

Which ends up setting CR3 here:
https://github.com/kiznit/rainbow-os/bl ... 64.hpp#L89

I know this parts work because I printed the value on on the screen and it matches what I see in the exception info from QEMU.

You should start qemu with monitor, and try "info mem" and "info tbl" to list paging tables to see if they're correct. Also you could dump (with "x") the tables and compare them with the dump made in bochs. I think they should be the same bit by bit (unless you allocate the memory for the page tables dynamically). If the tables differ, then you should investigate why. Comparing the control registers' values could be useful too.

Thanks, I'll take a look tonight as I am at work right now. I don't have much experience debugging with QEMU. I am much more used to Bochs, where things work for some reason. The tables cannot match bit by bit as both emulator use a different BIOS and the memory for the page tables is dynamically allocated (ending up in different locations under each emulator). But I could try to use static tables instead to see if it helps track down the issue. It's a good idea to try to have both emulators be in the same state.

As a general rule, if your code works in one emulator but not the other, you should always look for any difference, then try to figure out what's causing that difference. Once you've found that, you'll know how to fix.

I run the exact same code on both emulators. So I am looking for differences between Bochs and QEMU, but this kind of information is hard to come by. Hence my post.

We should see if CR3=00000000bffcf000 is correct or not (that's around 3G, it's likely that the vm has less RAM).

My page allocator uses the memory info provided by GRUB. I initialize QEMU with 8 GB, which is correctly reported by GRUB.

Just a sidenote, the elf loader relies on ELF Section Headers. They may not exists, it would be better to use Program Headers.

This doesn't sound very likely... Both readelf and objdump confirms that my kernel does have section headers. The code also crashes after I load the kernel and before I jump to it (i.e. the crash is within the bootloader and has nothing to do with the kernel loading).

The exact same image run fines under Bochs. The ia32 version also runs fine other both Bochs and QEMU.

Thanks for your help!

[bzt]

I run the exact same code on both emulators. So I am looking for differences between Bochs and QEMU, but this kind of information is hard to come by. Hence my post.

What I meant is dump registers and memory at cr3 in both, then compare those dumps.

My page allocator uses the memory info provided by GRUB. I initialize QEMU with 8 GB, which is correctly reported by GRUB.

That's a lot of RAM, but right, it's not that then. Worth a shot, would been an easy fix.

This doesn't sound very likely... Both readelf and objdump confirms that my kernel does have section headers. The code also crashes after I load the kernel and before I jump to it (i.e. the crash is within the bootloader and has nothing to do with the kernel loading).

This is a totally unrelated thing to your crash. It could be that you have section headers, but according to spec they only required for linking, not for loading. But of course you can say that your kernel elf must have section headers in order to be loaded, it is your OS after all

The exact same image run fines under Bochs. The ia32 version also runs fine other both Bochs and QEMU.

Thanks for your help!

Yeah, maybe it runs, but if the code was correct it would run under all emulators. So there must be some nasty bug somewhere, meaning you were lucky with Bochs. I'm sorry I couldn't help more. I wish you good luck with the dumps! Hope you'll find the problem soon!

Cheers,
bzt

[kzinti]

This is a totally unrelated thing to your crash. It could be that you have section headers, but according to spec they only required for linking, not for loading. But of course you can say that your kernel elf must have section headers in order to be loaded, it is your OS after all

That's very interesting. I didn't know this. I use the section headers to find the symbols in order to apply relocations (my kernel is a relocatable executable aka a shared object / .so). Is there another way to go about relocating my binary?

[MichaelPetch]

Just dealing with the build issues,I started from scratch on each of these tests by using git clone --single-branch --branch bochs https://github.com/kiznit/rainbow-os.git.I did this because I discovered this afternoon that doing a "clean" doesn't seem to do a proper clean and appeared to keep files laying around. This caused the situation where the build appeared to work except for building the image. When I started from scratch things were quite different.

I'm using Debian 9 "Stretch" and my GCC is:

Code: Select all

Using built-in specs. COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/6/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 6.3.0-18+deb9u1' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)

So the first thing I did was run make MACHINE=bios run-bochs as you suggested and got:

Code: Select all

rm -f -rf /root/so/bootload/booter288/rainbow-os//build/x86_64
root@debianwheezy:~/so/bootload/booter288/rainbow-os# make MACHINE=bios run-bochs
mkdir -p /root/so/bootload/booter288/rainbow-os//build/x86_64/boot && cd /root/so/bootload/booter288/rainbow-os//build/x86_64/boot && MACHINE=bios make -f /root/so/bootload/booter288/rainbow-os//boot/Makefile
make[1]: Entering directory '/root/so/bootload/booter288/rainbow-os/build/x86_64/boot'
Adding module '.' at '/root/so/bootload/booter288/rainbow-os/boot/'
Adding module 'arch' at '/root/so/bootload/booter288/rainbow-os/boot/arch/'
Adding module 'machine/bios' at '/root/so/bootload/booter288/rainbow-os/boot/machine/bios/'
Adding module 'graphics' at '/root/so/bootload/booter288/rainbow-os/graphics/'
Adding module 'metal' at '/root/so/bootload/booter288/rainbow-os/metal/'
gcc -m32 -mno-mmx -mno-sse -mno-avx -O2 -Wall -Wextra -Werror -ffreestanding -fbuiltin -fno-exceptions -fno-rtti -DKERNEL_ARCH=x86_64 -DKERNEL_X86_64 -I/root/so/bootload/booter288/rainbow-os/boot/ -I/root/so/bootload/booter288/rainbow-os/ -I/root/so/bootload/booter288/rainbow-os//third_party -I/root/so/bootload/booter288/rainbow-os//third_party/uefi -MMD -MP -c /root/so/bootload/booter288/rainbow-os/boot/crt.cpp -o crt.cpp.o
gcc -m32 -mno-mmx -mno-sse -mno-avx -O2 -Wall -Wextra -Werror -ffreestanding -fbuiltin -fno-exceptions -fno-rtti -DKERNEL_ARCH=x86_64 -DKERNEL_X86_64 -I/root/so/bootload/booter288/rainbow-os/boot/ -I/root/so/bootload/booter288/rainbow-os/ -I/root/so/bootload/booter288/rainbow-os//third_party -I/root/so/bootload/booter288/rainbow-os//third_party/uefi -MMD -MP -c /root/so/bootload/booter288/rainbow-os/boot/boot.cpp -o boot.cpp.o
gcc -m32 -mno-mmx -mno-sse -mno-avx -O2 -Wall -Wextra -Werror -ffreestanding -fbuiltin -fno-exceptions -fno-rtti -DKERNEL_ARCH=x86_64 -DKERNEL_X86_64 -I/root/so/bootload/booter288/rainbow-os/boot/ -I/root/so/bootload/booter288/rainbow-os/ -I/root/so/bootload/booter288/rainbow-os//third_party -I/root/so/bootload/booter288/rainbow-os//third_party/uefi -MMD -MP -c /root/so/bootload/booter288/rainbow-os/boot/elfloader.cpp -o elfloader.cpp.o
gcc -m32 -mno-mmx -mno-sse -mno-avx -O2 -Wall -Wextra -Werror -ffreestanding -fbuiltin -fno-exceptions -fno-rtti -DKERNEL_ARCH=x86_64 -DKERNEL_X86_64 -I/root/so/bootload/booter288/rainbow-os/boot/ -I/root/so/bootload/booter288/rainbow-os/ -I/root/so/bootload/booter288/rainbow-os//third_party -I/root/so/bootload/booter288/rainbow-os//third_party/uefi -MMD -MP -c /root/so/bootload/booter288/rainbow-os/boot/memory.cpp -o memory.cpp.o
gcc -m32 -mno-mmx -mno-sse -mno-avx -O2 -Wall -Wextra -Werror -ffreestanding -fbuiltin -fno-exceptions -fno-rtti -DKERNEL_ARCH=x86_64 -DKERNEL_X86_64 -I/root/so/bootload/booter288/rainbow-os/boot/ -I/root/so/bootload/booter288/rainbow-os/ -I/root/so/bootload/booter288/rainbow-os//third_party -I/root/so/bootload/booter288/rainbow-os//third_party/uefi -MMD -MP -c /root/so/bootload/booter288/rainbow-os/boot/arch/x86/vmm.cpp -o arch/x86/vmm.cpp.o
gcc -m32 -mno-mmx -mno-sse -mno-avx -DKERNEL_ARCH=x86_64 -DKERNEL_X86_64 -I/root/so/bootload/booter288/rainbow-os/boot/ -I/root/so/bootload/booter288/rainbow-os/ -I/root/so/bootload/booter288/rainbow-os//third_party -I/root/so/bootload/booter288/rainbow-os//third_party/uefi -MMD -MP -c /root/so/bootload/booter288/rainbow-os/boot/machine/bios/entry.S -o machine/bios/entry.S.o
gcc -m32 -mno-mmx -mno-sse -mno-avx -O2 -Wall -Wextra -Werror -ffreestanding -fbuiltin -fno-exceptions -fno-rtti -DKERNEL_ARCH=x86_64 -DKERNEL_X86_64 -I/root/so/bootload/booter288/rainbow-os/boot/ -I/root/so/bootload/booter288/rainbow-os/ -I/root/so/bootload/booter288/rainbow-os//third_party -I/root/so/bootload/booter288/rainbow-os//third_party/uefi -MMD -MP -c /root/so/bootload/booter288/rainbow-os/boot/machine/bios/multiboot.cpp -o machine/bios/multiboot.cpp.o
gcc -m32 -mno-mmx -mno-sse -mno-avx -O2 -Wall -Wextra -Werror -ffreestanding -fbuiltin -fno-exceptions -fno-rtti -DKERNEL_ARCH=x86_64 -DKERNEL_X86_64 -I/root/so/bootload/booter288/rainbow-os/boot/ -I/root/so/bootload/booter288/rainbow-os/ -I/root/so/bootload/booter288/rainbow-os//third_party -I/root/so/bootload/booter288/rainbow-os//third_party/uefi -MMD -MP -c /root/so/bootload/booter288/rainbow-os/graphics/graphicsconsole.cpp -o graphics/graphicsconsole.cpp.o
gcc -m32 -mno-mmx -mno-sse -mno-avx -O2 -Wall -Wextra -Werror -ffreestanding -fbuiltin -fno-exceptions -fno-rtti -DKERNEL_ARCH=x86_64 -DKERNEL_X86_64 -I/root/so/bootload/booter288/rainbow-os/boot/ -I/root/so/bootload/booter288/rainbow-os/ -I/root/so/bootload/booter288/rainbow-os//third_party -I/root/so/bootload/booter288/rainbow-os//third_party/uefi -MMD -MP -c /root/so/bootload/booter288/rainbow-os/graphics/pixels.cpp -o graphics/pixels.cpp.o
gcc -m32 -mno-mmx -mno-sse -mno-avx -O2 -Wall -Wextra -Werror -ffreestanding -fbuiltin -fno-exceptions -fno-rtti -DKERNEL_ARCH=x86_64 -DKERNEL_X86_64 -I/root/so/bootload/booter288/rainbow-os/boot/ -I/root/so/bootload/booter288/rainbow-os/ -I/root/so/bootload/booter288/rainbow-os//third_party -I/root/so/bootload/booter288/rainbow-os//third_party/uefi -MMD -MP -c /root/so/bootload/booter288/rainbow-os/graphics/vgafont.cpp -o graphics/vgafont.cpp.o
gcc -m32 -mno-mmx -mno-sse -mno-avx -O2 -Wall -Wextra -Werror -ffreestanding -fbuiltin -fno-exceptions -fno-rtti -DKERNEL_ARCH=x86_64 -DKERNEL_X86_64 -I/root/so/bootload/booter288/rainbow-os/boot/ -I/root/so/bootload/booter288/rainbow-os/ -I/root/so/bootload/booter288/rainbow-os//third_party -I/root/so/bootload/booter288/rainbow-os//third_party/uefi -MMD -MP -c /root/so/bootload/booter288/rainbow-os/metal/crt.cpp -o metal/crt.cpp.o
gcc -m32 -mno-mmx -mno-sse -mno-avx -O2 -Wall -Wextra -Werror -ffreestanding -fbuiltin -fno-exceptions -fno-rtti -DKERNEL_ARCH=x86_64 -DKERNEL_X86_64 -I/root/so/bootload/booter288/rainbow-os/boot/ -I/root/so/bootload/booter288/rainbow-os/ -I/root/so/bootload/booter288/rainbow-os//third_party -I/root/so/bootload/booter288/rainbow-os//third_party/uefi -MMD -MP -c /root/so/bootload/booter288/rainbow-os/metal/log.cpp -o metal/log.cpp.o
gcc -m32 -mno-mmx -mno-sse -mno-avx -O2 -Wall -Wextra -Werror -ffreestanding -fbuiltin -fno-exceptions -fno-rtti -DKERNEL_ARCH=x86_64 -DKERNEL_X86_64 -I/root/so/bootload/booter288/rainbow-os/boot/ -I/root/so/bootload/booter288/rainbow-os/ -I/root/so/bootload/booter288/rainbow-os//third_party -I/root/so/bootload/booter288/rainbow-os//third_party/uefi -MMD -MP -c /root/so/bootload/booter288/rainbow-os/metal/x86/cpuid.cpp -o metal/x86/cpuid.cpp.o
ld -nostdlib --warn-common --no-undefined --fatal-warnings -z max-page-size=0x1000 -T /root/so/bootload/booter288/rainbow-os/boot//machine/bios/bios.lds ./crt.cpp.o ./boot.cpp.o ./elfloader.cpp.o ./memory.cpp.o arch/x86/vmm.cpp.o machine/bios/entry.S.o machine/bios/multiboot.cpp.o graphics/graphicsconsole.cpp.o graphics/pixels.cpp.o graphics/vgafont.cpp.o metal/crt.cpp.o metal/log.cpp.o metal/x86/cpuid.cpp.o /opt/cross710/lib/gcc/i686-elf/7.1.0/libgcc.a -o bootloader
make[1]: Leaving directory '/root/so/bootload/booter288/rainbow-os/build/x86_64/boot'
mkdir -p /root/so/bootload/booter288/rainbow-os//build/x86_64/kernel && cd /root/so/bootload/booter288/rainbow-os//build/x86_64/kernel && make -f /root/so/bootload/booter288/rainbow-os//kernel/Makefile
make[1]: Entering directory '/root/so/bootload/booter288/rainbow-os/build/x86_64/kernel'
Adding module '.' at '/root/so/bootload/booter288/rainbow-os/kernel/'
Adding module 'x86' at '/root/so/bootload/booter288/rainbow-os/kernel/x86/'
Adding module 'graphics' at '/root/so/bootload/booter288/rainbow-os/graphics/'
Adding module 'metal' at '/root/so/bootload/booter288/rainbow-os/metal/'
gcc -mno-mmx -mno-sse -mno-avx -mno-red-zone -mcmodel=kernel -mno-mmx -mno-sse -O2 -Wall -Wextra -Werror -ffreestanding -fbuiltin -fno-exceptions -fno-rtti -DARCH=x86_64 -I/root/so/bootload/booter288/rainbow-os/kernel/ -I/root/so/bootload/booter288/rainbow-os/ -MMD -MP -c /root/so/bootload/booter288/rainbow-os/kernel/console.cpp -o console.cpp.o
/root/so/bootload/booter288/rainbow-os/kernel/console.cpp:1:0: error: code model kernel does not support PIC mode
 /*

/root/so/bootload/booter288/rainbow-os//mk/rules.mk:31: recipe for target 'console.cpp.o' failed
make[1]: *** [console.cpp.o] Error 1
make[1]: Leaving directory '/root/so/bootload/booter288/rainbow-os/build/x86_64/kernel'
Makefile:66: recipe for target 'kernel' failed
make: *** [kernel] Error 2

Of note is the error error: code model kernel does not support PIC modeMy guess is that my 64-bit GCC on Debian produces position independent code by default and that somehow conflicts with your builds of the code.I would imagine then that you are using a GCC that has a different default. That is the type of reason why cross compilers are suggested. I tried a build from scratch the 32-bit version and it builds without a hitch and runs as expected in QEMU and BOCHs which is what you found.

I then decided to use my x86_64-elf-gcc 7.1.0 cross compiler (it has nomulitlib support) and I encountered a series of errors because functions were needed out of libgcc but the libgcc.a your make facility found was the 64-bit one (which is correct) but there was no 32-bit version. I was using make MACHINE=bios CROSS_COMPILE=x86_64-elf- run-bochs. I discovered that your make files has LIBGCC set, but doesn't work well for cross compiling. As a hack I modified the Makefile so that I was using libgcc.a from my i686-elf-gcc 7.1.0 cross compiler. When I did that I was able to use make MACHINE=bios CROSS_COMPILE=x86_64-elf- run-bochs. I was able to run it in Bochs, but it does fail on QEMU in a way that appears consistent with what you are seeing. When I get some time this evening I'll see what I can discover about the failure itself.I just have some errands to run first.

Some day I may try to see if it is possible to build a GCC cross compiler with multilib support. It may work, but I have never tried it.