OSDev.org

Link to code: https://github.com/io12/OS

My OS works fine when compiling with clang on -O0 and -O1, and also -O2 on gcc. All optimization levels above that cause an invalid opcode exception.

Register dump (clang):
eax=0x10000 ebx=0x0 ecx=0x8 edx=0x109000
esp=0x1088E0 ebp=0x8946 esi=0x10 edi=0x109000
cs=0x8 ds=0x10 ss=0x8946 es=0x10 fs=0x10 gs=0x10
eip=0x101803

0x101803 in malloc (from liballoc) in kernel.elf (clang):
0f 57 c0 xorps %xmm0,%xmm0

Register dump (gcc):
eax=0x1000 ebx=0x10B000 ecx=0x11B edx=0x11B
esp=0x10A934 ebp=0x10000 esi=0x8946 edi=0x10
cs=0x8 ds=0x10 ss=0x0 es=0x10 fs=0x10 gs=0x10
eip=0x101BFD

0x101bfd in malloc (from liballoc) in kernel.elf (gcc):
66 0f ef c0 pxor %xmm0,%xmm0

Xorps and pxor are not in the resulting kernel.elf with lower optimization levels. The area that the code fails on (I checked with gdb) is a line where a value is set to NULL. The problem fixed when I changed it to a memset call, but that caused the exception somewhere else.

Can someone give me a hint what the issue is?

The optimization seems to be using SSE instructions. Did you detect SSE beforehand? Did you enable it? Can you show us what CR0 and CR4 contain?

cr0: 0x60000011
cr4: 0x00000000

I didn't check for or enable SSE.

EDIT: I enabled SSE. Works now . Thanks.

I'd advise disabling sse optimisations so you won't do fxsave each time you get an interrupt.
If you are sure that your interrupt handler are not using SSE, you are safe.

I added -march=i386 to CFLAGS and both gcc and clang no longer create SSE instructions. I no longer have to enable SSE, which is good, because I don't really need it.

io12 wrote:I added -march=i386 to CFLAGS and both gcc and clang no longer create SSE instructions. I no longer have to enable SSE, which is good, because I don't really need it.

You really should use SSE for only certain things like memcpy, memset and other memory-related routines. They add a lot of performance.

Large memcpy's indicate you didn't do paging properly. In contrast, using FPU parts in kernel space requires that you have to save and restore the entire FPU context in interrupt handlers as well because the code might use those registers (but not always)