QEMU: GP fault with SSE instructions when using KVM

[austanss]

I'm testing my OS with the following command:

Code: Select all

	qemu-system-x86_64 -bios res/OVMF.fd -cdrom img/microNET.fat -serial stdio -net none -m 512M -accel kvm -cpu Penryn

I wrote a scroll feature that works as expected, buffered, all the bells and whistles, and it works without KVM. When I turn KVM on, however, it faults.
This is worrisome. KVM is supposed to be more realistic than basic QEMU, and KVM disallows some memory mistakes, iirc.

Anyways, I push on.

Using noshutdown/noreboot, I can read the registers dump from the kernel panic screen:

Code: Select all

    FATAL ERROR: CPU EXCEPTION d -/- ERROR CODE 
    DS: 10
    RDI: 2b | RSI: 1c12b071 | RBP: f | RSP: 10dec8
    RBX: 114080 | RDX: 1c12b013 | RCX: 1c12b021 | RAX: 32
    RIP: 10287a | CS: 8
    RFLAGS: 10006

OK, a GP fault at 0x10287a.
I look for that particular instruction:

Code: Select all

  102873:	48 01 ce             	add    rsi,rcx
  102876:	66 0f ef c0          	pxor   xmm0,xmm0
  10287a:	66 0f 7f 01          	movdqa XMMWORD PTR [rcx],xmm0 ; <------- here
  10287e:	48 83 c1 10          	add    rcx,0x10
  102882:	48 39 f1             	cmp    rcx,rsi
  102885:	75 f3                	jne    10287a <_ZN8terminal5shiftEv+0xe8>

An SSE instruction...
It looks like a memcopy, but I'm unsure.

Anyway, I don't know if this is a general bug or a KVM issue, so please let me know what you think.

[Octocontrabass]

The "A" in "MOVDQA" stands for "aligned".

Code: Select all

RCX: 1c12b021

Code: Select all

movdqa XMMWORD PTR [rcx],xmm0

That address is not aligned. The compiler apparently thinks it should be aligned, so it's a bug in your code.

[austanss]

There isn't much going on here:

Code: Select all

void terminal::shift()
{
    for (size_t i = 0; i < VGA_WIDTH; i++)
    {
        text_buffer[i] = vga_entry(0, 0);
    }

    for (size_t i = 0; i < VGA_WIDTH; i++)
    {
        for (size_t i = 0; i < (VGA_HEIGHT * VGA_WIDTH); i++) {
            text_buffer[i] = text_buffer[i + 1]; // shift text gfx::buffer to the left
        }
    }
	row--;

	render_buffer();
}

[kzinti]

That's interesting. I only become aware of "-accel kvm" after reading your recent posts. So I decided to give this a shot.

It works just fine for my 32 bits kernel (ia32) but I also get a crash on an SSE instruction when running in 64 bits mode (x86_64). Mine look like this:

Code: Select all

ffffffff80044c20 <_vfprintf_r>:
ffffffff80044c20:	55                   	push   %rbp
ffffffff80044c21:	48 89 e5             	mov    %rsp,%rbp
ffffffff80044c24:	41 57                	push   %r15
ffffffff80044c26:	41 56                	push   %r14
ffffffff80044c28:	49 89 d6             	mov    %rdx,%r14
ffffffff80044c2b:	41 55                	push   %r13
ffffffff80044c2d:	41 54                	push   %r12
ffffffff80044c2f:	53                   	push   %rbx
ffffffff80044c30:	48 89 fb             	mov    %rdi,%rbx
ffffffff80044c33:	48 81 ec 98 04 00 00 	sub    $0x498,%rsp
ffffffff80044c3a:	48 89 bd e0 fb ff ff 	mov    %rdi,-0x420(%rbp)
ffffffff80044c41:	48 89 b5 e8 fb ff ff 	mov    %rsi,-0x418(%rbp)
ffffffff80044c48:	48 89 8d f8 fb ff ff 	mov    %rcx,-0x408(%rbp)
ffffffff80044c4f:	e8 dc 82 ff ff       	callq  ffffffff8003cf30 <_localeconv_r>
ffffffff80044c54:	48 8b 00             	mov    (%rax),%rax
ffffffff80044c57:	48 89 c7             	mov    %rax,%rdi
ffffffff80044c5a:	48 89 85 80 fb ff ff 	mov    %rax,-0x480(%rbp)
ffffffff80044c61:	e8 0a 89 fe ff       	callq  ffffffff8002d570 <strlen>
ffffffff80044c66:	66 0f ef c0          	pxor   %xmm0,%xmm0
ffffffff80044c6a:	48 89 85 90 fb ff ff 	mov    %rax,-0x470(%rbp)
ffffffff80044c71:	0f 29 85 30 fc ff ff 	movaps %xmm0,-0x3d0(%rbp)

It crashes on the movaps. I get a #GPF and it appears to be an alignment issue as well.

This is code from newlib... Damn.

[austanss]

So I'm not the only one!

[kzinti]

I get a crash on this one:

Code: Select all

    printf("    Start CPU: id = %d, apic = %d, enabled = %d, bootstrap = %d\n", cpu->id, cpu->apicId, cpu->enabled, cpu->bootstrap);

If I comment out that line, code works just fine (lot of printf()'s before and lot afterward).

I wonder if my Cpu object might not be properly aligned. Something like new/malloc() aligning on 8 bytes when it should align on 16 bytes. Digging some more...

[kzinti]

Nope that's not it, even this will crash:

Code: Select all

printf("123");

What is going on! lol...

[Octocontrabass]

There isn't much going on here:

Is text_buffer a pointer? Is it correctly aligned for its type?

I get a #GPF and it appears to be an alignment issue as well.

How do you align your stack?

[austanss]

text_buffer is a pointer to u16 integers.
my allocation log reports it starting at 0x1c12b003/

[kzinti]

How do you align your stack?

I think you might have found my issue... My crash is happening when starting a new thread. Looking at my thread setup code, I didn't do any stack alignment calculations there. So this must be the problem. I will investigate further and report back.

I am curious why this is not causing any problems on real hardware and only through KVM acceleration.

[kzinti]

Looks like my stack is 8 bytes aligned when I start a new thread. I believe it needs to be 16 bytes on x86_64. The main thread doesn't suffer from this as it starts page-aligned (4096 bytes).

[Octocontrabass]

text_buffer is a pointer to u16 integers.
my allocation log reports it starting at 0x1c12b003/

A pointer to 16-bit integers needs to be two-byte aligned. You need to change your allocator.

[kzinti]

Turns out I had two such bugs: one when doing the setup for a new thread and another one when starting up other processors.

I must say I am pretty happy to discover "-accel kvm", now my OS runs way faster!