Page 1 of 1
QEMU: GP fault with SSE instructions when using KVM
Posted: Thu Jan 28, 2021 7:44 pm
by austanss
I'm testing my OS with the following command:
Code: Select all
qemu-system-x86_64 -bios res/OVMF.fd -cdrom img/microNET.fat -serial stdio -net none -m 512M -accel kvm -cpu Penryn
I wrote a scroll feature that works as expected, buffered, all the bells and whistles, and it works without KVM. When I turn KVM on, however, it faults.
This is worrisome. KVM is supposed to be more realistic than basic QEMU, and KVM disallows some memory mistakes, iirc.
Anyways, I push on.
Using noshutdown/noreboot, I can read the registers dump from the kernel panic screen:
Code: Select all
FATAL ERROR: CPU EXCEPTION d -/- ERROR CODE
DS: 10
RDI: 2b | RSI: 1c12b071 | RBP: f | RSP: 10dec8
RBX: 114080 | RDX: 1c12b013 | RCX: 1c12b021 | RAX: 32
RIP: 10287a | CS: 8
RFLAGS: 10006
OK, a GP fault at 0x10287a.
I look for that particular instruction:
Code: Select all
102873: 48 01 ce add rsi,rcx
102876: 66 0f ef c0 pxor xmm0,xmm0
10287a: 66 0f 7f 01 movdqa XMMWORD PTR [rcx],xmm0 ; <------- here
10287e: 48 83 c1 10 add rcx,0x10
102882: 48 39 f1 cmp rcx,rsi
102885: 75 f3 jne 10287a <_ZN8terminal5shiftEv+0xe8>
An SSE instruction...
It looks like a memcopy, but I'm unsure.
Anyway, I don't know if this is a general bug or a KVM issue, so please let me know what you think.
Re: QEMU: GP fault with SSE instructions when using KVM
Posted: Thu Jan 28, 2021 8:02 pm
by Octocontrabass
The "A" in "MOVDQA" stands for "aligned".
That address is not aligned. The compiler apparently thinks it should be aligned, so it's a bug in your code.
Re: QEMU: GP fault with SSE instructions when using KVM
Posted: Thu Jan 28, 2021 8:05 pm
by austanss
There isn't much going on here:
Code: Select all
void terminal::shift()
{
for (size_t i = 0; i < VGA_WIDTH; i++)
{
text_buffer[i] = vga_entry(0, 0);
}
for (size_t i = 0; i < VGA_WIDTH; i++)
{
for (size_t i = 0; i < (VGA_HEIGHT * VGA_WIDTH); i++) {
text_buffer[i] = text_buffer[i + 1]; // shift text gfx::buffer to the left
}
}
row--;
render_buffer();
}
Re: QEMU: GP fault with SSE instructions when using KVM
Posted: Thu Jan 28, 2021 8:28 pm
by kzinti
That's interesting. I only become aware of "-accel kvm" after reading your recent posts. So I decided to give this a shot.
It works just fine for my 32 bits kernel (ia32) but I also get a crash on an SSE instruction when running in 64 bits mode (x86_64). Mine look like this:
Code: Select all
ffffffff80044c20 <_vfprintf_r>:
ffffffff80044c20: 55 push %rbp
ffffffff80044c21: 48 89 e5 mov %rsp,%rbp
ffffffff80044c24: 41 57 push %r15
ffffffff80044c26: 41 56 push %r14
ffffffff80044c28: 49 89 d6 mov %rdx,%r14
ffffffff80044c2b: 41 55 push %r13
ffffffff80044c2d: 41 54 push %r12
ffffffff80044c2f: 53 push %rbx
ffffffff80044c30: 48 89 fb mov %rdi,%rbx
ffffffff80044c33: 48 81 ec 98 04 00 00 sub $0x498,%rsp
ffffffff80044c3a: 48 89 bd e0 fb ff ff mov %rdi,-0x420(%rbp)
ffffffff80044c41: 48 89 b5 e8 fb ff ff mov %rsi,-0x418(%rbp)
ffffffff80044c48: 48 89 8d f8 fb ff ff mov %rcx,-0x408(%rbp)
ffffffff80044c4f: e8 dc 82 ff ff callq ffffffff8003cf30 <_localeconv_r>
ffffffff80044c54: 48 8b 00 mov (%rax),%rax
ffffffff80044c57: 48 89 c7 mov %rax,%rdi
ffffffff80044c5a: 48 89 85 80 fb ff ff mov %rax,-0x480(%rbp)
ffffffff80044c61: e8 0a 89 fe ff callq ffffffff8002d570 <strlen>
ffffffff80044c66: 66 0f ef c0 pxor %xmm0,%xmm0
ffffffff80044c6a: 48 89 85 90 fb ff ff mov %rax,-0x470(%rbp)
ffffffff80044c71: 0f 29 85 30 fc ff ff movaps %xmm0,-0x3d0(%rbp)
It crashes on the movaps. I get a #GPF and it appears to be an alignment issue as well.
This is code from newlib... Damn.
Re: QEMU: GP fault with SSE instructions when using KVM
Posted: Thu Jan 28, 2021 8:40 pm
by austanss
So I'm not the only one!
Re: QEMU: GP fault with SSE instructions when using KVM
Posted: Thu Jan 28, 2021 8:43 pm
by kzinti
I get a crash on this one:
Code: Select all
printf(" Start CPU: id = %d, apic = %d, enabled = %d, bootstrap = %d\n", cpu->id, cpu->apicId, cpu->enabled, cpu->bootstrap);
If I comment out that line, code works just fine (lot of printf()'s before and lot afterward).
I wonder if my Cpu object might not be properly aligned. Something like new/malloc() aligning on 8 bytes when it should align on 16 bytes. Digging some more...
Re: QEMU: GP fault with SSE instructions when using KVM
Posted: Thu Jan 28, 2021 8:45 pm
by kzinti
Nope that's not it, even this will crash:
What is going on! lol...
Re: QEMU: GP fault with SSE instructions when using KVM
Posted: Thu Jan 28, 2021 8:45 pm
by Octocontrabass
rizxt wrote:There isn't much going on here:
Is text_buffer a pointer? Is it correctly aligned for its type?
kzinti wrote:I get a #GPF and it appears to be an alignment issue as well.
How do you align your stack?
Re: QEMU: GP fault with SSE instructions when using KVM
Posted: Thu Jan 28, 2021 8:48 pm
by austanss
text_buffer is a pointer to u16 integers.
my allocation log reports it starting at 0x1c12b003/
Re: QEMU: GP fault with SSE instructions when using KVM
Posted: Thu Jan 28, 2021 8:49 pm
by kzinti
Octocontrabass wrote:How do you align your stack?
I think you might have found my issue... My crash is happening when starting a new thread. Looking at my thread setup code, I didn't do any stack alignment calculations there. So this must be the problem. I will investigate further and report back.
I am curious why this is not causing any problems on real hardware and only through KVM acceleration.
Re: QEMU: GP fault with SSE instructions when using KVM
Posted: Thu Jan 28, 2021 8:57 pm
by kzinti
Looks like my stack is 8 bytes aligned when I start a new thread. I believe it needs to be 16 bytes on x86_64. The main thread doesn't suffer from this as it starts page-aligned (4096 bytes).
Re: QEMU: GP fault with SSE instructions when using KVM
Posted: Thu Jan 28, 2021 9:07 pm
by Octocontrabass
rizxt wrote:text_buffer is a pointer to u16 integers.
my allocation log reports it starting at 0x1c12b003/
A pointer to 16-bit integers needs to be two-byte aligned. You need to change your allocator.
Re: QEMU: GP fault with SSE instructions when using KVM
Posted: Thu Jan 28, 2021 11:06 pm
by kzinti
Turns out I had two such bugs: one when doing the setup for a new thread and another one when starting up other processors.
I must say I am pretty happy to discover "-accel kvm", now my OS runs way faster!