Help testing Bochs VMX and SVM support

[stlw]

Looks like you've just accidentally missed that line (cpu/vmexit.cc +324), as in other lines you've replaced magic numbers with BX_VMX_* .

Thanks, I added missed MSR fix line into the code.
Did you have a chance to try SVM more ?

Thanks,
Stanislav

[Nable]

Yes, this fix for SVM helped: guest starts booting. But not fully.

Guest linux printed "Probing EDD. edd=off to disable", then did some more boot things and then the whole bochs suddenly rebooted:

Code: Select all

...
05633849326e[CPU0 ] SVM VMEXIT: I/O port 0x03d5
05633854944e[CPU0 ] SVM VMEXIT: I/O port 0x03d4
05633860545e[CPU0 ] SVM VMEXIT: I/O port 0x03d5
05633866192e[CPU0 ] SVM VMEXIT: I/O port 0x03cc
05633871774e[CPU0 ] SVM VMEXIT: I/O port 0x03d4
05633877371e[CPU0 ] SVM VMEXIT: I/O port 0x03d4
05633885646e[CPU0 ] SVM VMEXIT: I/O port 0x03d4
05633891249e[CPU0 ] SVM VMEXIT: I/O port 0x03d5
05633896866e[CPU0 ] SVM VMEXIT: I/O port 0x03d4
05633902468e[CPU0 ] SVM VMEXIT: I/O port 0x03d5
05633909842e[CPU0 ] SVM VMEXIT: I/O port 0x0070
05633915419e[CPU0 ] SVM VMEXIT: I/O port 0x00a1
05633920867e[CPU0 ] SVM VMEXIT: I/O port 0x0021
05640000034i[XGUI ] charmap update. Font Height is 14
05837405993e[CPU0 ] interrupt(): vector must be within IDT table limits, IDT.limit = 0x0
05837405993e[CPU0 ] interrupt(): vector must be within IDT table limits, IDT.limit = 0x0
05837405993i[CPU0 ] CPU is in protected mode (active)
05837405993i[CPU0 ] CS.mode = 16 bit
05837405993i[CPU0 ] SS.mode = 16 bit
05837405993i[CPU0 ] EFER   = 0x00001901
05837405993i[CPU0 ] | EAX=00000901  EBX=10000563  ECX=c0000080  EDX=00000000
05837405993i[CPU0 ] | ESP=01e45040  EBP=00000000  ESI=00093770  EDI=efd3fbff
05837405993i[CPU0 ] | IOPL=0 id vip vif ac vm RF nt of df if tf sf zf af PF cf
05837405993i[CPU0 ] | SEG sltr(index|ti|rpl)     base    limit G D
05837405993i[CPU0 ] |  CS:0010( 0002| 0|  0) 00000000 ffffffff 1 0
05837405993i[CPU0 ] |  DS:0000( 0000| 0|  0) 00000000 00000000 0 0
05837405993i[CPU0 ] |  SS:0000( 0000| 0|  0) 00000000 00000000 0 0
05837405993i[CPU0 ] |  ES:0000( 0000| 0|  0) 00000000 00000000 0 0
05837405993i[CPU0 ] |  FS:0000( 0000| 0|  0) 00000000 00000000 0 0
05837405993i[CPU0 ] |  GS:0000( 0000| 0|  0) 00000000 00000000 0 0
05837405993i[CPU0 ] | EIP=81000142 (81000142)
05837405993i[CPU0 ] | CR0=0x80000011 CR2=0x81000142
05837405993i[CPU0 ] | CR3=0x01001000 CR4=0x000000a0
05837405993i[CPU0 ] 0xffffffff81000142: (instruction unavailable) page not present
05837405993e[CPU0 ] exception(): 3rd (13) exception with no resolution, shutdown status is 00h, resetting
05837405993i[SYS  ] bx_pc_system_c::Reset(HARDWARE) called

Last words of hypervisor were:

Code: Select all

Booting from 0000:7c00
EFER Read HI=0 LO=0
MSR write for msr 0xc0000080
EFER Write HI=0 LO=100
EFER Read HI=0 LO=100
MSR write for msr 0xc0000080
EFER Write HI=0 LO=901

Is it about IDT.limit caching (like it were for cpl)?

[stlw]

Yes, this fix for SVM helped: guest starts booting. But not fully.

Guest linux printed "Probing EDD. edd=off to disable", then did some more boot things and then the whole bochs suddenly rebooted:

Is it about IDT.limit caching (like it were for cpl)?

I don't think so. The IDTR has only two fields (base and limit) and they both read on VMENTER and saved on VMEXIT.

I had typo-bug in SVM code related to IDTR limit a while ago but it already fixed in SVN. The fix was SVN revision 10920. Which version are you running ?
Anyway, I expect that nobody changes the IDTR.LIMIT too often so you should see that on VMRUN you put valid value into it and on VMEXIT you still have it there.
It is possible to check with hypervisor debug print and also with Bochs debug print as well.

Anyway, I don't expect it to become zero ...

Stanislav

[Nable]

I use today's svn, so looks like that's not your old bug.

Now it's really strange, although, there were many bugs in Palacios that (when suddenly revealed) caused me to wonder "How it were working before?"
Ok, i'll try to debug it on wednesday.

[Nable]

Tested this stuff today - boots on real AMD but fails in a same way in bochs.

One more thing: according to reg values it seems to me that guest fails but bochs resets "host" cpus. Am I understanding smth wrong?

[stlw]

Tested this stuff today - boots on real AMD but fails in a same way in bochs.

One more thing: according to reg values it seems to me that guest fails but bochs resets "host" cpus. Am I understanding smth wrong?

So could you try to debug it a little?
I mean adding of Bochs print + hypervisor print about IDTR.LIMIT on every VMRUN and every VMEXIT.
It is strange to me how IDTR.LIMIT can become 0.

For "host" cpu reset - it is just Bochs artefact. The guest got tripple fault (because of IDTR.LIMIT=0 it cannot deliver any interrupt) - the real machine would go to shutdown state.
If hypervisor has intercept for shutdown it will catch it and kill the guest properly.
Bochs have option in .bochsrc to reset CPU in case of tripple fault and looks like it is set in your configuration.
If you clear that option - the result will be the shutdown (have no idea how can you exit it anyway) or VMEXIT if hypervisor intercept the shutdown.

Stanislav

[Nable]

So could you try to debug it a little?

Of course i'll try, as soon as i can. Today it was a somehow bad day for me (for example, i've wasted some hours due to strange bugs caused by degraded usb stick), so i hadn't much progress.

It is strange to me how IDTR.LIMIT can become 0.

It revealed to be simple, look at the EIP: guest has just entered long mode (mov cr0, eax is somewhere before 0x800000F0 (sorry, i don't have the disassemble of this particular kernel, only a bit older), then guest does far jump via push-push-retf and jumps to somewhere about 0x80000100, then it setups some additional things (such as better page tables instead of initial identity mapped) so guest interrupts are disabled and guest hasn't yet setup IDTR. Although, under bochs some fault/trap occures (page fault?), so we have this sudden interrupt. Today i've got and built some nice debugging guest pack (ready kernel + initrd image with busybox), so as soon as i feel enough better, i'll try to find what exact instruction causes this fault.

[stlw]

So could you try to debug it a little?

Of course i'll try, as soon as i can. Today it was a somehow bad day for me (for example, i've wasted some hours due to strange bugs caused by degraded usb stick), so i hadn't much progress.

It is strange to me how IDTR.LIMIT can become 0.

It revealed to be simple, look at the EIP: guest has just entered long mode (mov cr0, eax is somewhere before 0x800000F0 (sorry, i don't have the disassemble of this particular kernel, only a bit older), then guest does far jump via push-push-retf and jumps to somewhere about 0x80000100, then it setups some additional things (such as better page tables instead of initial identity mapped) so guest interrupts are disabled and guest hasn't yet setup IDTR. Although, under bochs some fault/trap occures (page fault?), so we have this sudden interrupt. Today i've got and built some nice debugging guest pack (ready kernel + initrd image with busybox), so as soon as i feel enough better, i'll try to find what exact instruction causes this fault.

Bochs has debug prints for every faults it can have. Just enable "debug: ignore, cpu0=report" and you will see all possible prints including reasons for all faults.

Stanislav

[Nable]

It becomes even more strange now.
Verbose bochs log:

Code: Select all

04392961890d[CPU0 ] RDTSC: ticks 0x00000001:05d74762
04392961921d[CPU0 ] RDTSC: ticks 0x00000001:05d74781
04392962052d[CPU0 ] VMSAVE VMCB ptr: 0x000000000469e000
04392962068d[CPU0 ] VMLOAD VMCB ptr: 0x0000000006b52000
04392962068d[CPU0 ] VMRUN VMCB ptr: 0x0000000006b52000
04392962068d[CPU0 ] VMRUN: Starting Nested Paging Mode !
04392962068d[CPU0 ] protected mode activated
04392962073d[CPU0 ] page walk for address 0x0000000081000142
04392962073d[CPU0 ] PAE PDPTE entry not present !
04392962073d[CPU0 ] page fault for address 0000000081000142 @ 0000000081000142
04392962073d[CPU0 ] exception(0x0e): error_code=0010
04392962073d[CPU0 ] interrupt(): vector = 0e, TYPE = 3, EXT = 1
04392962073e[CPU0 ] interrupt(): vector must be within IDT table limits, IDT.limit = 0x0
04392962073d[CPU0 ] exception(0x0d): error_code=0072
04392962073d[CPU0 ] exception(0x08): error_code=0000
04392962073d[CPU0 ] interrupt(): vector = 08, TYPE = 3, EXT = 1
04392962073e[CPU0 ] interrupt(): vector must be within IDT table limits, IDT.limit = 0x0
04392962073d[CPU0 ] exception(0x0d): error_code=0042
04392962073i[CPU0 ] CPU is in protected mode (active)
04392962073i[CPU0 ] CS.mode = 16 bit
04392962073i[CPU0 ] SS.mode = 16 bit
04392962073i[CPU0 ] EFER   = 0x00001901
04392962073i[CPU0 ] | EAX=00000901  EBX=10000563  ECX=c0000080  EDX=00000000
04392962073i[CPU0 ] | ESP=025769c0  EBP=00000000  ESI=00093ae0  EDI=efd3fbff
04392962073i[CPU0 ] | IOPL=0 id vip vif ac vm RF nt of df if tf sf zf af PF cf
04392962073i[CPU0 ] | SEG sltr(index|ti|rpl)     base    limit G D
04392962073i[CPU0 ] |  CS:0010( 0002| 0|  0) 00000000 ffffffff 1 0
04392962073i[CPU0 ] |  DS:0000( 0000| 0|  0) 00000000 00000000 0 0
04392962073i[CPU0 ] |  SS:0000( 0000| 0|  0) 00000000 00000000 0 0
04392962073i[CPU0 ] |  ES:0000( 0000| 0|  0) 00000000 00000000 0 0
04392962073i[CPU0 ] |  FS:0000( 0000| 0|  0) 00000000 00000000 0 0
04392962073i[CPU0 ] |  GS:0000( 0000| 0|  0) 00000000 00000000 0 0
04392962073i[CPU0 ] | EIP=81000142 (81000142)
04392962073i[CPU0 ] | CR0=0x80000011 CR2=0x81000142
04392962073i[CPU0 ] | CR3=0x01a05000 CR4=0x000000a0
04392962073i[CPU0 ] 0xffffffff81000142: (instruction unavailable) page not present
04392962073e[CPU0 ] exception(): 3rd (13) exception with no resolution, shutdown status is 00h, resetting
04392962073i[SYS  ] bx_pc_system_c::Reset(HARDWARE) called
04392962073i[CPU0 ] cpu hardware reset

Here is the disassemble:

Code: Select all

ffffffff81000100 <secondary_startup_64>:
ffffffff81000100:       b8 a0 00 00 00          mov    eax,0xa0
ffffffff81000105:       0f 22 e0                mov    cr4,rax
ffffffff81000108:       48 c7 c0 00 50 a0 01    mov    rax,0x1a05000
ffffffff8100010f:       48 03 05 fa ce a0 00    add    rax,QWORD PTR [rip+0xa0cefa]        # ffffffff81a0d010 <phys_base>
ffffffff81000116:       0f 22 d8                mov    cr3,rax
ffffffff81000119:       48 c7 c0 22 01 00 81    mov    rax,0xffffffff81000122
ffffffff81000120:       ff e0                   jmp    rax
ffffffff81000122:       b8 01 00 00 80          mov    eax,0x80000001
ffffffff81000127:       0f a2                   cpuid  
ffffffff81000129:       89 d7                   mov    edi,edx
ffffffff8100012b:       b9 80 00 00 c0          mov    ecx,0xc0000080
ffffffff81000130:       0f 32                   rdmsr  
ffffffff81000132:       0f ba e8 00             bts    eax,0x0
ffffffff81000136:       0f ba e7 14             bt     edi,0x14
ffffffff8100013a:       73 04                   jae    ffffffff81000140 <secondary_startup_64+0x40>
ffffffff8100013c:       0f ba e8 0b             bts    eax,0xb
ffffffff81000140:       0f 30                   wrmsr                                      # after returning from vmexit caused by this wrmsr
ffffffff81000142:       b8 33 00 05 80          mov    eax,0x80050033                      # <<-- we fail here (instruction is not executed, we have sudden #PF)
ffffffff81000147:       0f 22 c0                mov    cr0,rax
ffffffff8100014a:       48 8b 25 4f 78 aa 00    mov    rsp,QWORD PTR [rip+0xaa784f]        # ffffffff81aa79a0 <stack_start>
ffffffff81000151:       6a 00                   push   0x0
ffffffff81000153:       9d                      popf   
ffffffff81000154:       0f 01 15 a5 ce a0 00    lgdt   [rip+0xa0cea5]        # ffffffff81a0d000 <early_gdt_descr>

There was an idea that it's because using of NX by guest, but masking of NX feature in cpuid didn't help and lead to even more strange result:

Code: Select all

04848776749e[CPU0 ] interrupt(): vector must be within IDT table limits, IDT.limit = 0x0
04848776749e[CPU0 ] interrupt(): vector must be within IDT table limits, IDT.limit = 0x0
04848776749i[CPU0 ] CPU is in protected mode (active)
04848776749i[CPU0 ] CS.mode = 16 bit
04848776749i[CPU0 ] SS.mode = 16 bit
04848776749i[CPU0 ] EFER   = 0x00001101                  # << where has EFER.LMA gone?!
04848776749i[CPU0 ] | EAX=00000101  EBX=10000563  ECX=c0000080  EDX=00000000
04848776749i[CPU0 ] | ESP=025769c0  EBP=00000000  ESI=00093ae0  EDI=efc3fbff
04848776749i[CPU0 ] | IOPL=0 id vip vif ac vm RF nt of df if tf sf zf af PF cf
04848776749i[CPU0 ] | SEG sltr(index|ti|rpl)     base    limit G D
04848776749i[CPU0 ] |  CS:0010( 0002| 0|  0) 00000000 ffffffff 1 0
04848776749i[CPU0 ] |  DS:0000( 0000| 0|  0) 00000000 00000000 0 0
04848776749i[CPU0 ] |  SS:0000( 0000| 0|  0) 00000000 00000000 0 0
04848776749i[CPU0 ] |  ES:0000( 0000| 0|  0) 00000000 00000000 0 0
04848776749i[CPU0 ] |  FS:0000( 0000| 0|  0) 00000000 00000000 0 0
04848776749i[CPU0 ] |  GS:0000( 0000| 0|  0) 00000000 00000000 0 0
04848776749i[CPU0 ] | EIP=81000142 (81000142)        # same RIP
04848776749i[CPU0 ] | CR0=0x80000011 CR2=0x81000142
04848776749i[CPU0 ] | CR3=0x01a05000 CR4=0x000000a0
04848776749i[CPU0 ] 0xffffffff81000142: (instruction unavailable) page not present
04848776749e[CPU0 ] exception(): 3rd (13) exception with no resolution, shutdown status is 00h, resetting
04848776749i[SYS  ] bx_pc_system_c::Reset(HARDWARE) called
04848776749i[CPU0 ] cpu hardware reset

[stlw]

Thank you for investing time in it. Might I will need to take your disk image and try to look on it at home because things getting too weird now.

I am looking on your first dump. As I understand your guest is in long mode. Hopefully host is long mode as well - otherwise I might have a guess why it doesn't work
At least instruction that failed has long physical address 04392962073i[CPU0 ] 0xffffffff81000142: (instruction unavailable) page not present.
And EFER has long mode enabled.

But in other hand:
1. CPU is not in long mode (CPU is in protected mode (active))
2. It does a page walk for short 32-bit address (hi part of address was dropped). No surpise it crashing.
page fault for address 0000000081000142 @ 0000000081000142
3. The fault that happens possible ONLY in legacy PAE mode, not in long mode - so long mode had gone.

Sounds very weird. Might be it is some kind of memory overrun for cpu_mode variable in the BX_CPU_C class.
Could you try with latest SVN as well ?
Other wise it might require some kind of prints for current cpu mode inside the VMRUN function.
The CPU mode should be already settled after call to handleCpuContextChange() which calls to handleCpuModeChange().

In your case it prints "04392962068d[CPU0 ] protected mode activated".
I expect it to be "long mode activated".

So smth was corrupted in EFER ... or I don't have any guess because EFER is correct at the crash point and has LMA/LME set both.

Stanislav

[Nable]

Im always using last SVN

Code: Select all

$ svn info
Path: .
URL: https://bochs.svn.sourceforge.net/svnroot/bochs/trunk/bochs
Repository Root: https://bochs.svn.sourceforge.net/svnroot/bochs
Repository UUID: 5c200c8f-f062-490a-aca5-964d1c9e8edb
Revision: 11116
Node Kind: directory
Schedule: normal
Last Changed Author: sshwarts
Last Changed Rev: 11116
Last Changed Date: 2012-03-28 01:30:34 +0400 (Wed, 28 Mar 2012)

Here are my conf flags:

Code: Select all

CFLAGS="-march=nocona -O2 -fomit-frame-pointer" CPPFLAGS="-march=nocona -O2 -fomit-frame-pointer" ./configure --prefix='/usr' --enable-plugins --enable-a20-pin --enable-x86-64 --enable-smp --enable-long-phy-address --disable-ne2000 --enable-fast-function-calls --enable-disasm --enable-all-optimizations --enable-logging --enable-fpu --enable-vmx=2 --enable-svm --enable-misaligned-sse --enable-monitor-mwait --enable-pci --without-wx --with-x --with-x11 --disable-iodebug --disable-alignment-check --disable-usb
X_CFLAGS="-march=nocona -O2 -fomit-frame-pointer" make -j2

Just to pre-answer some questions: lack of --disable-alignment-check, disabling of -march and even -O0 -fno-omit-frame-pointer doesn't help, it smth with code, not code generation.

Here are files from my usb stick: http://rghost.net/37277382 , you can put them to any media with grub1 in boot sector, i haven't tested it with grub2

[stlw]

Here are files from my usb stick: http://rghost.net/37277382 , you can put them to any media with grub1 in boot sector, i haven't tested it with grub2

I am sorry, I am not OS developer, I never tried to create boot images from individual files.
Can you send me an instructrions how to do so or compile any image understandable by Bochs for me ?
Please also send me your .bochsrc file.

thanks
Stanislav

[Nable]

I've also never tried making images with grub, i'm just using usb stick with installed (some long time ago) grub and put files here.
So, i'll just give you ready image with some pieces of advice about modifing it (when you have ready image, you obviously need no usb sticks to modify it).

Here is sanitized (i've removed other files and filled free space with zeroes) raw image of this stick:
http://rghost.net/37408759

If you want to do smth with files on it, you can use (under linux)

Code: Select all

mount -t vfat -o loop,offset=16384 <path to image file> <mount point>

or ImDisk (under windows).

Here is my .bochsrc (don't forget to change paths to image and serial output) :

Code: Select all

# configuration file generated by Bochs
config_interface: textconfig
display_library: x
memory: host=1024, guest=3072
romimage: file="/usr/share/bochs/BIOS-bochs-latest"
vgaromimage: file="/usr/share/bochs/VGABIOS-lgpl-latest"
boot: disk
floppy_bootsig_check: disabled=0
# no floppya
# no floppyb
ata0: enabled=1, ioaddr1=0x1f0, ioaddr2=0x3f0, irq=14
ata0-master: type=disk, mode=flat, translation=lba, path="/tmp/flash", cylinders=5778, heads=11, spt=63, biosdetect=auto, model="Generic 1234"
ata1: enabled=0
ata2: enabled=0
ata3: enabled=0
parport1: enabled=0
parport2: enabled=0
com1: enabled=1, mode=file, dev="/tmp/ttyS0_bochs"
com2: enabled=0
com3: enabled=0
com4: enabled=0
pci: enabled=1, chipset=i440fx
vga: extension=vbe, update_freq=5
cpu: count=1:2:1, ips=50000000, quantum=5, model=phenom_8650_toliman, reset_on_triple_fault=1, cpuid_limit_winnt=0, ignore_bad_msrs=1, mwait_is_nop=0, msrs="msrs.def"
print_timestamps: enabled=0
port_e9_hack: enabled=0
private_colormap: enabled=0
clock: sync=none, time0=local
# no cmosimage
# no loader
log: bochsout.txt
logprefix: %t%e%d
panic: action=ask
error: action=report
info: action=report
debug: action=ignore
keyboard_type: mf
keyboard_serial_delay: 250
keyboard_paste_delay: 100000
keyboard_mapping: enabled=0, map=
user_shortcut: keys=none
mouse: enabled=0, type=none, toggle=ctrl+mbutton

My way of compilation:

Code: Select all

CFLAGS="-march=nocona -O2 -fomit-frame-pointer" CPPFLAGS="-march=nocona -O2 -fomit-frame-pointer" ./configure --prefix='/usr' --enable-plugins --enable-a20-pin --enable-x86-64 --enable-smp --enable-long-phy-address --disable-ne2000 --enable-fast-function-calls --enable-disasm --enable-all-optimizations --enable-logging --enable-fpu --enable-vmx=2 --enable-svm --enable-misaligned-sse --enable-monitor-mwait --enable-pci --without-wx --with-x --with-x11 --disable-iodebug --disable-alignment-check --disable-usb

vim Makefile # here i remove string 'list=`cd iodev/usb && echo *.la`; for i in $$list; do $(LIBTOOL) --mode=install install iodev/usb/$$i $(DESTDIR)$(plugdir); done' because it fails when --disable-usb is used (because there are no .la files in iodev/usb)

X_CFLAGS="-march=nocona -O2 -fomit-frame-pointer" make -j2
sudo make install

I hope that it will help you..

[stlw]

Yes, it helped a lot !

I just check in the fix for that issue. It was not very legal guest behavior which is correctly handled by hardware but wasn't handled propely by Bochs SVM code.

The last wrmsr that did VMEXIT was actually write to EFER which tried to disable EFER.LMA bit withput touching EFER.LMA.
Normally any write to EFER should keep LMA bit unmodifed. Moreover according to Intel's most recent manual EFER.LMA is not a real bit, it is just alias to (EFER.LME & CR0.PG).

So in your case guest wrote 0x1101 to EFER and got VMEXIT. Palacios updated EFER in SVM guest state and tried to enter back with guest EFER value 0x1101 with CR0.PG=1
The LME is set but LMA is clear which is illegal. But because LMA is not the real bit HW creates it back from (EFER.LME & CR0.PG) so EFER becomes 0x1501 as expected and everything keeps running.

Now the guest boots properly.

Thanks,
Stanislav

[Nable]

sorry for offtop but looks like i have to write it here:
Stanislav, please, check your PM inbox.