OSDev.org

Posted: **Wed Oct 29, 2025 7:30 pm**

Hi all, I'm messing with some VMX code, and I wanted to test getting into a VMX operation. I have a page aligned memory location for this at 0x214000, which is set to zero with memset. I'm able to set a VMXON environment like so.

Code: Select all

	uint64_t region64 = (uint64_t)((size_t)(memory_location) & 0xFFFFFFFF);
	asm("vmxon (%0);" ::"m"((uint32_t)region64));
	printf("VMXON instruction...");	did_vm_operation_fail();

memory_location is a void* that points to 0x214000. did_vm_operation_fail() just checks EFLAGS and sees if the operation failed:

Code: Select all

void did_vm_operation_fail(){
	uint32_t flags = 0;
	asm("pushf; pop %%eax; mov %%eax, %0" :"=r"(flags));
	if(flags & (1<<0)) printf("VMFailInvalid.\n");
	else if(flags & (1<<6)) printf("VMFailValid.\n");
	else printf("VMSuccess");
}

This outputs VMSuccess. So now, the next logical step is to set up the VMCS region, and I can use VMPTRLD for this:

Code: Select all

	*(uint32_t*)memory_location = msr.bits.rev_ident;
	asm("vmptrld %0" ::"m"(region64));
	did_vm_operation_fail();

msr.bits.rev_ident is the revision identifier from the IA32_VMX_BASIC MSR.
However, this ends up giving me a #UD, which according to the SDM, this
should only happen either when:

The operand is a register
If I'm not in a VMX operation

But I should be in a VMX operation since VMXON seemingly succeeded. The only thing I can think of is the operand is a register.
The exception happens at 0x200b79, so...

Code: Select all

(qemu) x/1i 0x200b79
0x00200b79:  0f c7 75 f0              vmptrld  -0x10(%ebp)

Hm, if -0x10(%ebp) really was causing the #UD then the VMXON should've done it too, since

Code: Select all

(qemu) x/1i 0x200b4e
0x00200b4e:  f3 0f c7 75 f0           vmxon    -0x10(%ebp)

but it doesn't! What's happening?

Yes, CR0.NE, CR0.PG, CR0.PE, and CR4.VMXE are all set before all this stuff was done.

Posted: **Wed Oct 29, 2025 9:24 pm**

avcado wrote: ↑Wed Oct 29, 2025 7:30 pm
Code: Select all
	asm("vmxon (%0);" ::"m"((uint32_t)region64));

The assembler template has unnecessary parentheses. You're casting the input operand to the wrong type.

avcado wrote: ↑Wed Oct 29, 2025 7:30 pm
Code: Select all
	asm("pushf; pop %%eax; mov %%eax, %0" :"=r"(flags));

This inline assembly returns a bogus value because some other instruction has modified EFLAGS. You're clobbering EAX without informing the compiler.

You need to fix your inline assembly before you can fix anything else. Something like this should work:

Code: Select all

bool cf;
bool zf;
asm volatile( "vmxon %2" : "=@ccc"(cf), "=@ccz"(zf) : "m"(region64) : "cc" );

Posted: **Wed Oct 29, 2025 10:24 pm**

Octocontrabass wrote: ↑Wed Oct 29, 2025 9:24 pm You need to fix your inline assembly before you can fix anything else. Something like this should work:
Code: Select all
bool cf;
bool zf;
asm volatile( "vmxon %2" : "=@ccc"(cf), "=@ccz"(zf) : "m"(region64) : "cc" );

That seems like relatively new syntax. I certainly have never seen it. I'd suggest something like

Code: Select all

size_t flags;
asm volatile("vmxon %1; pushf; pop %0" : "=rm"(flags) : "m"(region64) : "cc");

And then hand the flags over to did_vm_operation_fail().

But yes, for these often-used registers like EFLAGS, it is really important to put the operation that reads them into the same assembler snippet as the operation that writes them. Another example would be the x87 status word that gets modified by every x87 instruction.

Posted: **Wed Oct 29, 2025 11:25 pm**

nullplan wrote: ↑Wed Oct 29, 2025 10:24 pmThat seems like relatively new syntax.

GCC 6.1 was released April 27, 2016.

Posted: **Thu Oct 30, 2025 3:57 pm**

Octocontrabass wrote: ↑Wed Oct 29, 2025 9:24 pm This inline assembly returns a bogus value because some other instruction has modified EFLAGS. You're clobbering EAX without informing the compiler.

You need to fix your inline assembly before you can fix anything else. Something like this should work:
Code: Select all
bool cf;
bool zf;
asm volatile( "vmxon %2" : "=@ccc"(cf), "=@ccz"(zf) : "m"(region64) : "cc" );

That was the issue -- VMXON was actually giving VMFailInvalid because I hadn't put the revision identifier inside the VMCS area. Now, I'm in a VMX operation, however I'm confused by something. I read here that I needed to do VMXON, then VMCLEAR, then VMPTRLD to properly set up the VMM environment. However, I also read on the SDM for both VMCLEAR and VMPTRLD that

Code: Select all

...
        ELSIF addr = VMXON pointer
            THEN VMfail(VMPTRLD/VMCLEAR with VMXON pointer);
            ELSE
...

How does that work? What parameter should I give to VMCLEAR/VMPTRLD so that the VMCS region is proper? I read that VMXON sets the current VMCS pointer to 0xFFFFFFFFFFFFFFFF

Code: Select all

...
                        IF rev[30:0] ≠ VMCS revision identifier supported by processor OR rev[31] = 1
                            THEN VMfailInvalid;
                            ELSE
                                current-VMCS pointer := FFFFFFFF_FFFFFFFFH;
                                enter VMX operation;
                                block INIT signals;
                                block and disable A20M;
...

So clearly I can't use the current VMCS pointer, but what am I supposed to do with VMCLEAR/VMPTRLD?

EDIT 7:38PM(UTC-6): I realize this kinda question was stupid. Instead what I did was just create a different memory area for the VMXON region and have another one for the VMCS. I feel kind of stupid not thinking of this, but, whatever, VMCLEAR works now and shows VMSuccess!

OSDev.org

[SOLVED] VMPTRLD causes a #UD.

[SOLVED] VMPTRLD causes a #UD.

Re: VMPTRLD causes a #UD.

Re: VMPTRLD causes a #UD.

Re: VMPTRLD causes a #UD.

Re: VMPTRLD causes a #UD.