Guest IP and EPT Intel VT-x

[CRoemheld]

Hello,

I am currently trying to run vmlinux (made by "make tinyconfig" from source repository) in my OS using Intel VT-x. However I still fail to understand how to appropiately set the guest code segment base and guest instruction pointer.
The vmlinux ELF is currently loaded into memory by GRUB as a module:

Code: Select all

menuentry "cr0S" {
	multiboot /boot/cr0S-bootstrap.bin
	module /boot/cr0S-kernel.bin "cr0S"
	module /boot/vmlinux.bin "vmlinux"
}

When entering the 64-bit kernel, I can read from the multiboot struct passed on to by my bootstrap setup and check the physical location of the vmlinux ELF. In my case, it is located at physical address 0x003C4000 with an offset of 0x00001000, meaning the ELF is effectively located at address 0x003C5000 in physical memory.

Now some other configurations are being made, and I ultimately initialize VMX inside the kernel. There I used the following settings for the guest code segment:

• - Code segment selector: 0xf000
  - Code segment base: 0xffff0000
  - Code segment limit: 0x0000ffff

(Source: https://elixir.bootlin.com/linux/v4.2/s ... mx.c#L4720, https://0xax.gitbooks.io/linux-insides/ ... rap-1.html)

However, I also use the Extended Page Table translation mechanism to resolve GPAs to HPAs. How do I correctly map the addresses so that I can set the guest instruction pointer to point to the vmlinux entry point? Because with a code segment base of 0xffff0000, I fail to understand how to choose an instruction pointer which points to an entry point of vmlinux located at 0x003C5000 in physical memory. Or how to map the addresses in the EPT to get the correct address.

I hope someone can explain and help me through my struggle.

[Octocontrabass]

- Code segment selector: 0xf000
- Code segment base: 0xffff0000
- Code segment limit: 0x0000ffff

Those are the values used by the CPU immediately after reset. It's correct if you're trying to run a BIOS ROM, and wrong for running a Linux kernel.

If you want to run a Linux kernel, you must set up the CPU registers the same way a Linux bootloader would.

[CRoemheld]

But if you want to run an OS in a VM, wouldn't you use the exact same settings as if you would when you start a host OS?
Well obviously you would need to set things as where you want to jump to execute you guest OS and such, but I can't find out how KVM does this or where in KVM the guest instruction pointer is set.
The only thing I found is where the guest instruction pointer is written to the VMCS, but not where the value is coming from:

https://elixir.bootlin.com/linux/v4.2/s ... mx.c#L8317

Would you by any chance know how to set the values to get vmlinux to work, dependent on where the vmlinux module is loaded in physical memory?

[Octocontrabass]

But if you want to run an OS in a VM, wouldn't you use the exact same settings as if you would when you start a host OS?

That depends. If your goal is to emulate a complete PC, including things like firmware ROMs, then you'll execute the firmware ROM and the firmware ROM will load and run the bootloader, then the bootloader will load and run the Linux kernel. (QEMU can do this.)

On the other hand, if you want to skip all of that and go directly to running the Linux kernel, you need to set up the virtual CPU's state the same way the bootloader would. (QEMU can do this too.)

Well obviously you would need to set things as where you want to jump to execute you guest OS and such, but I can't find out how KVM does this or where in KVM the guest instruction pointer is set.

The only place where KVM sets the guest instruction pointer is in vmx_vcpu_reset(), which sets all registers to their default values after a reset. Everything else is controlled by the software that's using KVM. (For example, QEMU.)

Would you by any chance know how to set the values to get vmlinux to work, dependent on where the vmlinux module is loaded in physical memory?

No. You might find that information in the Linux kernel documentation (perhaps this is what you want?), or you can find it by looking at the source code of things that are capable of loading and running the Linux kernel (e.g. GRUB or QEMU).

[CRoemheld]

I see, so there are multiple options then, depending on what you want to achieve.

But if I say I would like to emulate an OS like Linux, the way QEMU does?
Meaning, I've got an ELF of the OS i want to run in a VM, and then kind of pass this ELF to my Hypervisor which tries to execute it by setting the values appropiately so that VMLAUNCH jumps to the entry point of this ELF (Is this what QEMU does exactly?)

If what you say applies, then QEMU would set the instruction pointer and/or Code segment values differently, meaning not like the values I posted above?
Would you know by any chance where QEMU for example sets the values accordingly?

It's really a shame to be honest, that there aren't that many hobby developers or documentations around explaining those steps for virtualization.

[Octocontrabass]

Meaning, I've got an ELF of the OS i want to run in a VM, and then kind of pass this ELF to my Hypervisor which tries to execute it by setting the values appropiately so that VMLAUNCH jumps to the entry point of this ELF (Is this what QEMU does exactly?)

On x86, QEMU cheats a little bit by using a BIOS ROM that includes an embedded Linux bootloader. However, there's no reason why you couldn't directly VMLAUNCH using the kernel's entry point. (QEMU probably does this because x86 Linux kernels tend to assume a lot about the BIOS ROM, which can be troublesome if you don't have a BIOS ROM.)

If what you say applies, then QEMU would set the instruction pointer and/or Code segment values differently, meaning not like the values I posted above?

The register values you posted above are incorrect for executing the kernel's entry point.

Would you know by any chance where QEMU for example sets the values accordingly?

Unfortunately not, but I can't imagine it would be too difficult to find in the QEMU source.

[CRoemheld]

On x86, QEMU cheats a little bit by using a BIOS ROM that includes an embedded Linux bootloader. However, there's no reason why you couldn't directly VMLAUNCH using the kernel's entry point. (QEMU probably does this because x86 Linux kernels tend to assume a lot about the BIOS ROM, which can be troublesome if you don't have a BIOS ROM.)

Actually I was thinking about using a pseudo BIOS ROM area in memory to act as the real BIOS ROM on startup of the VM. But I think this is too complicated right now, so I am discarding that idea for now.
Regarding the VMLAUNCH, jumping directly to the kernels entry point: I was trying to use the vmlinux ELF, made by "make menuconfig", with no changes to the initial menuconfig. The problem is that the entry point of vmlinux using this menuconfig is for 64-bit mode (phys_startup_64), which does not work when trying to virtualize it. Therefore I would need a vmlinux ELF which contains everything, meaning all switches from real-mode and protected-mode up to long-mode. Is it possible to make a vmlinux ELF by changing some settings in the menuconfig or using a predefined config?

The register values you posted above are incorrect for executing the kernel's entry point.

I changed them to CS base 0x003C000 and guest instruction pointer 0x5000 -> 0x003C5000. The kernel entry point is now reached via vmlaunch, but see answer to quote above (regarding phys_startup_64).

Unfortunately not, but I can't imagine it would be too difficult to find in the QEMU source.

Okay, this might take a while.

[Octocontrabass]

I changed them to CS base 0x003C000 and guest instruction pointer 0x5000 -> 0x003C5000. The kernel entry point is now reached via vmlaunch, but see answer to quote above (regarding phys_startup_64).

A 64-bit entry point expects your VM to be in 64-bit mode, where the CS base is always zero. Are you still trying to start the VM in real mode?

[CRoemheld]

A 64-bit entry point expects your VM to be in 64-bit mode, where the CS base is always zero. Are you still trying to start the VM in real mode?

If the entry point is a 64-bit one, for example in Linux with its 0xffffffff80100000 GVA, it would mean I also would need to setup the guests PML4 pagetable in the host. I was trying to avoid that by letting the guest setup its own PML4 pagetable, meaning the guest would need to follow its own startup routine, which means starting in real-mode, going into protected mode and setting up the 64-bit GDT and PML4 pagetable and then jump into the 64-bit kernel.

I was under the assumption this would be the correct "way" to run a guest OS, where the guest really acts on its own, just as if there was no VMM. The host merely would jump into action whenever a VM-exit occurs, handle the individual exit reasons and them switch back into the guest (VMX non-root). And since the Linux kernel cannot work without making all the prerequisites such as a PML4 table, the 64-bit GDT and so on, jumping directly into the 64-bit kernel would not make any sense, because the host, or in this sense, me, would need to setup everything, which somewhat defies the understanding of what I have about Virtualization.

EDIT:

I was originally trying to pass an ISO into memory and read it, just as if I were about to run Ubuntu from an ISO in qemu. If qemu gets an ISO passed to, it would also let the Ubuntu startup routine work, meaning qemu does not directly jump into the kernel but start from the 16-bit real mode where all prerequisites are being made, switch to protected mode and finally to long mode and the 64-bit kernel.

[Octocontrabass]

If the entry point is a 64-bit one, for example in Linux with its 0xffffffff80100000 GVA, it would mean I also would need to setup the guests PML4 pagetable in the host. I was trying to avoid that by letting the guest setup its own PML4 pagetable, meaning the guest would need to follow its own startup routine, which means starting in real-mode, going into protected mode and setting up the 64-bit GDT and PML4 pagetable and then jump into the 64-bit kernel.

The Linux startup routine does not include those parts. It expects to start in 64-bit mode, with identity mapping for Linux itself, the boot parameters, the command line, and enough memory for Linux to set up its own page tables.

And since the Linux kernel cannot work without making all the prerequisites such as a PML4 table, the 64-bit GDT and so on, jumping directly into the 64-bit kernel would not make any sense, because the host, or in this sense, me, would need to setup everything, which somewhat defies the understanding of what I have about Virtualization.

Linux expects a bootloader. Since you're trying to directly execute the Linux kernel in your VM, you must do the same things a bootloader would do.

I was originally trying to pass an ISO into memory and read it, just as if I were about to run Ubuntu from an ISO in qemu. If qemu gets an ISO passed to, it would also let the Ubuntu startup routine work, meaning qemu does not directly jump into the kernel but start from the 16-bit real mode where all prerequisites are being made, switch to protected mode and finally to long mode and the 64-bit kernel.

QEMU doesn't do anything special to load the ISO, it simply provides an emulated disc drive with the ISO loaded as a virtual disc. The BIOS ROM inside QEMU will then perform the same boot process that real hardware does. All of those "prerequisites" are handled by the BIOS ROM and the bootloader in the ISO (probably GRUB).