A Beginner VMX Question

[kemosparc]

Hi,

I want to start to learn how to use the virtualization capabilities from my own hobby OS.

I have a problem with understanding the basic concepts as all documentation starts explaining from a point where pre-understanding is needed.

I have basic questions that I need some answers on to be able to understand the Intel documentation and the VMX examples:

1. What is a virtual machine from the hardware point of view. Obviously it is not the same as the virtual machine started by QEMU/KVM for example ?! I think it is much simpler and is being used by the hypervisor, but I don't have a clear vision of it.
2. What is the use of VMX and what can I do with it?
3. What happens when you execute the VMXON instruction and what is supposed to be done next?
4. Is a started virtual machine with VMXON gets a pointer to a code segment to run or it waits for instructions to be passed to it for execution and if so how?

What I am looking for is something like VMX for dummies of VMX baby steps.

I just want to verify something and see if I have the correct imagination. If I have a hobby OS that I have wrote its boot loader following the boot loader baby steps presented in the OSDEV tutorial, and it can boot into real mode. I have followed the tutorial in http://wiki.osdev.org/Entering_Long_Mode_Directly to swicth to 64 bit long mode and was able to map all the available memory and start the available APs. My question is there a way to start a VM with VMXON and re-run the same OS image on multiple virtual machines? Is that one of the possible usage of it?


Kindly, if anyone has information, tutorials, or documentation about how to utilize VMX for a beginner starting from scratch step by step I would appreciate pointing it out as I think I need to read more primitive documentation before starting to look at the Intel manual and advanced tutorials that expect the reader to understand in the first place.


Thanks a lot.
Kemosparc.

[~]

It looks like VMX is a replacement to V86, which is a very good thing for 64-bit mode.

You could probably start tasks with an excellent level of protection with it, but the CPU needs to support the VMX extensions.

We can search this on Google:
vmx os tutorial

The VMX tutorial from the OSDev Wiki (http://wiki.osdev.org/VMX) apparently is just a stub.

But basically we have instructions to start, stop and in general manipulate the virtual machine/virtual machine manager (VMM) (it probably would only consist on handling the interrupts, the I/O ports or memory-mapped devices, and emulating the privileged instructions that are not allowed to be executed from the VMX task).


You need to assume that this problem is as if you have something like having to learn how to handle the GIF file format with its LZW algorithm, or how to use the HTML5 canvas, or how to draw Bresenham or Bézier lines, probably with or without antialiasing using Xiaolin Wu's antialiasing algorithm

That's the level of the tutorial quality we are needing here: A sample bootable kernel that implements multitasking using VMX.



It's probably a good time to start over implementing everything that is new technology, starting as tutorials that run in HTML5, under a regular operating system, under an emulator like Bochs and/or VirtualBox, and under real hardware as a standalone test and/or final kernel.

Maybe it's necessary again just like it was necessary to teach ModeX and how to program the SoundBlaster.

It's clear that we need new tutorials for the newest CPU architectures and peripherals. There are currently only tutorials from the time of OSDever.net and all tutorials are in general at that level (Pentium 4, IDE/ATA-ATAPI disks, PS/2 mouse and keyboard, standard VGA and BIOS along with VESA modes, almost only 32-bit mode). Those are excellent tutorials but they are mostly for 32-bit PS/2 machines with IDE interfaces, not so much for 64-bit machines mostly mostly with USB peripherals, DVI/HDMI instead of VGA, and the latests CPUs and motherboard hardware.

We need sample tutorials and kernels even if they don't run in all machines for being able to understand what's going on anew, after more than a decade for the very most fine classic code that already exists.

[zdz]

The Intel docs are pretty good at explaining it and the interaction between it and other features.

I'll try to answer your questions, but note that this is a short answer.
VMXON simply puts the CPU in VMX-root mode (some people call this ring -1). You launch a virtual machine (I'll call this a guest) by doing a VMLAUNCH. After VMLAUNCH, certain events will cause VMEXITs - this will switch from the guest (which can be in any mode of operation, any ring) to the host. The host has to handle the event and resume (VMRESUME) the guest. You can use this in order to control the guest. The life cycle is explained in the documentation. You exit the VMX-root by doing a VMXOFF.
What you can do with VMX? You can run something in a isolated (and more secure) environment (see what Windows 10 does with Device Guard). You can run multiple OS instances at the same time. I even saw someone talking here on OSdev about using VMX in order to run 16/32 bit code inside his 64bit OS without switching the CPU back and forth (I don't know how that ended). And even fancier things with additions like #VE.

It's hard to find a really good VMX tutorial on the internet. Especially because it is a complex feature.

You can learn it, step by step. You could take a minimal version of your kernel and try to build a minimal hypervisor over it. At first try to launch a guest OS (Linux, Windows, your own, doesn't matter) - your first goal should be to successfully boot and shut down the guest. My advice is to really use a minimal kernel. You should be able to do it just by keeping the memory manager, a basic IDT to handle PF, GP etc., a basic synchronization mechanism and the AP start up pieces (note that any AP that isn't started by you can be started by the guest and you can end up with the guest running on one virtual CPU and 3 real CPUs). See what you can learn this way just by trying to have the guest run in a decent manner. Then, you can start to worry about other things like virtual interrupts, power states, inter-CPU communication, scheduling, timers, VTD and so on. It's overwhelming if you try to understand it all at once.

There are a lot of design issues ranging from "how do I make sure that the guest doesn't try to use the physical memory in which the host is located" to "how do I communicate between CPUs?".
Also, note that there are some differences between Intel and AMD when it comes to this. I recommend Intel because the documentation is better.

You can look at Intel's IKGT and XEN Project to see what VMX can do.

[Nable]

The Intel docs are pretty good at explaining it and the interaction between it and other features.
[...]
Also, note that there are some differences between Intel and AMD when it comes to this. I recommend Intel because the documentation is better.

Huge "no" here. Judging from KVM development history and my personal experience, AMD's implementation and docs are far cleaner (and thus better) than Intel's one. AMD's implementation is quite simple to understand and it's description is very straight-forward. And debugging of VMX support took a lot of time and effort while my study and docs about it are very twisted, it takes a lot of time to find the truth about e.g. what values should be used for some bits, else you just get VMEXIT with "Invalid VM state" and it's all. Oh, Bochs saved me that time, although I've come accross some bugs there too.

It's hard to find a really good VMX tutorial on the internet. Especially because it is a complex feature.

This is true. Although I've found one tutorial that may be useful (at least, it's simple and written in a good way) when you are just starting. I don't remember the original link, but here is the page with the ~same text that helped me very much in 2012: http://www.codeproject.com/Articles/215 ... rogrammers

[zdz]

The Intel docs are pretty good at explaining it and the interaction between it and other features.
[...]
Also, note that there are some differences between Intel and AMD when it comes to this. I recommend Intel because the documentation is better.

Huge "no" here. Judging from KVM development history and my personal experience, AMD's implementation and docs are far cleaner (and thus better) than Intel's one. AMD's implementation is quite simple to understand and it's description is very straight-forward. And debugging of VMX support took a lot of time and effort while my study and docs about it are very twisted, it takes a lot of time to find the truth about e.g. what values should be used for some bits, else you just get VMEXIT with "Invalid VM state" and it's all. Oh, Bochs saved me that time, although I've come accross some bugs there too.

It's hard to find a really good VMX tutorial on the internet. Especially because it is a complex feature.

This is true. Although I've found one tutorial that may be useful (at least, it's simple and written in a good way) when you are just starting. I don't remember the original link, but here is the page with the ~same text that helped me very much in 2012: http://www.codeproject.com/Articles/215 ... rogrammers

Well, the documentation part is subjective, but I think Intel offers a more complete view.
I also stumbled on the same CodeProject tutorial in my beginner days. Oh, the memories.

[kemosparc]

Thank you all very very much, for spending the time to answer my questions.

This tutorial is exactly what I was looking for.

I will check it out in details.

Thanks,
Karim.

[Nable]

Well, the documentation part is subjective, but I think Intel offers a more complete view.

It sounds somehow strange because these docs cover different things (not a _different view_ on the same thing), i.e. AMD has (and describes) SVM but Intel has only VMX (it looks like they couldn't implement just the same technology due to patent coverage, oh, it's so sad).

[feryno]

Nable is right about differences between Intel and AMD CPU doc.
For debugging AMD hypervisor there is SimNow 4.6.2 public.
For Intel there is BOCHS which is really good and advanced and emulates everything well, first missing thing for me was VMCS shadowing (I did not want to modify BOCHS source code either recompile).
On Intel these are simplified steps (every step requires additional substeps)
vmxon
vmclear
vmptrld
a lot of vmwrite, one of them is into guest_RIP vmcs field
vmlaunch, it it succeeds, execution continues at guest_RIP, if it fails, execution continues after vmlaunch instruction where you can investigate why (carry flag, zero flag, vmx instruction error code)

So now on vmlaunch success, CPU starts to run from guest_RIP and some instructions cause vm exits. For that you have procedure handling vm exits, its address is written in host_RIP field of VMCS
vm exit handler examines some vmcs fields (read only fields, well current CPUs are able to write them but their name readonly stayed), and another VMCS fields. Some instructions have to be emulated (e.g. cpuid, mov CR, RDMSR/WRMSR etc) and after they are emulated by hypervisor, guest_RIP is increased to point after the emulated instruction and then last instruction of vm_exit_handler is vmresume at which execution continues in guest.

vm exit handler (address written in host_RIP VMCS field) does typically these steps:
few vmread instructions
emulate what needed for guest
if instruction causing vm exit emulated, then vmread vm exit instruction length, vmread guest RIP, add both together, vmwrite into guest RIP (remember, there are some vm exits which do not need to change guest RIP, e.g. at intercepting some exceptions and then injecting exceptions back to guest, etc)
vmresume

More than 95% of you work will be reading manuals and debugging, coding not more than 5%, trust me.

Btw the manual
http://www.codeproject.com/Articles/215 ... rogrammers
suggest to disable gate A20, that's nonsense. Your hypervisor will be useful to monitor OS running in guest mode or to support to run more OS-es at the same CPU. Both need gate A20 enabled as current OS-es use protected mode and much more RAM than gate A20 limits.
You can also run your first hypervisor with EPT disabled (in beginning of development), you'll implement it later (unlike article at codeproject).

[zdz]

It sounds somehow strange because these docs cover different things (not a _different view_ on the same thing), i.e. AMD has (and describes) SVM but Intel has only VMX (it looks like they couldn't implement just the same technology due to patent coverage, oh, it's so sad).

I wanted to say that Intel docs cover VMX better than AMD docs covers SVM.

I found it easier in the beginning to set up EPTs from the start (one to one with full rights).

[~]

It sounds somehow strange because these docs cover different things (not a _different view_ on the same thing), i.e. AMD has (and describes) SVM but Intel has only VMX (it looks like they couldn't implement just the same technology due to patent coverage, oh, it's so sad).

I wanted to say that Intel docs cover VMX better than AMD docs covers SVM.

I found it easier in the beginning to set up EPTs from the start (one to one with full rights).

I've always thought about reading the AMD and Intel manuals, then combine them in my own documentation that includes the best of both.

And the same about inspecting several OS projects to have a redundant picture of the problem and know better how an OS looks like in the same way a file format like PDF along with its implementation (creator and reader) should look like to make it possible to write an advanced one.

[kemosparc]

Hi,

I have been doing some reading and I will start applying practically soon.

I have a question that crossed by mind.

If my OS has the IDT loaded and the timer interrupt is configured.

What happens when a timer interrupt occurs while the vmlaunch is executed and the guest is executing transactions ?

Is the interrupts disabled when vmlaunch is called by the host until vmexit occurs?

The same question applies with the rest of the interrupts, what if I have an e1000 driver configured and a packet arrives while the VM is executing?

Thanks,
Karim.

[alexg]

Well it actually depends on how you setup your VMCS structures.
There is a bit in the Pin-Based VM-Execution Controls (External-interrupt exiting) which specifies how external interrupts are handled:

• If this bit is set => each time an interrupt is triggered while the guest is running an EXIT will occur to the host. This exit will happen indifferently if the interrupt belongs to you or the guest so you need to take care of the latter case, i.e. you should inject the interrupt back into the guest if it's his.
• If this bit is not set => if an interrupt is triggered while the guest is running it will be treated in the guest, i.e. it will use the guest IDT and the hypervisor will have no way of knowing an interrupt happened.

What I've told you, applies only to Intel VT-x. Some new processors may have some features like vLAPIC and posted-interrupts but I've never researched them (because I didn't have a capable processor), maybe these new features can be used to customize what interrupts you want exits on (instead of all or nothing).

[zdz]

You should read Chapter 29 "APIC virtualization and virtual interrupts". And related chapters. And about external interrupt exiting and posted interrupts.
Try to see what fits your needs. In essence, if you need interrupt inside the host you will want to use virtual interrupts. But it also depends on why you need interrupts inside the host.

[kemosparc]

Hi,

I have two questions:
1. What if I want to enter long mode in my guest? The code examples describes real and protected modes, which are set through the CR0, CR3, and CR4 registers which are available in the VMCS. I want to know if all what need to be done is to set the LME bit? If so how can I do this in the VMCS record of the guest before launching:

Code: Select all

    mov ecx, 0xC0000080 
    rdmsr
    or eax, 0x00000100 
    wrmsr

In other words how can I set the MSR of the guest to be in long mode

2. Can I use my original host PM4L, I know that this can cause problems if I have more than one guest where memory regions might be over written by different vms and even if I am using a single VM, inconsistency might appear between the VMM and the guest, but if I do that just to get things working and launch the VM, does this cause problems with the launching of the VM? My question is can the PM4L page table I created already to enter the long mode be used and passed in the VMCS of the guest?


Thanks a lot,
Karim.

[alexg]

For the answers to both of your questions I suppose your hypervisor is already in long mode:

1. If you read chapter 24.4 of Intel System Programming Manual Volume 3 (Guest-State Area) you will see there are VMCS fields which correspond to guest-state registers and which are loaded on VM entry. You will see the registers CR0, CR3 and CR4 and you will also see some MSRs: IA32_EFER is the one which interests you. You need to set these registers as you would set them in normal operating mode to indicate the guest is in long mode. Besides these registers you will also need to setup the segment selectors and the other control bits appropriately, see also chapter 26.3 (Checking and Loading Guest State).

2. The host and the guest will share the same paging tables, if you simply want to see the guest started I see no problem with this. However, when you will want to start an actual guest (lets say a commercial OS) you won't be able to do because you will invalidate the use of a hypervisor, the guest will have rwx access to all the memory (including the hosts).