JOS VMX (low level ept programming)

[bunkmaster]

Hi,

This is my first post:

I am trying to implement Intel Virtual Machine Extensions on MIT JOS. I am implementing them in the following project:

https://github.com/JulesWang/JOS-vmx

This project has 5 exercises and it is used as lab by Shanghai JiaoTong University. I have completed ex 1,2,4 and I am really really struggling to implement ex3 which is regarding low level ept programming.

The exercise can be found on this page:

https://github.com/JulesWang/JOS-vmx/bl ... m/vt_ept.c

I have read and re-read Ch. 28 from Intel manual regarding ept. The way this project is trying to implement ept is similar to that of Xen and so I tried learning how Xen does it, but still not able to understand how to go about it.

I have also read the following OSdev forum page regarding Intel ept:

http://f.osdev.org/viewtopic.php?f=1&t= ... bc#p223859

and all the links inside it, esp:

http://www.codeproject.com/Articles/215 ... rogrammers

and I still have no idea.

For heavens sake can someone help me or atleast give me a hint what the heck is going on in this code.......

(I have done some linux kernel assignments and have some experience of low level programming)

PS: This is not my HW or anything.

[bunkmaster]

The following is the code which on the page:

Code: Select all

#include <inc/hvm/vt_ept.h>
#include <inc/hvm/vt.h>

ept_control g_ept_ctl;

static void ept_create_table(struct Page* ept_pml4_page);
static int ept_update_identity_table ( void* pt, u8 level, gfn_t gfn, u32 p2m_type, u32 op_type);

void ept_setup(void)
{
	u32 error = 0;
	int i = 0;

	/* allocate ept PML4 page */
	struct Page *ept_pml4_page;

	error = page_alloc(&ept_pml4_page);
	assert(error==0);
	memset(page2kva(ept_pml4_page), 0, PAGESIZE);
	
	/* set ept */
	g_ept_ctl.ept_mt = EPT_DEFAULT_MT;
	g_ept_ctl.ept_wl = EPT_DEFAULT_WL;
	g_ept_ctl.asr = GFN(page2pa(ept_pml4_page));

	ept_create_table(ept_pml4_page);
}
static
void ept_create_table(struct Page *ept_pml4_page)
{
	gfn_t gfn;
	virt_t* ept_pml4_page_virt = page2kva(ept_pml4_page);

	//0xFFFFFFFF = 4GB; quick and dirty
	for(gfn = 0; gfn<GFN(0xFFFFFFFF); gfn += EPT_EACHTABLE_ENTRIES)
	{
		if(!ept_update_identity_table(ept_pml4_page_virt, 
					      EPT_DEFAULT_WL, 
					      gfn, 
					      P2M_FULL_ACCESS, 
					      P2M_UPDATE_ALL)
		  )
	   	{
			panic("In ept_create_table gfn:%#08x",gfn);
		}
	}
}
static int 
ept_update_identity_table (
	void* pt,
	u8 level,
	gfn_t gfn,
	u32 p2m_type,
	u32 op_type
)
{
	u64 offset = 0;
	ept_entry_t ept_entry;	
	u64 mask = 0;
	u32 error = 0;

	assert((gfn & (EPT_EACHTABLE_ENTRIES - 1))==0); 

	mask = (1ULL << ((level+1) * EPT_TABLE_ORDER)) - 1;
	offset = (gfn & mask) >> (level * EPT_TABLE_ORDER);

	if (level == 0)
	{
		u32 i = 0;
		for(i = 0; i < EPT_EACHTABLE_ENTRIES; i++)
		{
			panic("ept_update_identity_table not implemented");
			/*Ex3: set the mapping and attributes for all entries*/
			/* hint: 3B 25 EXTENDED PAGE TABLES (EPT) */
			/* NOTE: set MTRR_TYPE_UNCACHE to the emt of read only memory address*/
			/* NOTE: the max memory size of JOS is less than 64MB.
			 *      To make sure that VM will not overwrite this memory region,
			 *      we shall give a offset of the mapping of memory address 
			 *      above 0x0010 0000.(FIXME)
			 *      Guest PA                PA
			 *      --------------       ------------
			 *      |            |    ___|          |
			 *      --------------   /   ------------0x0400 0000(64MB)
			 *      |            |__/    |JOS memory|
			 *      --------------       ----------- 0x0010 0000(1MB)
			 *      |            |_______|          |
			 *      --------------       -----------
			 */
					
		}
		return true;
  	}
        
	ept_entry.epte =  ((u64 *)pt)[offset];

	phys_t sub_pt_phys;
	virt_t *sub_pt_virt;

	if (ept_entry.epte == 0 ) 
	{
		panic("ept_update_identity_table not implemented");
		/* Ex3: alloc page as sub page table and set the entry */
		/* hint: page_alloc */
		/* hint: ept_entry.mfn = GFN(sub_pt_phys); */
  	} 
	else 
	{
		sub_pt_phys = (phys_t)(ept_entry.mfn << PAGESIZE_SHIFT);
		sub_pt_virt = (virt_t *)KADDR(sub_pt_phys);
	}

	return ept_update_identity_table ((void *)sub_pt_virt, level-1, gfn, p2m_type, op_type);
}

[feryno]

Hi, creating EPT consumes a lot of memory for constructing translation tables.
I suggest you to use 3-level 2MB paging for most of memory, but be careful to set proper attributes (cacheable / uncacheable) - I got machine check exceptions on Intel when I didn't care. You can't set the whole memory uncacheable because of about 100-1000 times worse performance of guest (it will work but terrible slowly...). You can't set the whole memory cacheable, especially memory A0000-FFFFF and some memory in C0000000-FFFFFFFF needs to be uncacheable.
So I suggest to use paging mixed with 3-level 2MB paging and 4-level 4kB paging.
For first 2MB use 4-level 4kB paging where A0000-FFFFF is uncacheable.
Then use 3-level 2MB paging upto BFFFFFFF.
For memory range C0000000-FFFFFFFF use again 4-level 4kB paging and determine from UEFI BootServices GetMemoryMap whether it is physical memory and should be cacheable or whether it is not and then use uncacheable attributes. For BIOS based computer use INT15h / eax=0000E820h to determine physical memory map.
For memory above 4GB use again 3-level 2MB paging upto top of physical memory.
Using such mixed tables consumes only about 4MB of memory to construct EPT tables to cover PC with 32 GB of RAM which is acceptable.
At one of my PC the last level of identity mapping of the physical memory C0000000-FFFFFFFF looks like in the attachment (some pages are cacheable = almost first half, some uncacheable = second half).
Always check tables manually as there is huge amount of data to create and every mistake has ugly consequences.

Why don't you use identity map? Just to hide JOS? I'm using different approach to steal memory for my hypervisors.

Just briefly from my head, last 4th level of your 4kB EPT paging tables should look like:
dq 00000037h
dq 00001037h
dq 00002037h
dq 00003037h
dq 00004037h
dq 00005037h
dq 00006037h
...
dq 0009F037h ; still cacheable
dq 000A0007h ; uncacheable
dq 000A1007h
dq 000A2007h
...
dq 000FF007h ; uncacheable
; the above was 4-level 4kB identity map, now nonidentity map (guest PA = machine PA + 64 MB)
dq 04100037h ; this is cacheable mapping and the value means 00100037h + 64MB
dq 04101037h
...and so on...

When you shift memory in such way don't forget to identity map memory used for APIC and memory-mapped devices (in memory range C0000000-FFFFFFFF) else your OS will malfunction (and uncacheable attributes for memory ranges used for devices and APIC else you get #MCE it was hard to track down so avoid to waste your time by hunting reasons of crashes)