Paging or malloc

mmiikkee12 · Post by **mmiikkee12** » Sat Sep 15, 2007 9:50 pm

I have my kernel to a point where it boots, loads a GDT and IDT, and runs through an infinite loop in 4 seconds. (take that linux!

Now I don't know whether to do paging next or malloc. They kind of go hand in hand... paging will need malloc to allocate pagedirs and pagetables and pages, but malloc will need to ask the paging system for more pages if it runs out.

JackScott · Post by **JackScott** » Sat Sep 15, 2007 10:14 pm

If that's all there is to it, I might be inclined to design the interface first, then just roll a dice to decide what to do next.

In a slightly more serious tone, it might be a good idea to do paging first. There are a lot of drop in malloc()s you could use to test the other side of the interface, before doing your own.

frank · Post by **frank** » Sat Sep 15, 2007 10:15 pm

Do paging first then malloc. You don't need malloc to do paging but in general you need paging to do malloc. Instead of using malloc to allocate the page directories and page tables you should create a physical memory manager that uses a bitmap or a stack to figure out which pages are free.

mmiikkee12 · Post by **mmiikkee12** » Sat Sep 15, 2007 10:31 pm

Do paging first then malloc.

it might be a good idea to do paging first.

OK, thanks, but...

Instead of using malloc to allocate the page directories and page tables you should create a physical memory manager that uses a bitmap or a stack to figure out which pages are free.

That solves 2 different problems... Anyway, I already have a bitmap class I whipped up (did I just say that?) just in case I needed it for anything.

mmiikkee12 · Post by **mmiikkee12** » Sat Sep 15, 2007 10:48 pm

Hmm. I'm confused now...

If I have a Bitmap<1048576> free_pages; I'm keeping track of 4GB worth of pages. But the page at 0x80000000 may be at 0x80000000 in process 1, but it doesn't have to be in process 2. So which process am I keeping track for here? Or am I completely missing the point?

speal · Post by **speal** » Sat Sep 15, 2007 11:21 pm

The way I've done this is:

1. Create a simple physical memory allocator that gives out fixed-size blocks (1 page, of course). This allocator has no overhead when no physical memory has been added to the allocator. It pulls a page from the front of the added memory block when memory is added (if the pool of meta-data memory is too small).

2. Write the page table code that performs memory mappings (recursively, if you'd like). This will request pages of physical memory from the allocator when creating a new page table/directory.

3. A morecore implementation that gives out pages of virtual memory, and either maps them on allocation, or flags them as valid mappable pages for the page-fault handler (you'll need to implement that handler unless you go with the map-on-allocation strategy, at least for now).

4. Finally, a malloc that hands out variable-size blocks of memory. This requests a page of virtual memory using morecore when a malloc request cannot be fulfilled with the current pool (no sufficiently large regions available).

Of course, I'm working in long mode and can afford to have all of my physical memory mapped in for quick access. If you're in pmode and don't want to give up all that address space, this becomes a bit trickier. You'll need to find some way to provide the page table mapper with writable memory at a known physical address.

mmiikkee12 · Post by **mmiikkee12** » Sat Sep 15, 2007 11:32 pm

OK. So what exactly does the bitmap keep track of?

edit: I am using pmode btw. The x64 port is the job of another, yet unhired developer

jnc100 · Post by **jnc100** » Sun Sep 16, 2007 4:23 am

mmiikkee12 wrote:OK. So what exactly does the bitmap keep track of?

The bitmap keeps track of physical pages of memory (i.e. the actual RAM in your computer). Virtual address space is managed by some sort of system that divides up the address space (per process) into chunks such as .text, .data, .bss, stack and heap. Malloc further divides up the heap into small, manageable portions. Your paging code provides the glue that extends the virtual address space by asking the bitmap (physical memory allocator) for a page and then assigning it to a certain position within the virtual address space of a process.

Regards,
John.

JamesM · Post by **JamesM** » Sun Sep 16, 2007 7:19 am

You may want to look at:

http://www.jamesmolloy.co.uk/document.p ... section=6.

That gives an overview of how to enable paging with respect to writing a heap next. (The next tutorial will deal with writing a kernel heap). The tutorial site is in alpha and some links are broken. Sorry about that

t0xic · Post by **t0xic** » Sun Sep 16, 2007 10:05 am

James,

That is one of the best tutorials on paging I have ever seen! I commend you on a job very well done. I have been having trouble with my paging stack, so I'll try out your method

--Michael

JamesM · Post by **JamesM** » Mon Sep 17, 2007 3:07 am

awww shucks *blushes* cheers pal

mmiikkee12 · Post by **mmiikkee12** » Mon Sep 17, 2007 3:09 pm

OK, I started on paging. I have some code written, but it doesn't work very well.

00059023710e[DEV ] read from port 0x0083 with len 4 returns 0xffffffff
00059023743e[DEV ] read from port 0x0083 with len 4 returns 0xffffffff
00059023782i[CPU0 ] protected mode
00059023782i[CPU0 ] CS.d_b = 32 bit
00059023782i[CPU0 ] SS.d_b = 32 bit
00059023782i[CPU0 ] | EAX=00000000 EBX=00103000 ECX=d1020bb4 EDX=d1020bb4
00059023782i[CPU0 ] | ESP=00000000 EBP=00101480 ESI=000247d8 EDI=00030658
00059023782i[CPU0 ] | IOPL=0 id vip vif ac vm RF nt of df if tf sf ZF af PF cf
00059023782i[CPU0 ] | SEG selector base limit G D
00059023782i[CPU0 ] | SEG sltr(index|ti|rpl) base limit G D
00059023782i[CPU0 ] | CS:0008( 0001| 0| 0) 00000000 000fffff 1 1
00059023782i[CPU0 ] | DS:0010( 0002| 0| 0) 00000000 000fffff 1 1
00059023782i[CPU0 ] | SS:0010( 0002| 0| 0) 00000000 000fffff 1 1
00059023782i[CPU0 ] | ES:0010( 0002| 0| 0) 00000000 000fffff 1 1
00059023782i[CPU0 ] | FS:0010( 0002| 0| 0) 00000000 000fffff 1 1
00059023782i[CPU0 ] | GS:0010( 0002| 0| 0) 00000000 000fffff 1 1
00059023782i[CPU0 ] | EIP=00101487 (00101487)
00059023782i[CPU0 ] | CR0=0x80000011 CR1=0 CR2=0xfffffffc
00059023782i[CPU0 ] | CR3=0x0010d000 CR4=0x00000000
00059023782i[CPU0 ] >> add byte ptr ss:[esp+edx+0x100010], cl : 008C1410001000
00059023782e[CPU0 ] exception(): 3rd (14) exception with no resolution, shutdown status is 00h, resetting

Code: Select all

namespace Paging
{
	//Bitmap<1048576> free_pages; // of PHYSICAL memory.
	pte_t kpagedir[1024] __attribute__((aligned(4096)));
	pte_t kpagetables[1024*1024] __attribute__((aligned(4096)));
	
	// Symbol put at the end of the file by the linker. Used to find kernel size
	extern "C" uint_32 _end;
	
	void set_page_directory(pte_t *pagedir)
	{
		asm volatile
		(
			"mov %0, %%cr3"
		::	"r" (pagedir)
		);
		
		uint_32 cr0;
		asm volatile
		(
			"mov %%cr0, %0"
		:	"=r"(cr0)
		);
		cr0 |= 0x80000000; // Enable paging
		asm volatile
		(
			"mov %0, %%cr0"
		::	"r"(cr0)
		);
	}

	void set_pte(pte_t *pte, uint_32 addr, bool global, bool cache_disable, bool write_through,
		bool user, bool writable, bool present)
	{
		pte->present = present;
		pte->writable = writable;
		pte->user = user;
		pte->write_through = write_through;
		pte->cache_disable = cache_disable;
		pte->accessed = pte->dirty = false;
		pte->global = global;
		pte->unused = 0;
		pte->addr = addr >> 12;
	}

	void init()
	{
		TextIO::print("setting PDEs: ");
		for (uint_32 i = 0; i < 1024; i++)
		{
			TextIO::print("%d ", i);
			set_pte(&kpagedir[i], (uint_32)&kpagetables[1024 * i], false, false, false, false, true, true);
		}
		
		TextIO::print("\n\nsetting PTEs: ");
		// Kernel size, rounded up to next page boundary, and divided by 4K for number of kernel pages.
		uint_32 ksize = Misc::round_up<uint_32> ((uint_32)&_end, 4096) / 4096;
		for (uint_32 i = 0; i < 1024; i++)
		{
			TextIO::print("%d ", i);
			set_pte(&kpagetables[i], (4096 * i), false, false, false, false, true, true);
		}
	
		TextIO::print("\nsetting page directory\n");
	
		set_page_directory(kpagedir);
	}
}

The crash happens after it sets the page directory. A bit of poking around later, I found out that it only happens if I try to print text after enabling paging.

(This is what I don't like about osdev - having to conform to Intel's stupid rules. I like the part where I get to make up specs that really shouldn't work, and then write code to make them work

Combuster · Post by **Combuster** » Mon Sep 17, 2007 3:57 pm

00059023743e[DEV ] read from port 0x0083 with len 4 returns 0xffffffff

what device are you programming that needs a dword I/O read?

ESP=00000000

this value doesn't make sense... what did you do with your stack?

vhg119 · Post by **vhg119** » Mon Sep 17, 2007 4:21 pm

JamesM wrote:You may want to look at:

http://www.jamesmolloy.co.uk/document.p ... section=6.

That gives an overview of how to enable paging with respect to writing a heap next. (The next tutorial will deal with writing a kernel heap). The tutorial site is in alpha and some links are broken. Sorry about that

James, your section 6 has the title of section 5. I just thought you should know.

JamesM · Post by **JamesM** » Tue Sep 18, 2007 1:31 am

Heh, yes I know. Thanks. They all have the 5. title at the moment - It's really hard to test my code when whenever I make a change I have to plead with the server admin to restart apache and so flush the perl cache. GRRRRRR!!!