physical memory management troubles

Question about which tools to use, bugs, the best way to implement a function, etc should go here. Don't forget to see if your question is answered in the wiki first! When in doubt post here.
Post Reply
User avatar
xDDunce
Member
Member
Posts: 173
Joined: Tue Aug 12, 2008 4:04 pm
Contact:

physical memory management troubles

Post by xDDunce »

hiya all,

I have just started re-implementing a physical memory manger (been a long time, been busy with college work) and I have quite a simple concept. I got it implemented, and in theory it should work. I've double-triple-quadruple-quituple checked it, and i just can't find the error. anyway...

I currently have 2 ordered arrays. they are used for storing allocated memory(so i don't lose track of it) and freed memory (so i know where the holes are). they are ordered by the address of the memory allocated.

my allocate function has 2 methods of allocating memory. if there are any entries in the freed memory array, it checks the size of the hole and if it fits; it allocates it, removes it from freemem, compares the size wanted with the size of the hole and adds (if necessary) the unneeded memory back to the freed memory array. if it doesn't fit, or there are no entries in the freed memory array, then it just allocates it from the end of known (i.e. pre-allocated) memory and notes it in the used memory array, ready to be added to the freed memory array.

now, my problem is that testing my code on paper it all works. whereas when I test in bochs and parallels it doesn't, although it does allocate off the end of the kernel properly :D. upon a little digging i found that the size attribute of the ordered arrays never increase when i insert a value into them. once again, on paper the code works, in testing it doesn't.

I may just be rushing in desperation to get it working, having developed the idea behind it for about a week now, but having spent 2 days away from coding; i still don't see the problem.

insert algorithm for ordered arrays:

Code: Select all

 
void insert(T newEntry){
		int i = 0;
		while(sort(array[i],newEntry) == true){							//while the array entry at i is less than the entry
			i++;													// ignore the entry
		}
		if(i > size){												// if i references a higher entry than has been recorded
			array[++size] = newEntry;									// add to the end of the array
		}
		else{														// otherwise...
			T sortEntry;											
			while(i <= (size+1)){										//move everything along one
				sortEntry = array[i];
				array[i] = newEntry;
				newEntry = sortEntry;
				i++;
			}
			size += 1;
		}
	}

mem allocate algorithm:

Code: Select all

void * Memory::alloc(dWord size){
	asm volatile("cli");
	
	void *address;												//address to be returned
	if (freemem.getSize() == 0){									//if no known free memory
		address = (void *)endmem;									// allocate from the beginning of unknown memory
		endmem += size;										// adjust the pointer for the size of the allocation
	}															
	else {													// if there is some known free memory
		int i=0;
		while (freemem.getItem(i)->getSize() < size){					//find out if there is a hole big enough
			i++;
			if(i > freemem.getSize()){								//if there are no more entries
				i = 4097;										//confirm no entry
				break;										//break the loop
			}
		}
		if(i <= freemem.getSize()){								//there there was an entry found
			memoryAddress *hole = freemem.getItem(i);				//the the info
			address = hole->getSource();							//extract the address to be returned
			freemem.remove(i);									//remove from known free memory
			if(hole->getSize() > size){								// if the hole is a bit big give some memory back
				memoryAddress *newHole;							//the new hole to be added
				newHole->setSource ((void *)(						//set the address of the hole to
									((dWord)hole->getSource())	//the address of the hole we found
											 + size));			// plus the size we wanted.
				newHole->setSize(hole->getSize() - size);				// the size is just the bit we didn't want
				freemem.insert(newHole);							//insert the new hole into known free memory
			}
		}
		else{													// if there was no entry found
			address = (void *)endmem;								// allocate unknown memory
			endmem += size;									//adjust pointer as neccessary
		}
	}
	
	
	memoryAddress *usedhole;									// add the new allocated memory to used memory
	usedhole->setSource(address);									// address of the block
	usedhole->setSize(size);										// size we wanted/are using
	usedmem.insert(usedhole);									// insert
	asm volatile("sti");
	
	return address;
	
}
 
mem free algorithm:

Code: Select all

void Memory::free(void *addr){

	freemem.dumpSize();										// debug info. always returns 0!!!!!
	memoryAddress *usedHole;
	int i = 0;
	while(i <= usedmem.getSize()){									//find all entries in used memory
		if(usedmem.getItem(i)->getSource() != addr) {					// if the addresses don't match
			i++;												// get the next entry
		}
		else													//if it is
			break;											// i is the index in usedmem, so break the loop
	}
	usedHole = usedmem.getItem(i);								// get the entry at i
	freemem.insert(usedHole);									// insert it into freemem
	usedmem.remove(i);											// remove it from usedmem
}
 
freemem.dumpsize() just prints the size of the freed memory array to the top left corner of the screen and is always 0, showing that the ordered array is broken.

my testing code just allocates memory for 2 variables 4 bytes in length each, frees them, and then re allocates them. at the moment, they are just linearly allocated from the end of the kernel.

all help is very much appreciated.

Cheers,
James.
User avatar
Combuster
Member
Member
Posts: 9301
Joined: Wed Oct 18, 2006 3:45 am
Libera.chat IRC: [com]buster
Location: On the balcony, where I can actually keep 1½m distance
Contact:

Re: physical memory management troubles

Post by Combuster »

I currently have 2 ordered arrays. they are used for storing allocated memory(so i don't lose track of it) and freed memory (so i know where the holes are). they are ordered by the address of the memory allocated.
Algorithmic complexity fail :shock:

Does the code actually get executed? Tried printf debugging? (Or bochs' debugger)

For the rest?

Code: Select all

if(i > size)
with items running from 0..i-1?

Code: Select all

array[++size]
is that a vector? I don't see any bounds checking...

Code: Select all

while(i <= (size+1))
Again indices, now off by 2?

Code: Select all

// allocate from the beginning of unknown memory
[-o< save me :shock:

Code: Select all

i = 4097;
magic numbers eat code monkeys in sleep.
"Certainly avoid yourself. He is a newbie and might not realize it. You'll hate his code deeply a few years down the road." - Sortie
[ My OS ] [ VDisk/SFS ]
User avatar
xDDunce
Member
Member
Posts: 173
Joined: Tue Aug 12, 2008 4:04 pm
Contact:

Re: physical memory management troubles

Post by xDDunce »

Combuster wrote:
I currently have 2 ordered arrays. they are used for storing allocated memory(so i don't lose track of it) and freed memory (so i know where the holes are). they are ordered by the address of the memory allocated.
Algorithmic complexity fail :shock:
complex? really? lol. back when all i ever did was faollow tutorials, i didn't like the way memory was handled in JamesM's tutorials. with all the headers and footers in the middle of memory, i wanted something i could keep track of better and know how much memory has been allocated, where some holes are within that memory and (later) tell where there has been a memory leak by a program or driver.
Combuster wrote: Does the code actually get executed? Tried printf debugging? (Or bochs' debugger)
yes it does, if i try to print something useful like the size of the orderedarray, it just prints 0.
(i have no direct printf function, as i didn't want to clutter my kernel with a text screen driver. planning on putting it into a driver, and soon.)
Combuster wrote:
For the rest?

Code: Select all

if(i > size)
with items running from 0..i-1?
hmm... good point...
Combuster wrote:

Code: Select all

array[++size]
is that a vector? I don't see any bounds checking...
I'm not using any bounds checking at the moment. i'm not doing a full test, only if the bare bones of it is working.
Combuster wrote:

Code: Select all

while(i <= (size+1))
Again indices, now off by 2?
i think i may have miscalculated everything by one, but it still doesn't explain why size always equals 0.
Combuster wrote:

Code: Select all

// allocate from the beginning of unknown memory
[-o< save me :shock:
this is another "because i know it works atm" situation. my next step is to get the size of usable memory and then enforce bounds checking to make sure i'm not using memory that isn't there. also called unknown memory because i haven't used it yet, and have no record of it. not because i don't know it's there.
Combuster wrote:

Code: Select all

i = 4097;
magic numbers eat code monkeys in sleep.
because the maximum size of the array is 4096, the size attribute can never be bigger than that. you'll notice 3 lines later

Code: Select all

 if(i <= freemem.getSize()){
so if i've found something that isn't valid, i discard it and allocate from the end of the kernel instead.(aka unknown memory)

Cheers,
James.

P.S. even with those changes the code still doesn't work. starting to lose my head now...
User avatar
Creature
Member
Member
Posts: 548
Joined: Sat Dec 27, 2008 2:34 pm
Location: Belgium

Re: physical memory management troubles

Post by Creature »

I think Combuster has a point. I've taken a quick glance over your code (so my interpretation may be incorrect), but as I see it. You're using 2 ordered arrays with a static size. I know JamesM uses a sorted array with a static size, but I don't think it's such a good idea). First of all: why use 2 sorted arrays to keep track of free and allocated headers? What if headers are split and all spots in the sorted array are full? It's not a vector, so it can't dynamically expand (which would be impossible anyway, since that'd require the heap, resulting in an infinite loop).

It'd probably be easier if you'd put a header in front of each block of memory, give a header a 'Next' (and possibly a 'Previous') field that points to the next (and previous) headers. When using a linked list, you'll never run into the problem of "not having enough space in your index".

I also understand that using a sorted array for holes can greatly increase the speed by which a hole is found when you need to allocate a new memory block (you'll also get the smallest block so a minimal amount of fragmentation occurs). But then again, what if there are more holes than the index/sorted array can handle? In my heap, I gave heap header's an additional field 'NextBigger' which points to the next hole that is bigger than the current hole (it's a linked list as well).

I'm not saying my heap is better or your approach is bad, but linked lists do have several advantages over static arrays:
  • No problems with ever running out of space in the index.
  • If the index isn't full, memory is wasted with the free space. If it is full, you can't add any more holes or headers.
IMHO, the only advantage a static array offers is the fact that you can immediately pick out a header at any index and you always know the exact size, so you can make assumptions whereas in linked lists, you can't always make these assumptions.

Last but not least, if you're using paging and ever want to allocate page directories or page-aligned blocks on your header, don't forget to write some code for that.

PS: I also thought the headers and footers in JamesM's code were a bit bloated, but he did say he wasn't proud of it and IMHO has a much better implementation in his new code. I had so much problems with his heap code, that I eventually decided to write the entire heap myself where I had problems understanding the theory as it is. After lots of tries, I finally succeeded in writing a pretty (*pretty*) stable heap (I hope).

Hope this helps,
Creature
When the chance of succeeding is 99%, there is still a 50% chance of that success happening.
User avatar
xDDunce
Member
Member
Posts: 173
Joined: Tue Aug 12, 2008 4:04 pm
Contact:

Re: physical memory management troubles

Post by xDDunce »

hmm... I see your point, but most of it i had already taken into consideration.

I know a static sized array is one of the worst ways to handle this type of situation, which is why I'm planning on adding a dynamic reconfiguration of the array, upon getting this working. if i was to implement it now, i would just have more problems.

since a static sized ordered array is easier to implement than a dynamic one without having a reasonable memory manager, i can't do much else right now can I? i hope to have a small internal function which allocates memory straight off the end of all known memory, so i don't have to go through the infinite loop of accessing itself to increase it's size requiring an entry in itself which needs resizing again etc etc.

also, i have no headers. the way I have designed it is to have a descriptor for all free and used memory addresses. the free memory will eventually have a Contract function which will shrink the array if any adjacent entries meet in memory (address + size = address of next entry).

lastly, i chose to use 2 arrays because it leaves me with a list of descriptors for all allocated memory (used memory) and a list of descriptors for all unused memory below the linear allocation address. i am planning on adding a page aligned option, in fact that was my next plan, seeing as it tests my size checking and replacement algorithm.

Cheers,
James.
madeofstaples
Member
Member
Posts: 204
Joined: Thu Apr 12, 2007 8:15 am
Location: Michigan

Re: physical memory management troubles

Post by madeofstaples »

xDDunce wrote:
Combuster wrote:
I currently have 2 ordered arrays. they are used for storing allocated memory(so i don't lose track of it) and freed memory (so i know where the holes are). they are ordered by the address of the memory allocated.
Algorithmic complexity fail :shock:
complex? really?
algorithmically, yes.

recommended (and urgent) reading
Some people are offended by the verifiable truth; such people tend to remain blissfully unencumbered by fact.
If you are one of these people, my posts may cause considerable discomfort. Read at your own risk.
User avatar
xDDunce
Member
Member
Posts: 173
Joined: Tue Aug 12, 2008 4:04 pm
Contact:

Re: physical memory management troubles

Post by xDDunce »

madeofstaples wrote: algorithmically, yes.

recommended (and urgent) reading
circumstantially, yes. if i happen to fill up say 100MB of memory with variable constructs, and then release 10MB's worth of ints/chars etc, then yes it could be a problem. but if a number of small hole are required after all of that, then these holes get filled and relatively quickly.

Although, I agree that my free algorithm could do with a little more thought... i don't think it would be too much trouble for the time being though. (and thanks for the link, that will make some good reading. I have fell into that trap far too often, and only now do i see exactly why...)

Cheers,
James.
madeofstaples
Member
Member
Posts: 204
Joined: Thu Apr 12, 2007 8:15 am
Location: Michigan

Re: physical memory management troubles

Post by madeofstaples »

xDDunce wrote:(and thanks for the link, that will make some good reading. I have fell into that trap far too often, and only now do i see exactly why...)
If you have access to Portal, ScienceDirect, IngentaConnect, or any research site with access to the Journal of Systems and Software, you may also find the paper "An efficient data structure for dynamic memory management" by Chang, J. M., Daugherty, C.H. worthwhile. I haven't found a free version of the paper, sorry.

Portal
ScienceDirect
IngentaConnect
Some people are offended by the verifiable truth; such people tend to remain blissfully unencumbered by fact.
If you are one of these people, my posts may cause considerable discomfort. Read at your own risk.
User avatar
Combuster
Member
Member
Posts: 9301
Joined: Wed Oct 18, 2006 3:45 am
Libera.chat IRC: [com]buster
Location: On the balcony, where I can actually keep 1½m distance
Contact:

Re: physical memory management troubles

Post by Combuster »

xDDunce wrote:circumstantially, yes. if i happen to fill up say 100MB of memory with variable constructs, and then release 10MB's worth of ints/chars etc, then yes it could be a problem. but if a number of small hole are required after all of that, then these holes get filled and relatively quickly.
Wrong thinking. If you do 100 allocations/frees compared to 10, then any algorithm is slower. The problem with your approach, is that when you go from 10 to 100 allocations, your code suddenly takes 100x that time, rather than 10x as you might have been thinking...

Its like, you are doing bubble sort, while the rest knows how to do heapsort...
"Certainly avoid yourself. He is a newbie and might not realize it. You'll hate his code deeply a few years down the road." - Sortie
[ My OS ] [ VDisk/SFS ]
Post Reply