mem manager

[slacker]

i am trying to write a memory manager. i was wondering how to deal with the fact that programs have a set starting location like 0x7c00 or 0x100000 but with a memory manager these apps can be loaded into a variety of random locations. are there any methods of getting around this. i'm not using paging.

[Pype.Clicker]

there are various solutions (i think i have detailed them in a previous thread, but i can't remember which by now ... i should search for it ...)

What you could for instance do is :
- assume each program will have its very own code & data segments and thus that it begins at offset 0x1000 (keeping 1 page for catching NULL pointers) within its segment area.
- assume your programs are written like PIC libraries (read the ELF specification to get an idea of how it works and which tricks you should provide to offer this to your programs)
- assume nothing at all and make your programs carry a relocation list that will help the system to "patch" it at load-time so that it fits its new location...

[Tim]

Another option:

- use paging

That way each application has its own set of addresses, and it can't interefere with any other applications' memory.

[Solar]

The relocation list is the way classic AmigaOS (which did not use an MMU at all) handled these things. If you want to walk that road, you might want to look for AmigaOS internals manuals, like the ROM Kernel Reference Manuals. That might require a bit of searching though.

[Pype.Clicker]

relocation is also the way DLL works, iirc, except DLL are usually compiled for a "favourite target address" at which they can be loaded without requiring relocation.

[Solar]

Just checked the AmigaOS way regarding licenses (HUNK format):

- license is requested from kernel;
- kernel loads library into memory, relocates;
- kernel returns to user space, passing "base address" of library;
- library has an offset table - each library function has an associated offset which is constant across lib versions; at that (negative) offset from library base is the starting point of the function (allowing function lengths to differ from lib version to lib version, since offset into offset table remains constant);
- user space uses library base (returned by kernel) and offset (known by library include / stub) to jump into lib functions.

On a side note, this handling allows a v2 library to overwrite v1 functions with the same prototype, and offer either v1 or v2 functionality depending on which version was requested from the kernel, by virtue of returning with a v1 or v2 library base... there's some beauty in this, but I don't know if this can be salvaged for ELF or other, more "modern" targets.

[Neo]

I have another memory management question.
I have implemented a basic memory manager . Right now the kmalloc() allows the kernel to create linked lists,binary trees etc. All this memory is allocated from the kernel heap starting at 0xd000_0000 to 0xe000_0000, now i have a doubt. If i want to create another process i have to allocate memory for the process, how do i do this? if i call kmalloc() this only allocates from the kernel heap. do i have to create the process in the kernel heap? if not then what is the actual way to do this? and is the kernel heap only used for storing control info etc?
Any help appreciated

[BI lazy]

You have to build an adress space for the process: allocate a pagedirectory (ask the phys. memmgr for a page of memory), fill in the kernel adress space, map the page directory to the bottom of the adress space and hand out the pagedirectory to a kind of process manager which keeps also a list of allocated memory. It should enter the page directory somewhere in a kind of process struct, from where it is picked at context switch. (simply a cr3 reload)

have the initial process eip point to a well known invalid adress - aka 0xdeadbeef *gg*.

If you have a page fault handler installed, have it tell this process manager: hey, there is a pagefault and it stems from a newly created process. The process manager tells memory manager who installs the rest of the adress space (since it can run in the faulting process adress space) and copies the process image from a buffer to the proper location and finally corrects the value for eip - and lets go again. and see, the process will be executed. I've sketched a rough scheme of how it could be done. In a micro kernel, all this includes some messages and some split up of work.

Mark: memory management not only includes heap management but also process adress space management.

stay safe

@the gurus: plz feel free to correct me if i am in err for i am only the wizards apprentice ... and beware of the ghosts I've summoned ...

[Pype.Clicker]

The memory manager is usually providing many services. kernel-dedicated heap is one of them (and splitting it between process-specific kernel heap and system-wide kernel heap may sometimes be considered), allocating virtual addresses for 'objects' like programs image, stacks, shared libraries, mapped files etc. is another one.
Finally providing physical memory for virtual addresses is a third kind of service.

I have opted for a double-binary tree management of the virtual address space (to know which areas are free and which are used for some objects) i think Linux uses tree aswell. For other systems, honestly, i don't know too much ...

[BI lazy]

Well, I use tree memory management at the moment too.

But I only keep record of *used* areas on a per-process base. Free areas are simply handed back to the stack of phys. pages.

so, my memory manager service is responsible - in cooperation with the pager, which in future will be responsible for loading executables into the process adress space page by page - for creating processes and keeping track of their adress space. It's merely a linked list of process objects each of whose has attached a binary tree of allocated adress areas.

I am still unsure of where to put the block cache. It would mean to split the filesystem management in a pure management process and a block cache driver which loads the required blocks from the block device in question. This could be done by the pager f. ex. It *would* ease file access for user processes, since a certain part of copying would vanish. - lots of decisions, really. Maybe it would be better to leave the *process loading* to the pager. such blocks needn't be kept in memory, would they?

stay safe.

[Pype.Clicker]

Well, I use tree memory management at the moment too.

But I only keep record of *used* areas on a per-process base. Free areas are simply handed back to the stack of phys. pages.

hum, no, when i said 'free areas', i was talking about "virtual addresses that has not been affected to a specific object so far".
Those are in the "per-address" tree (so that you can easily merge two contiguous areas at freeing time by looking at the area's neighbnour in the address tree) and in the "per-size" tree (which allow you to quickly find an area that will hold your 4.3MB shared library you'd like to load)

It's merely a linked list of process objects each of whose has attached a binary tree of allocated adress areas.

I'm unsure i get it correctly ... if you need to find, for instance, the object responsible for a page fault, what would you do ?

tree::find(list::lookup(process.objects, addr),addr) ?

Maybe it would be better to leave the *process loading* to the pager. such blocks needn't be kept in memory, would they?

You mean 'should i cache the memory-mapped executable when i load a process?' ... i'd say "for sure" ... Just thing at how many "exec('/usr/bin/gcc')" a single "make kernel" command may issue ... you don't want to reload them everytime, do you ?

Btw, i'm just starting to sketch the virtual memory (swapping, caching and the like) by now, so take my words with the appropriate scientific scepticism

[BI lazy]

Roughly spoken yes: I do so: First fetch the process by the Process id from the list. Then go and check its vmm areas.

its first: process = find_process(pid);
then: adress_area=find_adress_area(process.vmmalloc,addr);
in such a way i handle it. Veeery complicated, with one word.

I 've even split up memory management between service and driver so that there is an independent process which does the management stuff. Of course it would have been easier if I've put this in kernel space: the actual process is the faulting one. Just look in its tree of allocated memory for the adress in question.

But what would life be if it were easy? *gg* Nooo fun at all.

[BI lazy]

btw: the keeping of blocks of executables in memory: I've thought about this too - but what, if one *shares* the *text* part of the executable between all the processes? Oh sweet memory management ... I've got a silent feeling as if several design issues are simply put in the wrong direction...

because a block of data once fetched from the disk is relatively independent from the filesystem. BEcause of this it *is* possible to keep the block cache in kernel space to ease the mapping of files. To ease the loading of executables. you just need to keep some information in such a block struct: the device number and/or the superblock.

[slacker]

if pages are of a fixed length(4mb max), how can programs larger than this be written and still work since each new page has an offset 0x00.?

[Tim]

A standard page is 4KB long. The whole 4GB address space is divided into pages. So an individual page can have any address that is a multiple of 4096 bytes. The same applies to 4MB pages, except that these must have an address that is a multiple of 4MB.

Don't think of pages as being the same as segments. A segment is a fixed-size container for an object. A page is the term given to a part of the address space. You could refer to individual bytes instead of pages, except that it is 4096 times as hard to keep track of individual bytes as it is pages.