OSDev.org

Ok, sorry if this is a sudden wave of question .

Now I'm going to be straight out.

I don't know anything about memory management
I don't know anything about how the stack works.
I really don't know when I need to allocate memory (using malloc or an equivilent of it) and how much to allocate.

I'm thinking leave the allocation of seperate programs to the discretion of the programmer (I don't know if memory must be allocated on program startup)

I need to know hot to differenciate how the kernel handles memory and how the programs request memory from the kernel.

I might be totally off or I might be right on the dot, I don't know really. If anyone could post some insight for a total beginner I would be very greatful.

First, since you say you don't know anything about it, check out the articles on Bonafide and in the Wiki.

Second, the stack works in a straightforward way, you stick it someplace using ESP, then whenever you push something on the stack it grows downwards (towards 0), within SS of course.

Third, you allocate memory whenever the allocation cannot be anticipated (if you know you need something to start with then you use the BSS, but for things like the process and thread data for example, you don't know how many you need so rather than allocating 5000 at compile time, you allocate as you go - also applies to the memory bitmap, unless you want to reserve enough space to store a bitmap to cover 4GB {130KB bitmap} [or 64GB {8MB bitmap} if you support PAE] even if the user only has 64MB {2KB bitmap}, you allocate only what you need)

Fourth, It doesn't work that way, the program requests memory and the kernel (or memory service) has to find a free page of memory to give the program, it's also probably a good idea to load the page containing the entry point into memory at startup (although you can just let it pagefault and load it then).

Fifth, The Kernel's internal memory is the heap, process memory comes from the physical and virtual managers (also in the Kernel) which track and map chunks of physical RAM into processes (or the Kernel heap if it needs more).

A note about how processes work on existing OSes: The programs data is mapped in a virtual address space someplace, usually low (Windows defaults at 4MB) the program's heap is at the end of the BSS, then there is a wide gap between that and stack at the end of the virtual memory (2GB Virtual on Windows).
[tt]
|--0-4MB--|---4-xMB---|---x-yMB---|---y-2GB---|---2-3GB---|---3-4GB---|
NULL Pointer Program Empty Stack Libs/DLLs Kernel
Trap[/tt]

<offtopic target="AR">
the [ code ] is fine for posting code (avoid smileys and other [ ... ] stuff), but it doesn't render as typewriter font for everyone. When posting ascii arts, consider using [ tt ] instead
</offtopic>
tt:
[tt]
+-----+
| x |
+-----+
[/tt]

OT@pype: why? it doesn't matter, does it? maybe we like the nice blue background?

Thanks! I'm reading the Memory Management 1 article and It say's to find my total physical memory and I remember reading somewhere that GRUB stores that somewhere?

In the Multiboot data structure. Check the GRUB / Multiboot documentation for details.

http://www.mega-tokyo.com/forum/index.p ... eadid=7383

in that post curufir has written a small tutorial about how to create a malloc function.

The OSFAq also has tips and algorithms for memory management which points towards many interresting posts (just added poseidon's suggestion)

So I need to setup paging before I can implement malloc() and free() .

malloc is used when you cannot anticipate how much memory you need so you don't allocate a general amount just how much you need at that specific time. Ok.

So how do you know how much you need when the need arises (to allocate memory that is) . You would also have to justify which memory is in use but I assume paging does that for you so when you finish execution you can free() whatever you allocated. Correct?

Then again I've never done anything like this so I may be way off.

another thing in the FAQ

Well, let's try to make unix process simple.

Your application has a "heap" model, that is, a bunch of virtual memory which the kernel 'knows' to be used by the application. malloc() and free() are functions in a library that eases management of what occurs within that heap. And being in user-space, those functions don't care about whether there is paging or not. All they do is return you a block of free (virtual) memory from a pool of free (virtual) memory.

Now, it may happen that the pool malloc/free becomes too small for the application's needs. In that case, the library will ask the kernel to extend the pool by a system call (usually 'brk' or 'morecore').

Paging is just a way to support that 'morecore' call. If a user process no longer use a collection of pages, they'll be swapped out for another use.

Now, within the kernel, you may need dynamic memory allocation (sometimes, you can't just use tables of fixed size for everything). Therefore, kernel programmers like to have kmalloc and kfree that will operate on a heap within kernel space. Most of the time, that heap cannot grow and kmalloc/kfree are tighly bound to paging since you may not want to rely on a swapping daemon *within* kernel space.

Hope it cleared things, but from your questions, i guess you might need to dive into books such as Tannenbaum's one or get more practice about system programming before you're ready to address this one ...

malloc() and free() are not functions used by applications rather than for use with system functions that require memory where that amount is unknown. Correct? Ok.

Application has the heap which is a lump of virutal memory they use to do thier own thing. When thier lump of virtual memory becomes to small for thier needs they ask for more then I grant them more. Overall the Kernel doesn't safeguard anything but it'self in a way and will only offer help if the application asks. (In a nutshell)

Paging manages the free pages and used pages and pages not in use into free pages. IIRC All malloc does is select the first page which is big enough for thier request?

Thanks for putting alot of this in perspecitive for me.

Warrior wrote: malloc() and free() are not functions used by applications rather than for use with system functions that require memory where that amount is unknown. Correct? Ok.

[tt]O_o -- <confused>[/tt]

please. malloc() and free() *are* userlevel things, involved in application-managed heap. whatever you implement in kernel should *never* collide with standard user-level library names or we're all going to lose ourselves into chaos...

All malloc does is select the first page which is big enough for thier request?

All pages have equal size of 4KB what's the point of picking a page "big enough" ??

So how do you know how much you need when the need arises (to allocate memory that is) .

Let's say you want to keep the content of a directory in your system. A directory can contain any number of items and each item has a name ranging from 1 to 255 characters.

When you write code for reading the directory, you can't tell how much memory you'll need, but when you call it, you can tell how much entries the directory contain and what's the length of each string, so *that* tells you how much memory you need to store the directory information.

You would also have to justify which memory is in use but I assume paging does that for you so when you finish execution you can free() whatever you allocated. Correct?

[tt]>_< -- **smashed**[/tt]

beware, beware, beware ... it's rather easy to return all the pages a user process contains when the process exits. that's indeed just sweeping page tables and releasing all the page frames that are present.

but that has *nothing* to do with malloc/free. Let's say for the purpose of file name lookup, you have malloc'ated a buffer that should contain the directory info. The only way that buffer can be freed is to insert a "free(buffer)" in your code at the proper place, after ensuring nothing points towards that buffer.
Not freeing the buffer will mean your application (or kernel if your're playing with kmalloc/kfree) slowly eats all the heap space available...

Hope it made things clearer ...

Warrior wrote: malloc() and free() are not functions used by applications rather than for use with system functions that require memory where that amount is unknown. Correct? Ok.

Nope, not correct. They are the functions used by applications, and you pass the exact amount of memory required to malloc().

Application has the heap which is a lump of virutal memory they use to do thier own thing. When thier lump of virtual memory becomes to small for thier needs they ask for more then I grant them more.

Not quite, but almost. See below.

Paging manages the free pages and used pages and pages not in use into free pages. IIRC All malloc does is select the first page which is big enough for thier request?

No.

You have the kernel, which handles the physical and virtual memory (which includes paging). It also manages the setting up of new processes, and offers the system services.

An executable image consists of:

• .text section which holds the executable code,
• .data section which holds data for which size and value was known at compile time, and
• a number defining .bss size for data for which only size but not value was known at compile time.

When a process is set up, the address space is filled with the executable image, with the .bss size reserved and initialized to zero. In addition, the stack is set up. In addition, somewhere in the address space there's the C runtime provided by the system. malloc() and free() are part of that C runtime. Both function modify the heap, which is empty at start.

The C runtime holds an internal pool of memory. Let's just assume the traditional approach, where that memory starts at the end of .bss, with the stack growing downward from the top of the address space. Not all of the 4G address space are actually mapped to physical or virtual memory, however. (Or each process would enlargen the swap space requirement by a full 4G.) The C runtime just "knows" where the "available" addresses end.

This is, you might note, still completely ignorant of paging et al. - for the runtime, memory is contiguous. (And it must be, because applications customarily require contiguous memory in excess of a single page size.)

If your application now comes to the point of requiring memory that wasn't known at compile-time - like, allocating a buffer large enough to hold a certain bitmap file - it calls malloc(). If the chunk requested can be satisfied from the internal pool held by the C runtime, the runtime marks that part of its internal pool as reserved and returns a pointer to it. The kernel is blissfully unaware of this.

If the request cannot be satisfied by the internal pool held by the runtime, the runtime calls the kernel to enlarge that internal pool, i.e. increase the amount of memory actually mapped anywhere, physical or virtual. (The function is called sbrk() in POSIX, morecore() or something like that in other OS'.) The kernel does whatever is necessary - paging out dormant memory areas to free up physical memory, for example - and attaches additional pages to the process' address space. The kernel returns the new limit to the available heap to the C library, which in turn returns a pointer to a large-enough chunk of memory to the application.

On the reverse, when the application returns memory chunks to the C runtime with free(), at some point the C runtime holds lots of unused memory which should be returned to the kernel, so the kernel could pass physical memory to other processes and shrink the used swap space.

Doing all this in a fast and efficient manner, that's the art of memory management.

I hope I clarified the how-to a bit.

Pype beat me to it.

Sorry if I'm reviving a dead topic .

Okay malloc allocates memory when it cannot be anticipated so that you don't have to use a fixed legnth and can avoid exceeding that length and accidently (or intentionally) overwriting other things in the memory.

The C runtime lib allocates memory fine and the user actually uses malloc when the need arrises that the runtime library cannot anticipate.

The kernel has a seperate malloc function : kmalloc() for allocation within the kernel.

I have thought this over and think is my current understanding.