Problems completely understanding memory management.

Question about which tools to use, bugs, the best way to implement a function, etc should go here. Don't forget to see if your question is answered in the wiki first! When in doubt post here.
User avatar
zhiayang
Member
Member
Posts: 368
Joined: Tue Dec 27, 2011 7:57 am
Libera.chat IRC: zhiayang

Problems completely understanding memory management.

Post by zhiayang »

Not again. I hate violating some kind of unspoken forum etiquette, but I'm kinda racking my brains about this.

I can't find a concrete tutorial or article that explains, in a concrete and complete manner (with or without code examples, I think I'll be fine) about memory management, from physical, to paging, then the heap.


A bunch of questions:

1. What does it mean by 'A stack expands downwards'? Can somebody explain, using concrete memory addresses to me?
2. How is physical memory management done? I think it's something along the lines of simply keeping track of what you can and cannot use, splitting up that memory and passing pointers around?

3. I mostly get paging… *mostly*. I get the part where there's an address in memory, and you apply these flags to it… Like

Code: Select all

pd[1022] = pmm_alloc_page () | PAGE_PRESENT | PAGE_WRITE;
(Taken from the James Molloy tutorial on google code).

Problem: As I understand, pmm_alloc_page() returns a pointer to somewhere in memory. Is this what is going on: pd is somewhere in memory, and in the 1022'th index of that bit of memory, the pointer to the allocated memory is flagged with PAGE_PRESENT and PAGE_WRITE?

How, then, is paged memory managed?


4. What is the heap? (yes, I am clueless.) From what I know, malloc() can use physical memory allocation, however it has no way to be free()'ed. (right?) Whereas once a heap is setup, both malloc() and free() can be used to manage chunks of memory?

Forgive the dumbness of my questions, I am not strong with The Force.

Also, is there an IRC channel for osdev?
User avatar
JamesM
Member
Member
Posts: 2935
Joined: Tue Jul 10, 2007 5:27 am
Location: York, United Kingdom
Contact:

Re: Problems completely understanding memory management.

Post by JamesM »

Hi,
1. What does it mean by 'A stack expands downwards'? Can somebody explain, using concrete memory addresses to me?
A stack is a data structure with two operations only: "push" and "pop". "push" adds another item onto the stack, "pop" removes the last item added. Your question is related to how "push" and "pop" are implemented. Consider these two code fragments:

Code: Select all

int *stack;
void push(int a) {
  *stack++ = a;
}
int pop() {
  return *--stack;
}

Code: Select all

int *stack;
void push(int a) {
  *stack-- = a;
}
int pop() {
  return *++stack;
}
These code fragments do exactly the same job, but in the first the stack expands upwards - that is, as we push items onto the stack the stack pointer (int *stack) gets higher. The second expands downwards, so as you push items to the stack, the stack pointer gets lower.

Most architectures use a downwards expanding stack for several reasons.
2. How is physical memory management done? I think it's something along the lines of simply keeping track of what you can and cannot use, splitting up that memory and passing pointers around?
As with most problems, there are many ways to implement a physical memory manager (stack, bitmap, buddy etc) - each have a set of disadvantages and advantages. The problem of physical memory management itself however is trivially simple: you have a set of "pages" that are free, and a set of pages that are used. Expose an interface such that a user can request a page (that must be free, now mark as used) and return pages (were used, now free).

That is what the function "pmm_alloc_page()" in your code snippet does. It just returns the address of the start of a page in physical memory that was free. It does *not* guarantee that page is accessible (if paging is enabled, it may need to be mapped into the virtual address space before it can be accessed).
3. I mostly get paging… *mostly*. I get the part where there's an address in memory, and you apply these flags to it… Like
Problem: As I understand, pmm_alloc_page() returns a pointer to somewhere in memory. Is this what is going on: pd is somewhere in memory, and in the 1022'th index of that bit of memory, the pointer to the allocated memory is flagged with PAGE_PRESENT and PAGE_WRITE?
pmm_alloc_page() requests a free physical page from the physical memory manager. It returns the address of a free page in physical memory. That line is saying, in English: "Request a new physical page. OR its address with the "present" and "write-enable" flags, then store that computed value into the 1022'd index of "pd", which is an array of 32-bit integers".

This works, because at the time when that code is run, paging is actually disabled. So physical addresses == virtual addresses. That's why you can write to 'pd' just through a pointer to what would normally be inaccessible physical memory.
How, then, is paged memory managed?
I don't know exactly what you're asking here. A physical memory manager manages the availability of physical pages. A virtual memory manager maps these pages into the virtual address space. You may have another manager to partition up the available virtual address space.
4. What is the heap? (yes, I am clueless.) From what I know, malloc() can use physical memory allocation, however it has no way to be free()'ed. (right?) Whereas once a heap is setup, both malloc() and free() can be used to manage chunks of memory?
A heap sits atop the virtual and physical memory managers. It is given an area of virtual address space to operate in, and will allocate physical pages (using the PMM) to fill that space, mapping them in using the VMM.

It will then use some algorithm to carve that space up into manageable,trackable chunks to give back to callers of "kmalloc" and returning them with "kfree".

Cheers,

James
User avatar
Griwes
Member
Member
Posts: 374
Joined: Sat Jul 30, 2011 10:07 am
Libera.chat IRC: Griwes
Location: Wrocław/Racibórz, Poland
Contact:

Re: Problems completely understanding memory management.

Post by Griwes »

There is magical link at the top of the forum: "The OSDev.org Wiki - Got a question? Search this first!". There are pages for stack, memory management, paging and memory allocation; also, wikipedia has pages on - I believe - all of them. Furthermore, most of it is described in Intel Developer Manuals, ARM manuals or any manual for your target platform, so your questions from 1. to 4. qualify as RTFM.

As for osdev IRC channel - #osdev @ Freenode.
Reaver Project :: Repository :: Ohloh project page
<klange> This is a horror story about what happens when you need a hammer and all you have is the skulls of the damned.
<drake1> as long as the lock is read and modified by atomic operations
User avatar
JamesM
Member
Member
Posts: 2935
Joined: Tue Jul 10, 2007 5:27 am
Location: York, United Kingdom
Contact:

Re: Problems completely understanding memory management.

Post by JamesM »

Furthermore, most of it is described in Intel Developer Manuals, ARM manuals or any manual for your target platform, so your questions from 1. to 4. qualify as RTFM.
No, reference manuals and the wiki are not fantastic resources for explaining concepts. The encyclopaedia format isn't as good as a pointed answer (StackOverflow or this forum) or guides written in the tutorial style.
User avatar
Griwes
Member
Member
Posts: 374
Joined: Sat Jul 30, 2011 10:07 am
Libera.chat IRC: Griwes
Location: Wrocław/Racibórz, Poland
Contact:

Re: Problems completely understanding memory management.

Post by Griwes »

Maybe not, but still, after few seconds of typing a query into Google, you can get information about it written in "nicer" way than in manuals...
Reaver Project :: Repository :: Ohloh project page
<klange> This is a horror story about what happens when you need a hammer and all you have is the skulls of the damned.
<drake1> as long as the lock is read and modified by atomic operations
User avatar
JamesM
Member
Member
Posts: 2935
Joined: Tue Jul 10, 2007 5:27 am
Location: York, United Kingdom
Contact:

Re: Problems completely understanding memory management.

Post by JamesM »

Griwes wrote:Maybe not, but still, after few seconds of typing a query into Google, you can get information about it written in "nicer" way than in manuals...
Yes, but the questions asked weren't stupid - they're exactly what this forum is for. Please take your RTFM's to other questions that aren't described in such detail or are otherwise poorly specified.
User avatar
VolTeK
Member
Member
Posts: 815
Joined: Sat Nov 15, 2008 2:37 pm
Location: The Fire Nation

Re: Problems completely understanding memory management.

Post by VolTeK »

requimrar wrote:1. What does it mean by 'A stack expands downwards'? Can somebody explain, using concrete memory addresses to me?
JamesM wrote:weren't stupid
Assembler is required isn't it?
User avatar
JamesM
Member
Member
Posts: 2935
Joined: Tue Jul 10, 2007 5:27 am
Location: York, United Kingdom
Contact:

Re: Problems completely understanding memory management.

Post by JamesM »

VolTeK wrote:
requimrar wrote:1. What does it mean by 'A stack expands downwards'? Can somebody explain, using concrete memory addresses to me?
JamesM wrote:weren't stupid
Assembler is required isn't it?
No, it's not a stupid question. It is a direct question regarding understanding of a fundamental concept.

The stupidity of a question is generally unrelated to the subject matter and more the way the question is asked, and background reading done by the asker.

Stop being so bloody elitist.
User avatar
Kazinsal
Member
Member
Posts: 559
Joined: Wed Jul 13, 2011 7:38 pm
Libera.chat IRC: Kazinsal
Location: Vancouver
Contact:

Re: Problems completely understanding memory management.

Post by Kazinsal »

VolTeK wrote:
requimrar wrote:1. What does it mean by 'A stack expands downwards'? Can somebody explain, using concrete memory addresses to me?
JamesM wrote:weren't stupid
Assembler is required isn't it?
"How do I use the stack in assembly language" is a stupid question. "What are the concepts behind the stack" is not. Just because you're so used to the x86 stack and working with it so your whole operating system doesn't tumble over doesn't mean that questions about how stacks work are automatically stupid.

I'm going to go out on a limb here and say that none of these questions are stupid because they are theory questions and not "fix this please".
User avatar
zhiayang
Member
Member
Posts: 368
Joined: Tue Dec 27, 2011 7:57 am
Libera.chat IRC: zhiayang

Re: Problems completely understanding memory management.

Post by zhiayang »

JamesM wrote:
These code fragments do exactly the same job, but in the first the stack expands upwards - that is, as we push items onto the stack the stack pointer (int *stack) gets higher. The second expands downwards, so as you push items to the stack, the stack pointer gets lower.
So in visual terms… It's kinda like a floating stack of blocks that can magically hold on to things attached to it's bottom?

As with most problems, there are many ways to implement a physical memory manager (stack, bitmap, buddy etc) - each have a set of disadvantages and advantages. The problem of physical memory management itself however is trivially simple: you have a set of "pages" that are free, and a set of pages that are used. Expose an interface such that a user can request a page (that must be free, now mark as used) and return pages (were used, now free).

That is what the function "pmm_alloc_page()" in your code snippet does. It just returns the address of the start of a page in physical memory that was free. It does *not* guarantee that page is accessible (if paging is enabled, it may need to be mapped into the virtual address space before it can be accessed).
*my* code snippet…? I think you mean *your* code snippet. Unless of course you didn't actually write it.
But yes, now I get it.


pmm_alloc_page() requests a free physical page from the physical memory manager. It returns the address of a free page in physical memory. That line is saying, in English: "Request a new physical page. OR its address with the "present" and "write-enable" flags, then store that computed value into the 1022'd index of "pd", which is an array of 32-bit integers".

This works, because at the time when that code is run, paging is actually disabled. So physical addresses == virtual addresses. That's why you can write to 'pd' just through a pointer to what would normally be inaccessible physical memory.
So because paging is disabled, all the addresses are physical and you can just access it through pointers?

I don't know exactly what you're asking here. A physical memory manager manages the availability of physical pages. A virtual memory manager maps these pages into the virtual address space. You may have another manager to partition up the available virtual address space.
Well I guess it's just a PMM with virtual pages?
A heap sits atop the virtual and physical memory managers. It is given an area of virtual address space to operate in, and will allocate physical pages (using the PMM) to fill that space, mapping them in using the VMM.

It will then use some algorithm to carve that space up into manageable,trackable chunks to give back to callers of "kmalloc" and returning them with "free".
What's uh… 'some algorithm'?

Also, thanks for defending me. (:
User avatar
Kazinsal
Member
Member
Posts: 559
Joined: Wed Jul 13, 2011 7:38 pm
Libera.chat IRC: Kazinsal
Location: Vancouver
Contact:

Re: Problems completely understanding memory management.

Post by Kazinsal »

requimrar wrote:So in visual terms… It's kinda like a floating stack of blocks that can magically hold on to things attached to it's bottom?
Imagine you have some blocks. You take a red block and put it down. You put a green block on top of it, then a blue block on top of that one. Now, if you take any block off the top, everything's fine. But if you take any blocks out of the middle or bottom, the whole thing will fall over. So, if you want to take the green block back, you need to first take off the blue one, then the green one. You can then put the blue one back on.
User avatar
bubach
Member
Member
Posts: 1223
Joined: Sat Oct 23, 2004 11:00 pm
Location: Sweden
Contact:

Re: Problems completely understanding memory management.

Post by bubach »

I think it might be kind of confusing for a newbie to call chunks of physical memory "pages" before paging is involved. Unless you actually use paging, the physical allocations could be any size you desire and do not have to conform to paging friendly sizes. But maybe that's just me, since I do not use paging at all in my OS.
"Simplicity is the ultimate sophistication."
http://bos.asmhackers.net/ - GitHub
User avatar
JamesM
Member
Member
Posts: 2935
Joined: Tue Jul 10, 2007 5:27 am
Location: York, United Kingdom
Contact:

Re: Problems completely understanding memory management.

Post by JamesM »

requimrar wrote:
JamesM wrote:
These code fragments do exactly the same job, but in the first the stack expands upwards - that is, as we push items onto the stack the stack pointer (int *stack) gets higher. The second expands downwards, so as you push items to the stack, the stack pointer gets lower.
So in visual terms… It's kinda like a floating stack of blocks that can magically hold on to things attached to it's bottom?
See Blacklight's answer for a concise explanation, or see the wikipedia page for "Stack (Computing)".

As with most problems, there are many ways to implement a physical memory manager (stack, bitmap, buddy etc) - each have a set of disadvantages and advantages. The problem of physical memory management itself however is trivially simple: you have a set of "pages" that are free, and a set of pages that are used. Expose an interface such that a user can request a page (that must be free, now mark as used) and return pages (were used, now free).

That is what the function "pmm_alloc_page()" in your code snippet does. It just returns the address of the start of a page in physical memory that was free. It does *not* guarantee that page is accessible (if paging is enabled, it may need to be mapped into the virtual address space before it can be accessed).
*my* code snippet…? I think you mean *your* code snippet. Unless of course you didn't actually write it.
But yes, now I get it.
You quoted it, not me ;) And yes, I did actually write it.

pmm_alloc_page() requests a free physical page from the physical memory manager. It returns the address of a free page in physical memory. That line is saying, in English: "Request a new physical page. OR its address with the "present" and "write-enable" flags, then store that computed value into the 1022'd index of "pd", which is an array of 32-bit integers".

This works, because at the time when that code is run, paging is actually disabled. So physical addresses == virtual addresses. That's why you can write to 'pd' just through a pointer to what would normally be inaccessible physical memory.
So because paging is disabled, all the addresses are physical and you can just access it through pointers?
Yes.

I don't know exactly what you're asking here. A physical memory manager manages the availability of physical pages. A virtual memory manager maps these pages into the virtual address space. You may have another manager to partition up the available virtual address space.
Well I guess it's just a PMM with virtual pages?
There are two concepts here - often a "virtual memory manager" is referred to as the component that maps and unmaps physical pages to virtual pages. A "Virtual address space allocator" is another component that, like you suggest, carves up the virtual address space in much the same way as a physical memory manager might (albeit with differing requirements - often you need to allocate N bytes of virtual address space consecutively (something the PMM never needs to do)).
A heap sits atop the virtual and physical memory managers. It is given an area of virtual address space to operate in, and will allocate physical pages (using the PMM) to fill that space, mapping them in using the VMM.

It will then use some algorithm to carve that space up into manageable,trackable chunks to give back to callers of "kmalloc" and returning them with "free".
What's uh… 'some algorithm'?

Also, thanks for defending me. (:
dlmalloc, buddy, Slab are examples.
User avatar
zhiayang
Member
Member
Posts: 368
Joined: Tue Dec 27, 2011 7:57 am
Libera.chat IRC: zhiayang

Re: Problems completely understanding memory management.

Post by zhiayang »

Finally, revelation. But I still have a few more questions…

1. Higher half. Is it simply a matter of:
- Enable paging
- Map first x MB where kernel resides in memory (at y)
- Map x amount of virtual memory somewhere high up (3GB?) to y.
- Far jump to virtual y and execute the kernel?


2. What are the differences in paging in higher half and normal?
If I load the kernel at 3GB virtual, where in virtual memory should I allow for normal allocations? Anywhere (excluding things like BIOS of course) below the 3GB mark?


3. PAE Paging. It's similar to paging, just that there's another structure (PDPT) that is an array of page directories?

4. 64-bit. Assuming I load my 64-bit kernel as a module in GRUB, then read its ELF info (gonna have to read on that)… Then what?

The wiki says
- Remember to set up the higher half addressing!
What's that about? I'm going to assume that if my bootstrap 32-bit mini kernel is already in higher half, (following the Higher_Half_Bare_Bones tutorial) I'm going to have problems, since my loader is already somewhere there and is occupying the location where the actual 64-bit kernel will be?

Also: when they say
Set up Long Mode readying stuff (PAE, *PML4* etc)
,
What is PML4? I can't find a concise explanation telling me what it is.

Also: Do I have to enable (aka set the 31st bit in CR0), put the physical address of the *PDPT* into cr3, set bit 5 in cr4 and *then* jump to my kernel?

Also, what is this talk about PSE? Is it page size extensions, and how (and what) does in relate to PAE and long mode?
User avatar
zhiayang
Member
Member
Posts: 368
Joined: Tue Dec 27, 2011 7:57 am
Libera.chat IRC: zhiayang

Re: Problems completely understanding memory management.

Post by zhiayang »

Blacklight wrote:
requimrar wrote:So in visual terms… It's kinda like a floating stack of blocks that can magically hold on to things attached to it's bottom?
Imagine you have some blocks. You take a red block and put it down. You put a green block on top of it, then a blue block on top of that one. Now, if you take any block off the top, everything's fine. But if you take any blocks out of the middle or bottom, the whole thing will fall over. So, if you want to take the green block back, you need to first take off the blue one, then the green one. You can then put the blue one back on.
Well yes… but it's kinda hard to imagine a stack that expands downwards… physically. Is it simply a matter of reversing the gravitational direction? :p
Post Reply