OSDev.org

Hey guys!

My kernel is getting a little more advanced now, and I just wanted to hear a few opinions about my memory design (which is the only thing that is very different to the most approaches I see around here). First thing to know, my kernel is supposed to run a virtual machine. That means that I don't need process security, so all my processes run in ring 0 (what also gives me better portability to other architectures, later).

When it comes to memory management, I do this:
- there is a physical memory manager that gives me free page-aligned chunks for whatever I need them. this manager only gives me memory above ~16MB (you'll read further on about why it is like this) and reads what is available from the GRUB memory map.
- there is a virtual memory manager, which implements paging. it creates one global directory, which is shared by all processes. the reason for this is, that i want all my processes to share the same memory and like this have direct access to (synchronized) memory areas.

My memory is then split in the following areas:
LOWER MEMORY 0x00000000 -> 0x00100000:
the lower memory megabyte is identity-mapped. there is a so-called "lower memory allocator" that operates in the free space there, and gives me for example stacks for the VM86 processes

KERNEL MEMORY 0x00100000 -> ca. 0x01500000:
this memory is also identity-mapped. it contains the kernel binary, the ramdisk modules and there are (currently) ~16MB reserved as a kernel heap. this kernel heap is managed by the "kernel memory allocator", that gives me the kernel stacks for processes. the end of this area is calculated from a linker symbol (endKernel) + size of modules + size of kernel heap.

HEAP AREA ca. 0x01500000 -> whats requested:
this is where the global heap starts. the area from here on is virtual-mapped and grows everytime sbrk() wants more heap. then the sbrk (syscall) asks the physical memory manager for another page and appends it to get a contiguous memory area.

DMA AREA area from 0xFFFFF000 growing downward:
here I'm creating virtual mappings for DMA. for example, if the linear framebuffer of VESA wants to be at 0xFC000000, I'm calling my function: createDma(0xFC000000, sizeOfFrameBuffer). this function then returns me a virtual address that points to exactly this area and i can write my pixels. (only con imho: its currently not dynamic, so any mapped DMA-area cannot be unmapped / is this a con?) also, i have to add a check here, if the physical addresses i want to access via DMA area already mapped as memory, if so, i have to copy the pages somewhere else and update the mapping (shouldn't be a problem, but might get slow because the entire directory must be scanned. just a little fiddling).

Now my question is mainly - will this work out for anything that might come? By now I did not have any case that would interfere with this my model, DMA is possible, VM86 is possible (though the ~470KB might run out if somebody does weird VM86 calls - what shouldn't happen^^, they are mainly for VESA).

Any suggestions / critics / warnings / clarification of possible problems are greatly appreciated.

Thank you!

Apart from my general dislike for the sbrk() interface, which makes you inflexible for little reason: Why do you have separate virtual memory areas for a kernel heap and a second heap for other ring 0 code?

max wrote:- there is a physical memory manager that gives me free page-aligned chunks for whatever I need them. this manager only gives me memory above ~16MB (you'll read further on about why it is like this)

You forgot to explain this.

this is where the global heap starts. the area from here on is virtual-mapped and grows everytime sbrk() wants more heap. then the sbrk (syscall) asks the physical memory manager for another page and appends it to get a contiguous memory area.

What does "syscall" actually mean in this context? I thought you only have a VM running, which is part of the kernel?

this function then returns me a virtual address that points to exactly this area and i can write my pixels. (only con imho: its currently not dynamic, so any mapped DMA-area cannot be unmapped / is this a con?)

First of all, this isn't DMA, but MMIO. If it's only about MMIO, then it's probably not a problem for the forseeable future as you're probably not going to implement PCI hot (un)plug anytime soon.

But I suppose you use the same mechanism for real DMA as well, so: Yes, it is. It is probably also a big con that you can't DMA from/to any buffer on the heap. A typical use case is that your application has a buffer on the heap and wants to write it to disk or send it over the network (same for read/receive, of course). You don't want to copy all the data around.

also, i have to add a check here, if the physical addresses i want to access via DMA area already mapped as memory, if so, i have to copy the pages somewhere else and update the mapping

When would this happen? For MMIO areas, if you had it mapped as memory, that's a bug in your physical memory manager. For DMA, it doesn't make a lot of sense to specify a physical address anyway except for DMAing from/to an already existing buffer - and then copying it away doesn't make sense.

Kevin wrote:Apart from my general dislike for the sbrk() interface, which makes you inflexible for little reason: Why do you have separate virtual memory areas for a kernel heap and a second heap for other ring 0 code?

The reason is, that the kernel heap is identity mapped, and so the process stacks are too: like this I can disable paging (for example to expand the real heap) and still have the process running without messing up memory somewhere where it thinks it's stack is.

Yeah, sbrk might not be optimal, but I am able to have a contiguous area so it works. Maybe I'll switch to mmap later

Kevin wrote:

max wrote:- there is a physical memory manager that gives me free page-aligned chunks for whatever I need them. this manager only gives me memory above ~16MB (you'll read further on about why it is like this)

You forgot to explain this.

Yes sorry. The point above should explain this too, I use 16MB as identity-mapped kernel heap to allow disabling paging.

Kevin wrote:

this is where the global heap starts. the area from here on is virtual-mapped and grows everytime sbrk() wants more heap. then the sbrk (syscall) asks the physical memory manager for another page and appends it to get a contiguous memory area.

What does "syscall" actually mean in this context? I thought you only have a VM running, which is part of the kernel?

I'm using "syscalls" to call kernel-functions that may not be interrupted/are not reentrant, like expanding the "user" heap (what changes the global page directory).

Kevin wrote:

this function then returns me a virtual address that points to exactly this area and i can write my pixels. (only con imho: its currently not dynamic, so any mapped DMA-area cannot be unmapped / is this a con?)

First of all, this isn't DMA, but MMIO. If it's only about MMIO, then it's probably not a problem for the forseeable future as you're probably not going to implement PCI hot (un)plug anytime soon.

But I suppose you use the same mechanism for real DMA as well, so: Yes, it is. It is probably also a big con that you can't DMA from/to any buffer on the heap. A typical use case is that your application has a buffer on the heap and wants to write it to disk or send it over the network (same for read/receive, of course). You don't want to copy all the data around.

Oh. I confounded these two. Okay, I see the problem... DMA needs a contiguous physical space, right? So I need to find contiguous, free chunks of size X and give them to my device, and then map them somewhere in my virtual directory?

Kevin wrote:

also, i have to add a check here, if the physical addresses i want to access via DMA area already mapped as memory, if so, i have to copy the pages somewhere else and update the mapping

When would this happen? For MMIO areas, if you had it mapped as memory, that's a bug in your physical memory manager. For DMA, it doesn't make a lot of sense to specify a physical address anyway except for DMAing from/to an already existing buffer - and then copying it away doesn't make sense.

Hmm, I thought that maybe there are MMIO areas that the multiboot memory map doesn't report to me. Then it could happen that my physical page allocator gives these chunks out... can I be sure that the memory map is correct and reports every MMIO-address to me?

max wrote:like this I can disable paging (for example to expand the real heap) and still have the process running without messing up memory somewhere where it thinks it's stack is.

If you ever need to disable paging once your OS is initialised, you're doing something seriously wrong.

I'm using "syscalls" to call kernel-functions that may not be interrupted/are not reentrant, like expanding the "user" heap (what changes the global page directory).

That is, a function that disables interrupts is called a syscall in your OS?

Kevin wrote:But I suppose you use the same mechanism for real DMA as well, so: Yes, it is. It is probably also a big con that you can't DMA from/to any buffer on the heap. A typical use case is that your application has a buffer on the heap and wants to write it to disk or send it over the network (same for read/receive, of course). You don't want to copy all the data around.

Oh. I confounded these two. Okay, I see the problem... DMA needs a contiguous physical space, right? So I need to find contiguous, free chunks of size X and give them to my device, and then map them somewhere in my virtual directory?

Depends on the hardware whether you need a single physically contiguous memory buffer. Modern hardware allows you to use scatter/gather lists

But bascially, yes, you need physical addresses and some physically contiguous memory (even if it's just a single page). Except that, as I said above, you generally don't want to allocate new buffers, but use existing heap buffers if at all possible.

Hmm, I thought that maybe there are MMIO areas that the multiboot memory map doesn't report to me. Then it could happen that my physical page allocator gives these chunks out... can I be sure that the memory map is correct and reports every MMIO-address to me?

That would be a BIOS bug then. And a serious one - I think you can imagine what may happen when you treat some memory as normal RAM while you're really writing random stuff to device registers...

Kevin wrote:If you ever need to disable paging once your OS is initialised, you're doing something seriously wrong.

Otherwise I would have to map my page directory and all page tables into virtual space to access them later.. and as expanding the heap doesn't happen often, is it really bad?

EDIT: Okay, there is also the method of letting the last entry of dir/table point to itself - but then I can't have a contiguous area anymore..

Kevin wrote:That is, a function that disables interrupts is called a syscall in your OS?

I kept it this way to keep the possible option open to be able to implement user mode processes if I ever need them for one reason.

Kevin wrote:Depends on the hardware whether you need a single physically contiguous memory buffer. Modern hardware allows you to use scatter/gather lists

But bascially, yes, you need physical addresses and some physically contiguous memory (even if it's just a single page). Except that, as I said above, you generally don't want to allocate new buffers, but use existing heap buffers if at all possible.

Okay, I can implement this.

That would be a BIOS bug then. And a serious one - I think you can imagine what may happen when you treat some memory as normal RAM while you're really writing random stuff to device registers...

Alright, then I can leave my MMIO-mapping this way

max wrote:Otherwise I would have to map my page directory and all page tables into virtual space to access them later..

So what? Having everything mapped isn't hard and doesn't hurt.

I kept it this way to keep the possible option open to be able to implement user mode processes if I ever need them for one reason.

That's fine, just trying to get the terminology right for your OS.

Kevin wrote:So what? Having everything mapped isn't hard and doesn't hurt.

But: my physical page allocator also operates on the physical memory - when initializing my system, I read the memory map from GRUB, and store the address of every free page in something I call the "chunkTableDirectory", which then contains "chunkTables" (yep, this is just the same model as used in paging :'D). And when there is a chunk requested from somewhere, I take the address of a free chunk from this directory and then set it to 0 in my chunkDir.

Now, I would have to map
1. the chunk table directory
2. all chunk tables
3. the page directory
4. all page tables
somewhere to virtual memory - but how should I keep track of which page table belongs where in the page directory, and also which chunk table belongs where in the chunk directory? There I would again have to have some kind of map containing dirIndex+pageIndex->virtual address-mapping...

Kevin wrote:That's fine, just trying to get the terminology right for your OS.

Okay, it just sounded a little like this memory model was the worst you have ever seen

EDIT: Oh and a syscall in my kernel is not just a function that disables interrupts, its a int 0x80

max wrote:but how should I keep track of which page table belongs where in the page directory, and also which chunk table belongs where in the chunk directory? There I would again have to have some kind of map containing dirIndex+pageIndex->virtual address-mapping...

Well, the traditional self-mapping page directory does the job for the paging structures and takes 4 MB of virtual memory, but no additional physical memory. The chunk table is something that you only ever want to address by virtual addresses anyway, so its physical address doesn't even matter and it's just a random array that can be anywhere. (Not quite sure why you need a two-level structure there, do you have any major sparse areas?)

Okay, it just sounded a little like this memory model was the worst you have ever seen

Let me phrase it like this: I've seen enough broken code that this isn't anything extraordinary.

EDIT: Oh and a syscall in my kernel is not just a function that disables interrupts, its a int 0x80

Right, a function that disables interrupts and is implemented inefficiently then.