Kernel and its components' location in memory

[XCHG]

I have started writing a memory manager using paging. My kernel is located at the 1MB mark and I have set some rules pertaining to how many bytes the kernel should use, where the Page Directory Entries and Page Table Entries should be placed. I was wondering if you guys could give me some pointers on whether this scheme is good enough or not. I am also really concerned about the future of the kernel and not just the things that I might need now.

Kernel: MB 0 to MB 8 is allocated for the kernel and its components. (Length=8MB)
Memory Manager and its components: MB 8 to MB 12 (Length=4 MB)
Page Directory Entries: MB 12 to MB12+4KB (Length=4KB)
Page Table Entries: MB12+4KB to MB 16+4KB (Length = 4MB)
Free space for core device drivers: MB16+4KB to MB20 (Length = 4MB-4KB)

With this scheme, the Operating System would take 20 MB of RAM and the rest will be allowed to be allocated and/or deallocated by the memory manager and its related components. I have not yet coded my memory allocator and deallocator so I have not yet given MB8 to MB12 any schemes yet. I’d really appreciate it if you guys could tell me whether this is an optimal way of dividing the memory or not. Thank in advance.

Code: Select all

  ; 8 Megabytes are allocated for the Kernel             0x00000000 ... 0x007FFFFF     (8 MB)
  ; 4 Megabytes are allocated for Memory Manager Maps    0x00800000 ... 0x00BFFFFF     (4 MB)
  ; 4 Kilobytes are allocated for Page Directory Entries 0x00C00000 ... 0x00C00FFF     (4 KB)
  ; 4 Megabytes are allocated for Page Table Entries     0x00C01000 ... 0x01000FFF     (4 MB)
  ; (4 Megabytes - 4 Kilobytes) of free space            0x01001000 ... 0x013FFFFF     (4 MB-4KB)

[AJ]

Hi,

The scheme looks generally workable, but there are a few things to think about if you are concerned about future expansion (I don't know what the ultimate goal of your os is, so you may already have thought about some of these/some may not apply):

• Where do you want application space. With multitasking, it can ve *very* useful to have kernel space mapped in to every application space. In your scheme, this means linking applications at 20mb+ (this may not be a problem).
  Are the values you mention physical, virtual or identity mapped? If physical, you may not want to assign 4mb for page tables at the outset.
  Can you compile your kernel in to an exe format with relocation information? If so, you needn't leave such a large chunk for the kernel itself - you can just assign the correct amount of RAM.
  You leave 4mb for your memory manager. If you are using a bitmap-based frame allocator this could be *far* too much.

Overall, my advice would be to leave everything as dynamic as possible. My kernel is higher-half doesn't care whether it is linked at 0x80000000 or 0xC0000000 (or any 4kb aligned boundary in between) - as long as it can find the symbol for its entry and end points, it locates everything else in virtual RAM based on that - including its stack and heap.

Wherever possible, I then assign all other memory from the heap, so when the kernel is running without debug output, I don't actually know where half my essential data structures are. Of course, the major advantage is that if I need to change the kernel location because of something I didn't design for, I just have to change its entry point.

Hope this helps and that I wasn't telling you stuff you already knew!

Cheers,
Adam

[Kevin McGuire]

You could push all the free memory into a physical page manager. Then tell the physical page manager to reserve so many pages for the page directories and tables. Tell it to get rid of the pages the kernel is loaded on, and also let it get rid of the pages it uses it's self.

Then have the virtual memory manager only allocate those special pages from the physical memory manager that you reserved earlier for the page directories and tables. Make the virtual memory manager always identity map these directories and tables, and only in the kernel space < 2GB. Then switch into virtual space with the kernel and allocate anywhere in between zero to two gigabytes for pages to hold the drivers and other things so that you can prevent having to have a set location for everything.

[muisei]

How do you benefit from mapping your kernel in application space?

[Kevin McGuire]

When a application makes a system call say for "open(...)" it only has to issue a interrupt to change the code descriptor. If I did not map the kernel into the user space (application space) the entire address space would have to be switched which would be setting cr3 to something different. This takes a huge amount of cycles which IIRC is why user space linux suffers in performance.

Not only that but once inside the kernel the system calls now have direct access to the user space process's memory to do such things as copy data or read in a buffer passed as a argument in the system call.

void syscall(void *buffer){
// the kernel has direct access to buffer instead of having to switch
// cr3 just to gain access to it or modify the page directory and tables
// to get access to it.
}

g_pmm_stat_vmm_allocated=0x3000(bytes)
The memory the virtual memory manager took to manage the current address space right now.
g_pmm_stat_allocated=0x3D000(bytes)
0x4000 bytes of this total are in the heap right now.
g_pmm_stat_internal_allocated=0x3000(bytes)
The memory the pmm took to manage thirty-two megabytes of memory.
g_pmm_stat_free=0x1F59000(bytes)
What is left from the thirty-two megabytes of memory after my operating system, BIOS, and memory mapped I/O kicked in.
g_pmm_stat_kernel=0x8000(bytes) [with -Os]

My statistics are not subtracting the g_pmm_stat_kernel from the g_pmm_stat_free, but never the less it really is not that bad of memory usage for thirty-two megabytes. Of course no threads running which even if they did should only take a couple of pages of memory for something small.

The rest is completely dynamic, while right now I have only told the memory managers to reserve one hundred pages for the vmm. Then reserve the lower 268.435456 megabytes of virtual memory for the kernel space, while user space gets the remaining.

I just stick them in there.

Code: Select all

/// All in page counts and page addresses.
/// kernel space
#define VMM_K_MINPAGE           0x100
#define VMM_K_MAXPAGE	       0x10000
// identity mapping for page directories and tables.
#define PMM_VMM_PAGE		0x0
#define PMM_VMM_PAGECOUNT  0x100
// the entire space after the identity mapping region.
#define PMM_PAGE		     0x100
#define PMM_PAGECOUNT		(0xfffff - 0x100)

[iammisc]

i think you should dynamically allocate pages using a physical memory manager. So you could do things like allocate pages when needed. coupled with paging, this would allow you to only use the needed space and allocate it anytime. you could even make two pages appear contiguous in virtual memory although fragmented in real life.

[XCHG]

Alright these information were really useful but as per my kernel's memory references, everything for now is a physical address. I really am not good with virtual memories and I have very little knowledge about implementing them so I guess I should start reading about virtual memories too. But thank you again everyone. It really helped a lot.

[AJ]

Ho, sorry - I missed the obvious points

* If you are planning to allow DMA transfers in the future, you will need to have some low memory free. For this reason, sticking your kernel at the 0-8mb mark may not be such a good idea.

* If you want to support v86 tasks with BIOS interrupts, you will need to keep the IVT at 0x0000-0x0500 untouched.

Cheers,
Adam

[XCHG]

AJ,

I certainly do want to enable those features in my OS in the near future. The problem is that I start from point X and make to point Y and then realize that there should be something done in the middle of the way that I have not done. Case in point, I coded some basic memory allocation procedures that allowed Page Directory Entries or Page Table Entries to be allocated from a certain point in the memory. Then I realized how am I going to be able to tell which process has allocated what region of memory and for how many bytes. Then I thought of storing a Process ID in the memory block (4MB) that I had left for my memory management components.

Therefore, each page would have 4 bytes associated for it in the memory management memory block and this 4 bytes will be filled with the ID of the process which has allocated that page. Then just after I planned this I realized that I have not enabled multitasking yet lol. See I have good plans and I know that the Wiki says there are no specific orders in which an Operating System should/can be created but at least someone who has been there before could tell others what should/can be done first and what next and etc.

[AJ]

Hi,

I've had the same problem here - coding before designing

I think basic heap management has to come very early in OS dev. If you don't have reliable memory management routines, it can be very difficult debugging other parts of the OS.

I am currently looking at user space memory management myself. The way I see it is like this:

* I already have a separate page directory for each task.
* My kernel is mapped to the 3GB mark of each task (at 3GB+, the page tables are exactly the same for each process space).
* Therefore, when returning process memory to the system, all I need to do is increment through that processes page directory from 0-3GB and free up all the physical RAM there.

Of course, exceptions have to be made for memory shared between processes etc..., but if you keep a list of shared RAM, you should be able to design this in to your system.

I found that initially, I kept doing kernel rewrites due to lack of design foresight - especially in lack of flexibility in memory management. I regard these as 'learning kernels'. The version I am working on now has been going for several months and is up to the multitasking with separate process spaces stage and I think I have now designed in enough flexibility that any revisions will be achievable by 'patching' the existing code.

Cheers,
Adam