OSDev.org

Posted: **Tue Oct 21, 2014 1:18 am**

PatSanf wrote:I'm hoping that what I now have will meet all of the physical memory manager requirements that I need to implement before writing a virtual memory manager for paging. The only thing I think I might be iffy on is the assumption that I'll have at least 4MB of memory available immediately after the kernel. I'm guessing that this is a safe assumption to make. If it isn't, then I can correct it by looping over the memory map to find a guaranteed area large enough for the stack, or putting a line reading ". = . + 4M" in my linker script to eliminate the assumption (I think). What do you guys think of my assumption, possible solutions if it causes a problem, and code?

Don't assume. Read the memory map, exclude the area that your kernel needs and find a free space by doing so.

PatSanf wrote:I am getting curious about memory mapping. If I don't have the addresses that the kernel occupies on the stack will I be able to properly identity map the memory, or is that something I only need to worry about with a higher half kernel?

That does not matter. Read about paging and you'll find out.

Posted: **Tue Oct 21, 2014 9:02 pm**

max wrote:
PatSanf wrote:I'm hoping that what I now have will meet all of the physical memory manager requirements that I need to implement before writing a virtual memory manager for paging. The only thing I think I might be iffy on is the assumption that I'll have at least 4MB of memory available immediately after the kernel. I'm guessing that this is a safe assumption to make. If it isn't, then I can correct it by looping over the memory map to find a guaranteed area large enough for the stack, or putting a line reading ". = . + 4M" in my linker script to eliminate the assumption (I think). What do you guys think of my assumption, possible solutions if it causes a problem, and code?
Don't assume. Read the memory map, exclude the area that your kernel needs and find a free space by doing so.

Here's what I've done.
I now have the stack being placed in the first free region of high memory, not occupied by the kernel, that's large enough.
What I do is:
* Figure out how much memory the system has, in MB. Use that to figure out how many 4KB chunks of memory I'll need to store addresses for (theoretical maximum).
* Go through each region of the memory map.
* If it's a free region in high memory
* Figure out where it starts and ends
* If the base address goes to a memory address that's occupied by the kernel, then advance it until isn't (on a copy so I don't overwrite the memory map).
* Advance the address to the next 4K aligned address.
* Make sure there's enough space for the stack in the region from that point, and if there is
* Put the stack in there.

To populate the stack I then:
* Go through each region of the memory map.
* If it's free, and if it's in high memory then
* Figure out where the first 4K aligned address in the free region is
* Make sure it isn't an address occupied by the stack, or the kernel
* Push it to the stack.
* Advance the counter by 4096, push it to the stack, and repeat until I've pushed all the addresses onto the stack.

You can see these changes here: https://github.com/PSanf2/POS_C9/blob/m ... _manager.c

What's your opinion of that? Think it will serve my purposes?

max wrote:
PatSanf wrote:I am getting curious about memory mapping. If I don't have the addresses that the kernel occupies on the stack will I be able to properly identity map the memory, or is that something I only need to worry about with a higher half kernel?
That does not matter. Read about paging and you'll find out.

Believe me, I have been.

What I'm mainly worried about is making sure the instruction pointer doesn't get screwed up. I read somewhere that you need to make sure the virtual memory address maps to the physical memory address for the kernel, or something relating to the instruction pointer will get messed up as soon as you enable paging and bugger the system.

Posted: **Wed Oct 22, 2014 6:49 am**

Some minor things:
1. What if (j & 0xfff && 0x2000-j % 0x1000>*length) ?
2. I'd define a macro BLOCK_SIZE, UNIT_SIZE or PAGE_SIZE for 0x1000. You may want to use another value on another architecture.
3. You don't need the brackets if there is only one instruction after a "if", "for" or "while" statement.
4. You can shorten

Code: Select all

if (my_base_addr >= kernel_start && my_base_addr < kernel_end)
			{
				while ((my_base_addr >= kernel_start && my_base_addr < kernel_end))
				{
					my_base_addr = my_base_addr + 1;
				}
			}
			
			// make sure the address is page aligned.
			// don't know if i need this, but it may come in handy.
			if (my_base_addr & 0xFFF)
			{
				my_base_addr &= ~(0xFFF);
				my_base_addr = my_base_addr + 0x1000;
			}

to

Code: Select all

			while ((my_base_addr+stack_size * sizeof(uintptr_t) >= kernel_start && my_base_addr < kernel_end))
				my_base_addr = my_base_addr& ~(BLOCK_SIZE-1) +BLOCK_SIZE;

note that "+stack_size*sizeof(uintptr_t)". Even if my_my_base_addr is below kernel_start, stack_high could be above. The second implementation should run much faster.
5. Many pieces like

Code: Select all

// variables used to define a stack.
u32int *stack_low;
u32int *stack_ptr = 0;
u32int *stack_high; // stack high getting set to 0 is being used as a flag value.

or

Code: Select all

			if ((end_addr - my_base_addr) > (stack_size * sizeof(u32int)))

require a 32 bit architecture. But you wrote here you are going to implement x86_64. I would use an uintptr_t type instead.

Code: Select all

if (*base_addr >= 0x100000)
			{
(...)
			}
			else
			{
				vga_buffer_put_str("\nRegion is in low memory. Addresses not pushed to stack. stack_ptr=");
				vga_buffer_put_hex((u32int) stack_ptr);
			}

6. Not everything in low memory must be always allocated. If your grub memory map says there is free memory below 0x100000, then it really will be free. Trust it!

max wrote:
martinFTW wrote:2. I don't like the idea of a stack-only PMM at all. You don't know how much free pages there are(and present-day PCs have many gigabytes), so you are very likely to have a bufferoverflow and to overwrite something in lower memory as max pointed out. This huge stack must be placed somewhere, where it does not overwrite code, data, the usual stack, memory allocated by the mmap or later by the PMM itself. I use a stack in combination with a bitmap. The stack is implemented as a static array with 1000 integers. So i don't need to care about hardcoded addresses or pointer arithmetics . if this stack is empty my kernel will refill it with 1000 addresses from bitmap. Certainly a disadvantage of this approach is that refilling the stack can mske up a very long latency. So I should not refill the PMM stack in interupt mode, but inside a kernel thread. It would be also a good idea to have something like a watermark for the bitmap.
You do know what and how many free pages are available, because either memory probing or the multiboot memory map that your bootloader gave you can tell you. See it realistically, in normal home computers you have (normally) at most 16GB memory at the moment, this amount of memory can put on a page stack with a size of 1024*1024*16 bytes, that's only 16MiB. If you first find out how much memory is actually available, and then size your stack accordingly, then find a place to store this stack, and then put the free page addresses on it it will be fine

In x86_64 it would be 1024*1024*16*1024/4096*8 which is 33 MB.
My approach still has some advantages. Some old DMA can require you to allocate continuesly physical memory at runtime(I can handle that by cleaning the stack) and my approach still requires less memory. But I know these are only minor advantages and I'm going to write a MICROkernel, which does not even have kernel threads so far. So you're right, that it might not be worth it.

Cheers,
Martin

Posted: **Wed Oct 22, 2014 7:32 am**

martinFTW wrote:3. You don't need the brackets if there is only one instruction after a "if", "for" or "while" statement.
4. You can shorten [...]

Imho you *can* shorten it by not using brackets, but you shouldn't because that will make the code far less readable. But thats completely opinion based

Posted: **Wed Oct 22, 2014 10:06 am**

max wrote:Imho you *can* shorten it by not using brackets, but you shouldn't because that will make the code far less readable.

I completely agree. The braces also help to eliminate silly bugs since you know where your blocks start and end.

Posted: **Wed Oct 22, 2014 10:37 am**

eryjus wrote:
max wrote:Imho you *can* shorten it by not using brackets, but you shouldn't because that will make the code far less readable.
I completely agree. The braces also help to eliminate silly bugs since you know where your blocks start and end.

Well some people write that one instruction without brackets in the same line. This may be confusing for some people and even lead to bugs. But if you indent this single instruction in a newline as usually and like I did here, it should be absolutely clear IMHO:

Code: Select all

         while ((my_base_addr+stack_size * sizeof(uintptr_t) >= kernel_start && my_base_addr < kernel_end))
                 my_base_addr = my_base_addr& ~(BLOCK_SIZE-1) +BLOCK_SIZE;

Posted: **Wed Oct 22, 2014 1:08 pm**

martinFTW wrote:6. Not everything in low memory must be always allocated. If your grub memory map says there is free memory below 0x100000, then it really will be free. Trust it!

Go ahead and overwrite the IVT and BDA on your first alloc. You won't need them later to do VBE interrupts, nor would you need them to list what serial and parallel ports are available to you, and neither will you ever need any space in low memory to do DMA transfers to floppy disks and soundblaster cards. Sure, go trust the memory map. It must be right!

Posted: **Wed Oct 22, 2014 1:57 pm**

Combuster wrote:
martinFTW wrote:6. Not everything in low memory must be always allocated. If your grub memory map says there is free memory below 0x100000, then it really will be free. Trust it!
Go ahead and overwrite the IVT and BDA on your first alloc. You won't need them later to do VBE interrupts, nor would you need them to list what serial and parallel ports are available to you, and neither will you ever need any space in low memory to do DMA transfers to floppy disks and soundblaster cards. Sure, go trust the memory map. It must be right!

Well I am trusting the memory map and it doesn't crash neither in Qemu nor on my real hardware. I have heard of cases where the grub memory map was wrong, but that didn't occur to me so far.

Ok I admit, What I am in fact doing is allocating the space from 0x0 to 0x1000, because there is the Bios data area and NULL pointers should stay null pointers. But there still is no need to mark the whole lower memory as used and to generally distrust the memory map there especially if you are not planning to support ancient floppy disks or soundblaster cards. Finally I'd like to mention, that even if PatSanf did not mark 0x0 to 0x1000 as allocated, he would not allocate it on the first try but in fact at last, because of his stack implementation.

Posted: **Thu Oct 23, 2014 5:54 am**

One minor practical snag, some devices do not accept memory page 0. Example is Intel E1000 network card. Best to just reserve it and use it for IVT, rather than risk more devices like this existing.

Posted: **Fri Oct 31, 2014 3:42 pm**

I didn't think this was a comment worthy of starting a new thread about, so I'm bumping this one because it's really a continuation of the discussion.

So I have paging enabled. Yay! You can see an updated version of my code on GitHub: https://github.com/PSanf2/POS_C9

OK... now what?

I'm now needing to know what I'm supposed to do for each type of error a page fault can have. I've set my kernel up to give me some options that'll let me trigger a read fault, or a write fault at will. You can see the code that I'm using here: https://github.com/PSanf2/POS_C9/blob/m ... nel.c#L110 and screenshots of the results here http://i.imgur.com/4BZACGH.png and here http://i.imgur.com/Szbtd8v.png

From the wiki, http://wiki.osdev.org/Paging#Handling , I'm gathering that I should evaluate the flags in the error code, and take one of eight possible actions. What I don't understand is what those actions should be. If I run a program and it tries to read or write from an invalid page should I be looking to run a page replacement algorithm to give the program more memory at that virtual address? Will all of my user-space programs be required to alloc some memory before they can use variables? What order should I handle the flag bits in? How would I go about specifying that a process is a supervisory process, or a user process? Is that just set with the user/supervisor bit for the page table entry on the page directory? Also, what about context switching? Is that just changing the address of the page directory on the control register?

I still need to implement a virtual memory manager, with functions like alloc and free, but I'd like to get some of the page fault handler laid out before I do. Is the virtual memory manager simply a process of keeping track of which virtual address are available, and splitting up areas of available virtual memory into smaller portions, like I was doing with my original implementation? That can be seen here: https://github.com/PSanf2/POS_C9/blob/f ... _manager.c Basically, I'm wondering if I should be able to recycle some of that old code into a virtual memory manager that works in a page directory to manage virtual addresses on a stack.

I know that to start page swapping I'll need to be able to access the hard drive first, which means I'm asking to get into another world of issues. I'm thinking I should reconfigure how I'm using grub-mkrescue and qemu. I'd like to get my environment setup to have my OS think it's living on a hard disk. This means I'll need to create a virtual HDD image that qemu can work with, write GRUB into the boot sector with my OS right after it, and get qemu to boot off the HDD image. Does anyone know of some good tools I can use to do this? Would I be better off just creating some type of an installer that'll write the OS data to the disk? Am I getting ahead of myself with all of this, and would it be best to stick with a bootable CD image for now?

Posted: **Fri Oct 31, 2014 8:57 pm**

PatSanf wrote:This means I'll need to create a virtual HDD image that qemu can work with, write GRUB into the boot sector with my OS right after it, and get qemu to boot off the HDD image. Does anyone know of some good tools I can use to do this?

I remember googling around for quite a while and found a script here. I leveraged that script customizing it to my needs and added some additional lines to create other disk formats from my raw image (using VirtualBox utilities).

Code: Select all

rm -f bin/harddisk.vdi
vboxmanage convertfromraw bin/harddisk.img bin/harddisk.vdi --format VDI --uuid=3bb5a804-7014-495b-9dbb-d21ea7482c04 2> /dev/null
 
rm -f "bin/OS Test.vmdk"
vboxmanage convertfromraw bin/harddisk.img "bin/OS Test.vmdk" --format VMDK  2> /dev/null

OSDev.org

How can I improve my OS code? What direction should I take t

Re: How can I improve my OS code? What direction should I ta

Re: How can I improve my OS code? What direction should I ta

Re: How can I improve my OS code? What direction should I ta

Re: How can I improve my OS code? What direction should I ta

Re: How can I improve my OS code? What direction should I ta

Re: How can I improve my OS code? What direction should I ta

Re: How can I improve my OS code? What direction should I ta

Re: How can I improve my OS code? What direction should I ta

Re: How can I improve my OS code? What direction should I ta

Re: How can I improve my OS code? What direction should I ta

Re: How can I improve my OS code? What direction should I ta