Initrd and physical memory manager.

Question about which tools to use, bugs, the best way to implement a function, etc should go here. Don't forget to see if your question is answered in the wiki first! When in doubt post here.
Sourcer
Member
Member
Posts: 58
Joined: Fri Jun 17, 2016 11:29 pm
Libera.chat IRC: WalterPinkman

Initrd and physical memory manager.

Post by Sourcer »

Hey,

My kernel uses an initrd that is loaded with GRUB2's "module" instruction.

1) Is there a way to control the loading address of the module?
2) Can you give me examples of how your physical memory managers take the initrd addresses in consideration?

My initrd is loaded immidietley after the kernel, my first approach was starting the management(bitmap and some management structs) right after the initrd,
and when i don't need the initrd i just shift the management structs back to the initrd's place. This seems like a bad approach, but couldn't think of another.
User avatar
Ch4ozz
Member
Member
Posts: 170
Joined: Mon Jul 18, 2016 2:46 pm
Libera.chat IRC: esi

Re: Initrd and physical memory manager.

Post by Ch4ozz »

Code: Select all

int i;
for(i = 0; i < mb_info->mbs_mods_count; i++)
{
	uintptr_t addr = mb_info->mbs_mods_addr[i].mod_start;
	
	while (addr < mb_info->mbs_mods_addr[i].mod_end)
	{
		malloc_mark_used_adr((void*)addr);
		addr += 0x1000;
	}
}
Im prretty sure grub will load the initrd to the specified module base in your linker script.
Sourcer
Member
Member
Posts: 58
Joined: Fri Jun 17, 2016 11:29 pm
Libera.chat IRC: WalterPinkman

Re: Initrd and physical memory manager.

Post by Sourcer »

Ch4ozz wrote:

Code: Select all

int i;
for(i = 0; i < mb_info->mbs_mods_count; i++)
{
	uintptr_t addr = mb_info->mbs_mods_addr[i].mod_start;
	
	while (addr < mb_info->mbs_mods_addr[i].mod_end)
	{
		malloc_mark_used_adr((void*)addr);
		addr += 0x1000;
	}
}
Im prretty sure grub will load the initrd to the specified module base in your linker script.
This code is pretty abstract and depends on one's design.
The problem is that my bitmap starts right after the kernel, and has a base address. The initrd makes my bitmap shift its location to a higher address.
Also, i had no idea that i can use my linker script to load the module to a specific address. Can you give me more info? Since the module has no relation with my final kernel executable(ELF)
User avatar
Ch4ozz
Member
Member
Posts: 170
Joined: Mon Jul 18, 2016 2:46 pm
Libera.chat IRC: esi

Re: Initrd and physical memory manager.

Post by Ch4ozz »

Let the bitmap start recording at address 0 and simply set its size to 0x20000 bytes (4GB depict) if your OS is x86
Sourcer
Member
Member
Posts: 58
Joined: Fri Jun 17, 2016 11:29 pm
Libera.chat IRC: WalterPinkman

Re: Initrd and physical memory manager.

Post by Sourcer »

Ch4ozz wrote:Let the bitmap start recording at address 0 and simply set its size to 0x20000 bytes (4GB depict) if your OS is x86
Isn't that a waste of space?
My design uses N bitmaps for N available memory regions(according to the info from the multiboot/BIOS). The RAM isn't always continuous, so 0x20000 bytes is a killer, no?

Also, when porting that design for 64bit, i'll have to use a bitmap sized 0x2000000000.
Last edited by Sourcer on Tue Sep 13, 2016 8:09 am, edited 1 time in total.
User avatar
Ch4ozz
Member
Member
Posts: 170
Joined: Mon Jul 18, 2016 2:46 pm
Libera.chat IRC: esi

Re: Initrd and physical memory manager.

Post by Ch4ozz »

Sourcer wrote:
Ch4ozz wrote:Let the bitmap start recording at address 0 and simply set its size to 0x20000 bytes (4GB depict) if your OS is x86
Isn't that a waste of space?
My design uses N bitmaps for N available memory regions(according to the info for the multiboot/BIOS). The address space isn't always continuous, so 0x20000 bytes is a killer, no?
Thats 128kb, if your OS wont run on a microcontroller you really dont have to think in kilobytes.
Sometimes simplicity is better then saving some RAM.
Sourcer
Member
Member
Posts: 58
Joined: Fri Jun 17, 2016 11:29 pm
Libera.chat IRC: WalterPinkman

Re: Initrd and physical memory manager.

Post by Sourcer »

Ch4ozz wrote:
Sourcer wrote:
Ch4ozz wrote:Let the bitmap start recording at address 0 and simply set its size to 0x20000 bytes (4GB depict) if your OS is x86
Isn't that a waste of space?
My design uses N bitmaps for N available memory regions(according to the info for the multiboot/BIOS). The address space isn't always continuous, so 0x20000 bytes is a killer, no?
Thats 128kb, if your OS wont run on a microcontroller you really dont have to think in kilobytes.
Sometimes simplicity is better then saving some RAM.
Can you answer the linker question too? thank you.
User avatar
Ch4ozz
Member
Member
Posts: 170
Joined: Mon Jul 18, 2016 2:46 pm
Libera.chat IRC: esi

Re: Initrd and physical memory manager.

Post by Ch4ozz »

Heres my linker script for an app:

Code: Select all

ENTRY(_main)

OUTPUT_FORMAT(elf32-i386)

SECTIONS
{
    . = 0xC0000000;

    .text : {
        *(.text)
    }
    .data ALIGN(4096) : {
        *(.data)
    }
    .rodata ALIGN(4096) : {
        *(.rodata)
    }
    .bss ALIGN(4096) : {
        *(.bss)
    }
}
Its base address will be 0xC0000000 now

And for 64gb memory support it would be 0x200000 bytes which is equal to 2mb which a 64 bit PC should have lol.
Sourcer
Member
Member
Posts: 58
Joined: Fri Jun 17, 2016 11:29 pm
Libera.chat IRC: WalterPinkman

Re: Initrd and physical memory manager.

Post by Sourcer »

Ch4ozz wrote:Heres my linker script for an app:

Code: Select all

ENTRY(_main)

OUTPUT_FORMAT(elf32-i386)

SECTIONS
{
    . = 0xC0000000;

    .text : {
        *(.text)
    }
    .data ALIGN(4096) : {
        *(.data)
    }
    .rodata ALIGN(4096) : {
        *(.rodata)
    }
    .bss ALIGN(4096) : {
        *(.bss)
    }
}
Its base address will be 0xC0000000 now

And for 64gb memory support it would be 0x200000 bytes which is equal to 2mb which a 64 bit PC should have lol.
I think your calculation is wrong, and still can't figure out what is the connection between the linker script and GRUB's module loading address. Is it guaranteed that the modules will appear right after the bss section in your linker script?
User avatar
max
Member
Member
Posts: 616
Joined: Mon Mar 05, 2012 11:23 am
Libera.chat IRC: maxdev
Location: Germany
Contact:

Re: Initrd and physical memory manager.

Post by max »

Sourcer wrote:Hey,

My kernel uses an initrd that is loaded with GRUB2's "module" instruction.

1) Is there a way to control the loading address of the module?
2) Can you give me examples of how your physical memory managers take the initrd addresses in consideration?

My initrd is loaded immidietley after the kernel, my first approach was starting the management(bitmap and some management structs) right after the initrd,
and when i don't need the initrd i just shift the management structs back to the initrd's place. This seems like a bad approach, but couldn't think of another.
Hey,

usually you shouldn't care where the module is loaded. Just read the GRUB memory map and check for each page if it's within one of the modules ranges. If it's marked as free & not within a module range, tell your physical page manager that it's free. Nearly all designs have in common that there's a physical page manager that keeps track of free pages.

That implementation could look similar to:
https://github.com/maxdev1/ghost/blob/m ... preter.cpp

Greets
User avatar
Ch4ozz
Member
Member
Posts: 170
Joined: Mon Jul 18, 2016 2:46 pm
Libera.chat IRC: esi

Re: Initrd and physical memory manager.

Post by Ch4ozz »

64gb = 68719476736 bytes =( div 4096 ) 16777216 pages =( div 8 ) 2097152 bytes = 0x200000

As I said, Im not 100% sure if GRUB can detect your elf file and will load it correctly (like it does with the kernel image file)
You can test that but I currently have no time to do so.
Sourcer
Member
Member
Posts: 58
Joined: Fri Jun 17, 2016 11:29 pm
Libera.chat IRC: WalterPinkman

Re: Initrd and physical memory manager.

Post by Sourcer »

Ch4ozz wrote:64gb = 68719476736 bytes =( div 4096 ) 16777216 pages =( div 8 ) 2097152 bytes = 0x200000

As I said, Im not 100% sure if GRUB can detect your elf file and will load it correctly (like it does with the kernel image file)
You can test that but I currently have no time to do so.
Theoratically x64-86 should be able to address 2^48 bytes. I did test it, but i don't want to make any assumptions that can result in undefined behavior.
User avatar
Ch4ozz
Member
Member
Posts: 170
Joined: Mon Jul 18, 2016 2:46 pm
Libera.chat IRC: esi

Re: Initrd and physical memory manager.

Post by Ch4ozz »

Sourcer wrote:
Ch4ozz wrote:64gb = 68719476736 bytes =( div 4096 ) 16777216 pages =( div 8 ) 2097152 bytes = 0x200000

As I said, Im not 100% sure if GRUB can detect your elf file and will load it correctly (like it does with the kernel image file)
You can test that but I currently have no time to do so.
Theoratically x64-86 should be able to address 2^48 bytes. I did test it, but i don't want to make any assumptions that can result in undefined behavior.
On x64 you use 4mb pages normally which will result in:
256tb = 281474976710656 bytes = 67108864 pages = 8388608 bytes = 0x800000 = 8mb
Boris
Member
Member
Posts: 145
Joined: Sat Nov 07, 2015 3:12 pm

Re: Initrd and physical memory manager.

Post by Boris »

You should be cautious of everything GRUB says. Any multiboot memory map entry may contain
- your kernel or its command line
- A module or its command line
- A multi boot structure. Il
- An area marked as free or unusable in another multiboot mem map entry.
- Walnuts

You have to double check before consuming a multi boot entry !
User avatar
Brendan
Member
Member
Posts: 8561
Joined: Sat Jan 15, 2005 12:00 am
Location: At his keyboard!
Contact:

Re: Initrd and physical memory manager.

Post by Brendan »

Hi,
Sourcer wrote:My kernel uses an initrd that is loaded with GRUB2's "module" instruction.

1) Is there a way to control the loading address of the module?
I don't know about GRUB; but for the multi-boot specification there's only an "align things on page boundaries" option with no other control.
Sourcer wrote:2) Can you give me examples of how your physical memory managers take the initrd addresses in consideration?
My physical memory manager doesn't take the initrd addresses in consideration.

I initialise a (temporary/initial) physical memory manager very early during boot (before loading "initrd" into memory); and (later) when I'm loading "initrd" I read the first sector/block (which contains the total file's size) and I allocate (virtual) pages for it, then load the file into the allocated (virtual) pages (and after it's loaded I do an "if(compressed) { relocate pages to higher virtual address, then decompress back to where it was}" thing).

After that (and after boot code has figured out which kernel to use for the computer and found the right kernel in "initrd") the kernel's initialisation code "cannibalises" the temporary/initial physical memory manager's data to setup the actual/final physical memory manager, which can be done with a relatively trivial "while( boot code still thinks there's memory free) { allocate physical page from boot code; free physical page using kernel's physical memory manager; }" loop.


Cheers,

Brendan
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.
Post Reply