Initrd and physical memory manager.

[Sourcer]

Hey,

My kernel uses an initrd that is loaded with GRUB2's "module" instruction.

1) Is there a way to control the loading address of the module?
2) Can you give me examples of how your physical memory managers take the initrd addresses in consideration?

My initrd is loaded immidietley after the kernel, my first approach was starting the management(bitmap and some management structs) right after the initrd,
and when i don't need the initrd i just shift the management structs back to the initrd's place. This seems like a bad approach, but couldn't think of another.

[Ch4ozz]

Code: Select all

int i;
for(i = 0; i < mb_info->mbs_mods_count; i++)
{
	uintptr_t addr = mb_info->mbs_mods_addr[i].mod_start;
	
	while (addr < mb_info->mbs_mods_addr[i].mod_end)
	{
		malloc_mark_used_adr((void*)addr);
		addr += 0x1000;
	}
}

Im prretty sure grub will load the initrd to the specified module base in your linker script.

[Sourcer]

Code: Select all

int i;
for(i = 0; i < mb_info->mbs_mods_count; i++)
{
	uintptr_t addr = mb_info->mbs_mods_addr[i].mod_start;
	
	while (addr < mb_info->mbs_mods_addr[i].mod_end)
	{
		malloc_mark_used_adr((void*)addr);
		addr += 0x1000;
	}
}

Im prretty sure grub will load the initrd to the specified module base in your linker script.

This code is pretty abstract and depends on one's design.
The problem is that my bitmap starts right after the kernel, and has a base address. The initrd makes my bitmap shift its location to a higher address.
Also, i had no idea that i can use my linker script to load the module to a specific address. Can you give me more info? Since the module has no relation with my final kernel executable(ELF)

[Ch4ozz]

Let the bitmap start recording at address 0 and simply set its size to 0x20000 bytes (4GB depict) if your OS is x86

[Sourcer]

Let the bitmap start recording at address 0 and simply set its size to 0x20000 bytes (4GB depict) if your OS is x86

Isn't that a waste of space?
My design uses N bitmaps for N available memory regions(according to the info from the multiboot/BIOS). The RAM isn't always continuous, so 0x20000 bytes is a killer, no?

Also, when porting that design for 64bit, i'll have to use a bitmap sized 0x2000000000.

[Ch4ozz]

Let the bitmap start recording at address 0 and simply set its size to 0x20000 bytes (4GB depict) if your OS is x86

Isn't that a waste of space?
My design uses N bitmaps for N available memory regions(according to the info for the multiboot/BIOS). The address space isn't always continuous, so 0x20000 bytes is a killer, no?

Thats 128kb, if your OS wont run on a microcontroller you really dont have to think in kilobytes.
Sometimes simplicity is better then saving some RAM.

[Sourcer]

Let the bitmap start recording at address 0 and simply set its size to 0x20000 bytes (4GB depict) if your OS is x86

Isn't that a waste of space?
My design uses N bitmaps for N available memory regions(according to the info for the multiboot/BIOS). The address space isn't always continuous, so 0x20000 bytes is a killer, no?

Thats 128kb, if your OS wont run on a microcontroller you really dont have to think in kilobytes.
Sometimes simplicity is better then saving some RAM.

Can you answer the linker question too? thank you.

[Ch4ozz]

Heres my linker script for an app:

Code: Select all

ENTRY(_main)

OUTPUT_FORMAT(elf32-i386)

SECTIONS
{
    . = 0xC0000000;

    .text : {
        *(.text)
    }
    .data ALIGN(4096) : {
        *(.data)
    }
    .rodata ALIGN(4096) : {
        *(.rodata)
    }
    .bss ALIGN(4096) : {
        *(.bss)
    }
}

Its base address will be 0xC0000000 now

And for 64gb memory support it would be 0x200000 bytes which is equal to 2mb which a 64 bit PC should have lol.

[Sourcer]

Heres my linker script for an app:

Code: Select all

ENTRY(_main)

OUTPUT_FORMAT(elf32-i386)

SECTIONS
{
    . = 0xC0000000;

    .text : {
        *(.text)
    }
    .data ALIGN(4096) : {
        *(.data)
    }
    .rodata ALIGN(4096) : {
        *(.rodata)
    }
    .bss ALIGN(4096) : {
        *(.bss)
    }
}

Its base address will be 0xC0000000 now

And for 64gb memory support it would be 0x200000 bytes which is equal to 2mb which a 64 bit PC should have lol.

I think your calculation is wrong, and still can't figure out what is the connection between the linker script and GRUB's module loading address. Is it guaranteed that the modules will appear right after the bss section in your linker script?

[max]

Hey,

My kernel uses an initrd that is loaded with GRUB2's "module" instruction.

1) Is there a way to control the loading address of the module?
2) Can you give me examples of how your physical memory managers take the initrd addresses in consideration?

My initrd is loaded immidietley after the kernel, my first approach was starting the management(bitmap and some management structs) right after the initrd,
and when i don't need the initrd i just shift the management structs back to the initrd's place. This seems like a bad approach, but couldn't think of another.

Hey,

usually you shouldn't care where the module is loaded. Just read the GRUB memory map and check for each page if it's within one of the modules ranges. If it's marked as free & not within a module range, tell your physical page manager that it's free. Nearly all designs have in common that there's a physical page manager that keeps track of free pages.

That implementation could look similar to:
https://github.com/maxdev1/ghost/blob/m ... preter.cpp

Greets

[Ch4ozz]

64gb = 68719476736 bytes =( div 4096 ) 16777216 pages =( div 8 ) 2097152 bytes = 0x200000

As I said, Im not 100% sure if GRUB can detect your elf file and will load it correctly (like it does with the kernel image file)
You can test that but I currently have no time to do so.

[Sourcer]

64gb = 68719476736 bytes =( div 4096 ) 16777216 pages =( div 8 ) 2097152 bytes = 0x200000

As I said, Im not 100% sure if GRUB can detect your elf file and will load it correctly (like it does with the kernel image file)
You can test that but I currently have no time to do so.

Theoratically x64-86 should be able to address 2^48 bytes. I did test it, but i don't want to make any assumptions that can result in undefined behavior.

[Ch4ozz]

64gb = 68719476736 bytes =( div 4096 ) 16777216 pages =( div 8 ) 2097152 bytes = 0x200000

As I said, Im not 100% sure if GRUB can detect your elf file and will load it correctly (like it does with the kernel image file)
You can test that but I currently have no time to do so.

Theoratically x64-86 should be able to address 2^48 bytes. I did test it, but i don't want to make any assumptions that can result in undefined behavior.

On x64 you use 4mb pages normally which will result in:
256tb = 281474976710656 bytes = 67108864 pages = 8388608 bytes = 0x800000 = 8mb

[Boris]

You should be cautious of everything GRUB says. Any multiboot memory map entry may contain
- your kernel or its command line
- A module or its command line
- A multi boot structure. Il
- An area marked as free or unusable in another multiboot mem map entry.
- Walnuts

You have to double check before consuming a multi boot entry !

[Brendan]

Hi,

My kernel uses an initrd that is loaded with GRUB2's "module" instruction.

1) Is there a way to control the loading address of the module?

I don't know about GRUB; but for the multi-boot specification there's only an "align things on page boundaries" option with no other control.

2) Can you give me examples of how your physical memory managers take the initrd addresses in consideration?

My physical memory manager doesn't take the initrd addresses in consideration.

I initialise a (temporary/initial) physical memory manager very early during boot (before loading "initrd" into memory); and (later) when I'm loading "initrd" I read the first sector/block (which contains the total file's size) and I allocate (virtual) pages for it, then load the file into the allocated (virtual) pages (and after it's loaded I do an "if(compressed) { relocate pages to higher virtual address, then decompress back to where it was}" thing).

After that (and after boot code has figured out which kernel to use for the computer and found the right kernel in "initrd") the kernel's initialisation code "cannibalises" the temporary/initial physical memory manager's data to setup the actual/final physical memory manager, which can be done with a relatively trivial "while( boot code still thinks there's memory free) { allocate physical page from boot code; free physical page using kernel's physical memory manager; }" loop.


Cheers,

Brendan