Passing data block from boot loader to the kernel

[Schol-R-LEA]

Just as a heads' up for those who have been helping me so far, I am now having an issue where the second stage loader is supposed to pass some data to the kernel, but the kernel isn't finding the data block at the memory address I expected it to be at.

The plan had been to place the data block on the stack, but I quickly realized that a) this conflicted with how I was intending page map the stack, b) it would leave the kernel with an extremely small usable stack, and c) passing the large amount of data in question would present a problem when defining the function signature for the kernel's main function. I may revisit that idea later, but for now, I am just passing the data as a block at the end of the first 64KiB of the kernel memory. A more complete solution would almost certainly involve a more elaborate setup for the page mapping.

Right now, I am - despite some misgivings on this - hard-coding the address I expect the data block to be at. I do define an enum representing the offsets into this block, but I am not certain if it is a correct reflection of the data block in memory.

The code in the kernel right now is:

Code: Select all

/* kernel.c */
#include <stdint.h>
#include "terminal.h"
#include "mem_map.h"

#define KDATA_OFFSET 0xc000fffc

enum KData
{
    mmap_cnt = 0,
    mmap = mmap_cnt + 1 + (24 * 16),
    drive_id,
    fat = drive_id + 1 + (9 * (512 / 4))
};


void kernel_main()
{
    clear_screen();
    kprints("Starting Kernel...\n\n", CYAN, BLACK, 0);

    uint32_t* fs_data = (uint32_t *) KDATA_OFFSET;

    kprints("Passed Data offset ", WHITE, BLACK, 0);
    kprintu((uint32_t)(fs_data + 1), 16, WHITE, BLACK, 0);

    kprints("\nDrive ID: ", WHITE, BLACK, 0);
    kprintu(*(fs_data - drive_id), 16, WHITE, BLACK, 0);
    kprints("\nFAT offset ", WHITE, BLACK, 0);
    kprintu((uint32_t)(fs_data - drive_id - 1), 16, WHITE, BLACK, 0);

    kprints("\nMemory Map Table offset ", WHITE, BLACK, 0);
    kprintu((uint32_t)(fs_data - mmap_cnt - 1), 16, WHITE, BLACK, 0);
    print_mmap(*(fs_data - mmap_cnt), (struct memory_map_entry*)(fs_data - mmap));
}

The code which moves the data block to FFFF:FFFC (which should be mapped to 0xC000FFFC in virtual memory) is here:

Code: Select all

kernel_base        equ 0xffff

kdata_offset       equ 0xfffc

struc KData
    .mmap_cnt      resd 1
    .mmap          resd High_Mem_Map_size * 16
    .drive         resd 1
    .fat           resd fat_size
endstruc

...

Code: Select all

;; Attempt to get the full physical memory map for the system
;; this must be done before the move to protected mode
get_mem_maps:
        ; get the low memory map
        write low_mem
        int LMBIOS
        mov si, print_buffer
        call print_decimal_word
        write kbytes
        ; get the high memory map
        push es
        push si
        push di
        push bp
        mov ax, kernel_base
        mov es, ax
        mov di, (kdata_offset - KData.mmap - mem_map_buffer_size)
        mov si, (kdata_offset - KData.mmap_cnt)
        call get_hi_memory_map
        mov di, (kdata_offset - KData.mmap - mem_map_buffer_size)
        mov bp, es:[kdata_offset - KData.mmap_cnt]
        call print_hi_mem_map
        pop bp
        pop di
        pop si
        pop es

load_kernel_data:
        push ax
        push es
        mov ax, kernel_base
        mov es, ax
        mov dx, word [bp - stg2_parameters.drive]
        mov es:[kdata_offset - KData.drive], edx
        lea ax, es:[kdata_offset - KData.fat - fat_size]
        memcopy_rm ax, [bp - stg2_parameters.fat_0], fat_size
        pop es
        pop ax

The goal of the transfer code is to write first the number of entries in the memory map, followed by the memory map itself, followed by the disk ID, and then finally followed by the entire FAT for the floppy disk.

Right now, it isn't finding any data at that location at all - it's all zeroed memory.

[Octocontrabass]

I do define an enum representing the offsets into this block, but I am not certain if it is a correct reflection of the data block in memory.

Maybe it is, maybe it isn't. But what I want to know is: why aren't you using a struct? It would make the pointer arithmetic easy:

Code: Select all

struct kdata * data_block = ((struct kdata *)KDATA_OFFSET) - 1;

FFFF:FFFC (which should be mapped to 0xC000FFFC in virtual memory)

It can't be. Real mode address FFFF:FFFC is linear address 0x10FFEC.

Code: Select all

        lea ax, es:[kdata_offset - KData.fat - fat_size]

What exactly is the purpose of the ES segment override on this instruction?

[Schol-R-LEA]

I do define an enum representing the offsets into this block, but I am not certain if it is a correct reflection of the data block in memory.

Maybe it is, maybe it isn't. But what I want to know is: why aren't you using a struct? It would make the pointer arithmetic easy:

Code: Select all

struct kdata * data_block = ((struct kdata *)KDATA_OFFSET) - 1;

For some reason, I was thinking that it wouldn't be feasible but yes, that should be possible. Let me see about reworking it that way. EDIT: this should do:

Code: Select all

struct kdata
{
    uint32_t mmap_cnt;
    struct memory_map_entry mem_table[16];
    uint32_t drive_id;
    uint8_t fat[9 * 512];
};

FFFF:FFFC (which should be mapped to 0xC000FFFC in virtual memory)

It can't be. Real mode address FFFF:FFFC is linear address 0x10FFEC.

I have the virtual addresses from 0xC0000000-0xC003FFFF mapped to physical addresses 0x00010000-0x0004FFFF, or vice versa. Or at least, I thought I did.

Code: Select all

        lea ax, es:[kdata_offset - KData.fat - fat_size]

What exactly is the purpose of the ES segment override on this instruction?

That's... a good question. I think I was thrashing a bit when I added the segment marker there.

I may need to take a break again.

[Schol-R-LEA]

Updated code (pushed to repo):

Code: Select all

/* kernel.c */
#include <stdint.h>
#include "terminal.h"
#include "mem_map.h"

#define KDATA_OFFSET 0xc0010000

struct kdata
{
    uint8_t fat[9 * 512];
    uint32_t drive_id;
    struct memory_map_entry mem_table[16];
    uint32_t mmap_cnt;
};

void kernel_main()
{
    clear_screen();
    kprints("Starting Kernel...\n\n", CYAN, BLACK, 0);

    struct kdata* fs_data = (struct kdata *) (KDATA_OFFSET - sizeof(struct kdata));

    kprints("Passed Data offset ", WHITE, BLACK, 0);
    kprintu((uint32_t)(fs_data), 16, WHITE, BLACK, 0);

    kprints("\nDrive ID: ", WHITE, BLACK, 0);
    kprintu(fs_data->drive_id, 16, WHITE, BLACK, 0);
    kprints("\nFAT offset ", WHITE, BLACK, 0);
    kprintu((uint32_t) &fs_data->fat, 16, WHITE, BLACK, 0);

    kprints("\nMemory Map Table offset ", WHITE, BLACK, 0);
    kprintu((uint32_t) &fs_data->mem_table, 16, WHITE, BLACK, 0);
    print_mmap(fs_data->mmap_cnt, fs_data->mem_table);
}

Code: Select all

kernel_base        equ 0xffff

struc KData
    .fat           resb fat_size
    .drive         resd 1
    .mmap          resb High_Mem_Map_size * 16
    .mmap_cnt      resd 1
endstruc

kdata_offset       equ 0xffff - KData_size

Code: Select all

;; Attempt to get the full physical memory map for the system
;; this must be done before the move to protected mode
get_mem_maps:
        ; get the low memory map
        write low_mem
        int LMBIOS
        mov si, print_buffer
        call print_decimal_word
        write kbytes
        ; get the high memory map
        push es
        push si
        push di
        push bp
        mov ax, kernel_base
        mov es, ax
        mov di, (kdata_offset + KData.mmap)
        mov si, (kdata_offset + KData.mmap_cnt)
        call get_hi_memory_map
        mov di, (kdata_offset + KData.mmap)
        mov bp, es:[kdata_offset + KData.mmap_cnt]
        call print_hi_mem_map
        pop bp
        pop di
        pop si
        pop es

load_kernel_data:
        push ax
        push es
        mov ax, kernel_base
        mov es, ax
        zero(edx)
        mov dx, word [bp - stg2_parameters.drive]
        mov es:[kdata_offset + KData.drive], edx
        mov ax, kdata_offset + KData.fat
        memcopy_rm ax, [bp - stg2_parameters.fat_0], fat_size
        pop es
        pop ax

[Schol-R-LEA]

OK, so I figured it out way too early this morning (I was unable to sleep so I ended up obsessing over this).

While thrashing about, I struck on the idea of using the print_mmap() routine to probe the memory a bit. By forcing the count to 2, I got the following results:


This showed that the data was in fact getting loaded, but not where I was anticipating it to be - the data shown was correct, but in the wrong location.

After a bit of experimentation and some more thrashing, I found that the data was offset from the area I expected it to be in by 16 bytes. Adding a negative offset to the code

Code: Select all

    struct kdata* fs_data = (struct kdata *) (KDATA_OFFSET - sizeof(struct kdata) - 16);

get the data correctly:



It took me a bit of thinking to realize that this was the 16 bits which are 'missing' in the 8086 High Memory Area.

I also noted that I am not loading non-constant initialized data correctly when I load the ELF file, but that is a problem I can address later - I am pretty sure that the wiki explains why this can happen, but I will put off reviewing that until I have the time.

So basically, as far as what is being displayed, this puts me where I was a few weeks ago with the second stage boot loader, but now I am in the kernel and can actually do something with that memory map.

I also noted that the data block in question is exactly 5000 bytes in size (0x1388 in hex) but that's not especially relevant right now.

[Octocontrabass]

I have the virtual addresses from 0xC0000000-0xC003FFFF mapped to physical addresses 0x00010000-0x0004FFFF, or vice versa. Or at least, I thought I did.

You do. Paging doesn't change the low 12 bits of the address, so it's impossible to map 0x10FFEC to 0xC000FFFC.

[Schol-R-LEA]

Ah, I misunderstood what Octocontrabass was saying. Yes, you are right.

Hence the 16 byte difference, then.

[devc1]

Since you are a beginner, I will give you how I do it.
My kernel is in the PE64 Format, so I create a section named "INITDATA" and allocate the initdata structure to it. Then the bootloader detects it and fills it. I got this idea when I was inspecting the Windows NT Kernel Executable Image "ntoskrnl.exe" which in Windows 11 is still at Version 10 (Windows 10 As I understand)

When you are planning to boot on UEFI and Legacy BIOS, these OFFSET Declaration would not work on all computers, because the Memory Map changes. So my kernel fixes that by being relocated by the bootloader to Conventional Memory, and all its structures are relocated. So never expect that Address 0x100000 is RAM on all PCs.

Then when it boots, my kernel relocates it self to the top of virtual memory (System Space) and continues the work.

Devc1,

[nullplan]

Since you are a beginner, I will give you how I do it.
My kernel is in the PE64 Format, so I create a section named "INITDATA" and allocate the initdata structure to it. Then the bootloader detects it and fills it.

Well, that is certainly one way, but requires absolute synchronicity between bootloader and kernel. Passing data on the boot stack affords more flexibility. And time has taught that flexibility will be needed. You will forget something, or need something different in the future.

When you are planning to boot on UEFI and Legacy BIOS, these OFFSET Declaration would not work on all computers, because the Memory Map changes. So my kernel fixes that by being relocated by the bootloader to Conventional Memory, and all its structures are relocated. So never expect that Address 0x100000 is RAM on all PCs.

The kernel itself can abstract its position in physical memory by way of virtual memory. And the bootloader can be made responsible for setting up the page tables. As I have often stated before, my kernel does not care about its position in physical memory, and could in theory be in pieces strewn all across physical memory. Now, all bootloader adapters I have written load the kernel en bloc, but the kernel itself does not depend on that fact.

However, for a legacy BIOS bootloader, it is normal to expect the 1MB area to hold at least some memory. Linux will in fact only run from that address (on x86; this is an architectural detail). The kernel itself obviously should not make that same assumption, at least not in architecture-independent code.

It is a good idea to relax assumptions and check them if possible, but sometimes the architecture defines things for you. There being RAM at 1MB has been a defining feature of PCs ever since the PC-AT. No manufacturer can build a PC that works any differently without running headlong into problems. And there would be no point. There is no reason not to have RAM at 1MB. There would be more of a reason not to have an I/O memory hole at 640kB (e.g. if you are building a legacy-free system. System boot can happen from flash mapped to 0xffff0000, and BIOS can copy itself to 0x000f0000 one DRAM is set up. Indeed, x86 memory controllers have lock bits to prevent write access to the BIOS copy area for precisely such a feature)