OSDev.org

Posted: **Wed Dec 16, 2020 1:58 am**

Octocontrabass wrote:
rdos wrote:EFI doesn't work because GetMemoryMap returns 0 for descriptor size,
No OS can boot if GetMemoryMap() returns garbage. Are you able to boot any other OS on this laptop? (Be sure to choose an OS with a 32-bit bootloader if your bootloader is 32-bit.)

Yes. Windows 10 boots. The memory map is a bit of a changing target when allocations & frees are performed in parallel, and so they might construct it so the Windows bootloader works. It's just that I don't know when it is valid and what triggers them to construct it.

Octocontrabass wrote:
rdos wrote:Legacy boot doesn't work since the real-mode interrupt vector for int 10h points to segment E9xx which I assume is not the BIOS
Why do you assume that? All the BIOS ROMs I've looked at map 128kB of runtime code and data in the 0xE0000-0xFFFFF range (aside from ancient ROMs that are smaller than 128kB). I agree that it's a strange address, though; VGA-compatible option ROMs are usually mapped in the 0xC0000-0xC7FFF range.

It's legacy cruft. Already when I wrote this many years ago the Exxx segment was unused and so I mapped it to RAM and placed data (stacks) there. Since the V86 monitor runs in its own process & context it should be easy to fix there. I just need to give calls a stack in another area and unity-map more memory.

Posted: **Wed Dec 16, 2020 3:06 pm**

So, I unity-mapped A0000 to FFFFF and let the stack reside in the F00 segment and then retried. The emulator now steps into real code, but a while into it accesses to gdt are encountered, and soon after that things go very wrong. When trying the real stuff, it once again hangs up. So the VBE interface tries to enable protected mode and so cannot be used in V86 mode.

No progress with EFI either. If I place getting the memory map at the beginning, it still fails, and when it is placed after ExitBootServices then ExitBootServices fails instead. I have no idea why this doesn't work. Perhaps because I load the kernel at a fixed location?

Posted: **Wed Dec 16, 2020 3:58 pm**

do you check the status, returned by GetMemoryMap()? what it is?

Posted: **Wed Dec 16, 2020 4:21 pm**

zaval wrote:do you check the status, returned by GetMemoryMap()? what it is?

It's success, but the descriptor size returned is 0 and the entries are garbage.

Posted: **Wed Dec 16, 2020 4:54 pm**

what I'd do in this situation, is verified my own code around this call, yet once, for example, that the descriptor structure declared in my headers are properly padded for alignment - because as it's declared in the spec, it's misaligned. and if it's all ok, then went to edk mailing list and pointed to that, because it's obviously some severe bug in the FW. they respond there.

Posted: **Wed Dec 16, 2020 6:45 pm**

rdos wrote:The memory map is a bit of a changing target when allocations & frees are performed in parallel, and so they might construct it so the Windows bootloader works.

What do you mean in parallel? Most UEFI functions prohibit any form of parallelism.

rdos wrote:So the VBE interface tries to enable protected mode and so cannot be used in V86 mode.

Perhaps it's time to seriously consider an x86 emulator instead of V86 mode?

rdos wrote:No progress with EFI either. If I place getting the memory map at the beginning, it still fails,

The beginning of what? Do you mean right after the firmware jumps to your bootloader's entry point?

rdos wrote:and when it is placed after ExitBootServices then ExitBootServices fails instead.

No surprise there; the firmware expects you to get the memory map first so you have a valid map key to pass to ExitBootServices.

rdos wrote:I have no idea why this doesn't work. Perhaps because I load the kernel at a fixed location?

UEFI doesn't guarantee available memory at any particular address. Do you check that the location is available and reserve it for your own use before you load the kernel there?

Posted: **Thu Dec 17, 2020 6:17 am**

Octocontrabass wrote:
rdos wrote:The memory map is a bit of a changing target when allocations & frees are performed in parallel, and so they might construct it so the Windows bootloader works.
What do you mean in parallel? Most UEFI functions prohibit any form of parallelism.

I mean that memory allocations, both direct and indirect might affect the memory map. That's why I call it just before ExitBootServices.

Here is the code related to the memory map:

Code: Select all


#pragma pack( push, 1 )

struct LoaderParam
{
    EFI_PHYSICAL_ADDRESS Lfb;
    unsigned int LfbWidth;
    unsigned int LfbHeight;
    unsigned int LfbLineSize;
    unsigned int LfbFlags;
    unsigned int MemEntries;
    EFI_PHYSICAL_ADDRESS AcpiTable;
};

struct MemMapEntry
{
    unsigned int Len;
    unsigned long long Base;
    unsigned long long Size;
    unsigned int Type;
};

#pragma pack( pop )

unsigned int LoaderParamPos = (unsigned int)(RDOS_LOADER + 2);
struct LoaderParam *LoaderData;

char MemMapBuf[4096];
EFI_MEMORY_DESCRIPTOR *MemMap = (EFI_MEMORY_DESCRIPTOR *)MemMapBuf;
unsigned int MemMapSize = 4096;
unsigned int MapKey;
unsigned int MemDescrSize;
unsigned int MemDescrVersion;

unsigned int MemMapCount;
struct MemMapEntry *MemMapArr;

static void ShowMem(unsigned long long Base, unsigned long long Size)
{
    unsigned int lsb, msb;

    lsb = (unsigned int)Base;
    msb = (unsigned int)(Base >> 32);
    printf("%08lX_%08lX-", msb, lsb);

    Base += Size;
    Base--;
    lsb = (unsigned int)Base;
    msb = (unsigned int)(Base >> 32);
    printf("%08lX_%08lX\n\r", msb, lsb);
}

static void DumpMem()
{
    int i;

    printf("ACPI: %08lX\n\r", AcpiTable);
    printf("LFB: %08lX\n\r", LfbBase);

    for (i = 0; i < MemMapCount; i++)
        ShowMem(MemMapArr[i].Base, MemMapArr[i].Size);
}

static void AddMem(unsigned long long Base, unsigned long long Size)
{
    MemMapArr[MemMapCount].Len = 0x14;
    MemMapArr[MemMapCount].Base = Base;
    MemMapArr[MemMapCount].Size = Size;
    MemMapArr[MemMapCount].Type = 1;
    MemMapCount++;
}

static int ConvertMemoryMap()
{
    int i;
    int count;
    char *ptr;
    EFI_MEMORY_DESCRIPTOR *memptr;
    unsigned long long Base;
    unsigned long long Size;
    int has_entry = 0;
    EFI_PHYSICAL_ADDRESS RdosMemBase = RDOS_MEM;

    MemMapCount = 0;
    MemMapArr = (struct MemMapEntry *)RdosMemBase;

    if (BS->GetMemoryMap(&MemMapSize, MemMap, &MapKey, &MemDescrSize, &MemDescrVersion) == EFI_SUCCESS)
    {
        ptr = (char *)MemMap;
        count = MemMapSize / MemDescrSize;

        AddMem(0, 0x90000);

        for (i = 0; i < count; i++)
        {
            memptr = (EFI_MEMORY_DESCRIPTOR *)ptr;

            switch (memptr->Type)
            {
                case EfiLoaderCode:
                case EfiLoaderData:
                case EfiBootServicesCode:
                case EfiBootServicesData:
                case EfiConventionalMemory:
                    if (has_entry)
                    {
                        if (Base + Size == memptr->PhysicalStart)
                            Size += memptr->NumberOfPages << 12;
                        else
                        {
                            if (Base >= 0x90000)
                                AddMem(Base, Size);
                            Base = memptr->PhysicalStart;
                            Size = memptr->NumberOfPages << 12;
                        }
                    }
                    else
                    {
                        Base = memptr->PhysicalStart;
                        Size = memptr->NumberOfPages << 12;
                        has_entry = 1;
                    }
                    break;

                default:
                    if (has_entry)
                    {
                        if (Base >= 0x90000)
                            AddMem(Base, Size);
                        has_entry = 0;
                    }
                    break;
            }
            ptr += MemDescrSize;
        }

        if (has_entry)
            AddMem(Base, Size);

        DumpMem();

        return 1;
    }

    return 0;
}

Octocontrabass wrote:
rdos wrote:So the VBE interface tries to enable protected mode and so cannot be used in V86 mode.
Perhaps it's time to seriously consider an x86 emulator instead of V86 mode?

Nah, I don't think so.

I think a better option is that the first AP core would setup VBE and collect the relevant information before leaving real mode. Then the BIOS is free to do whatever it likes as long as it returns in real mode. Well, even if it doesn't, then a second init could fix that too. This way the VBE initialization can occur after device drivers are initialized and the environment is setup. The obvious alternative is to do it in the boot-loader, but that will be a lot more tricky.

If the target processor doesn't have APIC or more than one core, then it could be done in the usual way. Older systems have much better VBE implementations, at least as long as they are not too old, but then text-mode would be the way to go.

Posted: **Thu Dec 17, 2020 11:32 am**

rdos wrote:Here is the code related to the memory map:

I don't see anything here that would cause GetMemoryMap() to return 0-byte descriptors, but I do see that you're assuming memory at specific addresses will be free. If you have code elsewhere that writes into arbitrary memory like that, it could explain the odd behavior.

rdos wrote:I think a better option is that the first AP core would setup VBE and collect the relevant information before leaving real mode. Then the BIOS is free to do whatever it likes as long as it returns in real mode.

I'm not sure giving the BIOS that much freedom is a good idea. Microsoft certainly didn't think so when they put an x86 emulator in Windows.

Posted: **Thu Dec 17, 2020 3:49 pm**

this is absolutely no no no.

Code: Select all

EFI_PHYSICAL_ADDRESS RdosMemBase = RDOS_MEM;

/* ... */
    MemMapArr = (struct MemMapEntry *)RdosMemBase;

if you really need that particular address, then try this:

Code: Select all

EFI_PHYSICAL_ADDRESS RdosMemBase = RDOS_MEM;

/* ... */

    Status = AllocatePages(AllocateAddress, EfiLoaderData, NeededNumberOfPages, &RdosMemBase);
    if(Status == EFI_OUT_OF_RESOURCES || Status == EFI_NOT_FOUND){
        ST->ConOut->OutputString(
            ST->ConOut,
            L"Oopsie, our reliance on the hardcoded address didn't work out on this machine...\r\n"
            L"trying to allocate not above our preferred range...\r\n"
        );
       RdosMemBase = RDOS_MEM + (NeededNumberOfPages << EFI_PAGE_SIZE_EXPONENT); /* this is the address of the first page, which we don't want to touch */
       Status = AllocatePages(AllocateMaxAddress, EfiLoaderData, NeededNumberOfPages, &RdosMemBase);
       if(Status == EFI_OUT_OF_RESOURCES || Status == EFI_NOT_FOUND){
          ST->ConOut->OutputString(
            ST->ConOut,
            L"Oopsie 2, our reliance on the hardcoded range address didn't work out as well on this machine...\r\n"
            L"trying to allocate the buffer at any address, just as it is supposed to be...\r\n"
          );

          Status = AllocatePages(AllocateAnyPages, EfiLoaderData, NeededNumberOfPages, &RdosMemBase);
          if(Status != EFI_SUCCESS){
              ST->ConOut->OutputString(
                ST->ConOut,
                L"Fatal error, couldn't allocate any address\r\n"
                L"shutting down the system...\r\n"
              );
              RS->ResetSystem(EfiResetShutdown, EFI_SUCCESS, 0, NULL);
          }
       }else if(Status == EFI_INVALID_PARAMETER){
            FATAL_ERROR();
       }
    }else if(Status == EFI_INVALID_PARAMETER){
        FATAL_ERROR();
    }

    /* here, RdosMem holds the valid address for MemMapArr */
    MemMapArr = (struct MemMapEntry *)RdosMemBase;

And you are sure, that your EFI_MEMORY_DESCRIPTOR isn't bytepacked like the MemMapEntry struct?

Posted: **Thu Dec 17, 2020 4:11 pm**

Octocontrabass wrote:
rdos wrote:I think a better option is that the first AP core would setup VBE and collect the relevant information before leaving real mode. Then the BIOS is free to do whatever it likes as long as it returns in real mode.
I'm not sure giving the BIOS that much freedom is a good idea. Microsoft certainly didn't think so when they put an x86 emulator in Windows.

I've experimented with it, and it works. It was a bit more complicated than I thought since I first need to use function 4F00 to get supported modes, and then for each mode I need to use 4F01 to get the info, and lastly I need to use 4F02 to set the best mode. However, using shared memory in real mode I could implement "IPC" in real mode so I could do these functions without rebooting the AP core multiple times. After I'm done setting the mode, I could just let the AP core do the normal boot. Another challenge was that real mode could only use a single 4k page for code & data.

The VBE interface supports the real display resolution of 1920x1020 in 32-bit mode using LFB at 0x60000000.

Posted: **Thu Dec 17, 2020 4:17 pm**

zaval wrote:this is absolutely no no no.

Code: Select all

EFI_PHYSICAL_ADDRESS RdosMemBase = RDOS_MEM;

/* ... */
    MemMapArr = (struct MemMapEntry *)RdosMemBase;

if you really need that particular address, then try this:

Code: Select all

EFI_PHYSICAL_ADDRESS RdosMemBase = RDOS_MEM;

/* ... */

    Status = AllocatePages(AllocateAddress, EfiLoaderData, NeededNumberOfPages, &RdosMemBase);
    if(Status == EFI_OUT_OF_RESOURCES || Status == EFI_NOT_FOUND){
        ST->ConOut->OutputString(
            ST->ConOut,
            L"Oopsie, our reliance on the hardcoded address didn't work out on this machine...\r\n"
            L"trying to allocate not above our preferred range...\r\n"
        );
       RdosMemBase = RDOS_MEM + (SOME_SIZE_IN_PAGES << EFI_PAGE_SIZE_EXPONENT); /* this is the address of the first page, which we don't want to touch */
       Status = AllocatePages(AllocateMaxAddress, EfiLoaderData, NeededNumberOfPages, &RdosMemBase);
       if(Status == EFI_OUT_OF_RESOURCES || Status == EFI_NOT_FOUND){
          ST->ConOut->OutputString(
            ST->ConOut,
            L"Oopsie 2, our reliance on the hardcoded range address didn't work out as well on this machine...\r\n"
            L"trying to allocate the buffer at any address, just as it is supposed to be...\r\n"
          );

          Status = AllocatePages(AllocateAnyPages, EfiLoaderData, NeededNumberOfPages, &RdosMemBase);
          if(Status != EFI_SUCCESS){
              ST->ConOut->OutputString(
                ST->ConOut,
                L"Fatal error, couldn't allocate any address\r\n"
                L"shutting down the system...\r\n"
              );
              RS->ResetSystem(EfiResetShutdown, EFI_SUCCESS, 0, NULL);
          }
       }else if(Status == EFI_INVALID_PARAMETER){
            FATAL_ERROR();
       }
    }else if(Status == EFI_INVALID_PARAMETER){
        FATAL_ERROR();
    }

    /* here, RdosMem holds the valid address for MemMapArr */
    MemMapArr = (struct MemMapEntry *)RdosMemBase;

And you are sure, that your EFI_MEMORY_DESCRIPTOR isn't bytepacked like the MemMapEntry struct?

I only posted part of the code. The full code can be found here: http://rdos.net/vc/viewvc.cgi/rdos/trun ... iew=markup

The RDOS image is loaded like this:

Code: Select all

static int LoadRdosBinary()
{
    unsigned int FileSize = MenuArr[SelectedRow].FileSize;
    RdosImagePages = FileSize / 0x1000 + 1;
    int ok = 0;
    int i;

    printf("Booting: <");
    ST->ConOut->OutputString(ST->ConOut, MenuArr[SelectedRow].FileName);
    printf(">, %d bytes\n\r", FileSize);

    Fs = MenuArr[SelectedRow].Volume;

    if (Fs->OpenVolume(Fs, &Root) == EFI_SUCCESS)
    {
        if (Root->Open(Root, &FileHandle, MenuArr[SelectedRow].FileName, EFI_FILE_MODE_READ, EFI_FILE_READ_ONLY | EFI_FILE_HIDDEN | EFI_FILE_SYSTEM) == EFI_SUCCESS)
        {
            if (BS->AllocatePages(AllocateAddress, EfiRuntimeServicesData, RdosImagePages, &RdosImageBase) == EFI_SUCCESS)
            {
                printf("Image base: %08lX\n\r", RdosImageBase);

                FileHandle->SetPosition(FileHandle, 0);
                FileHandle->Read(FileHandle, &FileSize, RdosImageBase);

//                for (i = 0; i < RdosImagePages; i++)
//                {
//                    FileHandle->Read(FileHandle, 0x1000, RdosImageBase);
//                    RdosImageBase += 0x1000;
//                }

                ok = 1;
            }
            else
                printf("Failed to allocate fixed memory for RDOS boot\n\r");

            FileHandle->Close(FileHandle);
        }
        else
            printf("Cannot open file\n\r");

        Root->Close(Root);
    }
    else
        printf("Cannot open volume\n\r");

    return ok;
}

The loader part (which I transfer control to from EFI) is loaded like this:

Code: Select all

static int LoadBootLoader()
{
    int ok = 0;
    int size = RdosLoaderPages * 0x1000;

    if (sizeof(void *) == 4)
        strcpy(str, "efi\\rdos\\boot32.bin");
    else
        strcpy(str, "efi\\rdos\\boot64.bin");

    printf("Loading: ");
    printf(str);
    printf("\n\r");

    ConvertToWide(wstr, str);

    Fs = MenuArr[SelectedRow].Volume;

    if (Fs->OpenVolume(Fs, &Root) == EFI_SUCCESS)
    {
        if (Root->Open(Root, &FileHandle, wstr, EFI_FILE_MODE_READ, EFI_FILE_READ_ONLY | EFI_FILE_HIDDEN | EFI_FILE_SYSTEM) == EFI_SUCCESS)
        {
            if (BS->AllocatePages(AllocateAddress, EfiBootServicesData, RdosLoaderPages, &RdosLoaderBase) == EFI_SUCCESS)
            {
                printf("Loader base: %08lX\n\r", RdosLoaderBase);

                if (FileHandle->Read(FileHandle, &size, RdosLoaderBase) == EFI_SUCCESS)
                {
                    ok = 1;

                    short int us = *(short int *)RdosLoaderBase;
                    printf("Loader start: %04hX\n\r", us);
                }
                else
                {
                    BS->FreePages(RdosLoaderBase, RdosLoaderPages);
                    printf("Failed to read RDOS loader\n\r");
                }
            }
            else
                printf("Failed to allocate fixed memory for RDOS loader\n\r");

            FileHandle->Close(FileHandle);
        }
        else
            printf("Cannot open loader file\n\r");

        Root->Close(Root);
    }
    else
        printf("Cannot open volume\n\r");

    return ok;
}

Posted: **Thu Dec 17, 2020 4:36 pm**

yeah, but as Octacontrabass pointed out, your assigning predefined address to the buffer (which you immediately use btw) is definitely a way to break things. all memory allocations should go through UEFI memory allocation servies. what I suggested is a quick way to check that particular function of yours (ConvertMemoryMap()). try to paste it instead of using hardcoded address and see if that changes anything.

Posted: **Thu Dec 17, 2020 4:46 pm**

zaval wrote:yeah, but as Octacontrabass pointed out, your assigning predefined address to the buffer (which you immediately use btw) is definitely a way to break things. all memory allocations should go through UEFI memory allocation servies. what I suggested is a quick way to check that particular function of yours (ConvertMemoryMap()). try to paste it instead of using hardcoded address and see if that changes anything.

I'm not. The buffer is a static array:

Code: Select all

char MemMapBuf[4096];
EFI_MEMORY_DESCRIPTOR *MemMap = (EFI_MEMORY_DESCRIPTOR *)MemMapBuf;
unsigned int MemMapSize = 4096;
unsigned int MapKey;
unsigned int MemDescrSize;
unsigned int MemDescrVersion;

Although, if the memory map requires more than 4k then it might cause trouble.

Posted: **Thu Dec 17, 2020 5:18 pm**

not that, this one:

Code: Select all

    EFI_PHYSICAL_ADDRESS RdosMemBase = RDOS_MEM;

    MemMapCount = 0;
    MemMapArr = (struct MemMapEntry *)RdosMemBase;

In the function ConvertMemoryMap(), you assign to global pointer MemMapArr the hardcoded address (above). then you use it in other functions and in that function as well. this is wrong. you need to use UEFI memory allocation services for initializing that pointer (see my example above).

PS. I looked at the file you posted, and unlike two other bases:

Code: Select all

#define RDOS_LOADER 0x110000
#define RDOS_MEM  0x400
#define RDOS_BASE 0x121000

where you do use AllocatePages() with the AllocateAddress allocation type, this RDOS_MEM value is just assigned to the global pointer MemMapArr and used straight (as your internal memory map array) as if you are sure you aren't messing up FW structures.
further, before converting from UEFI memory map to your own memory map, you do this:

Code: Select all

        AddMem(0, 0x90000);

which means, you count this range as always free memory. why?

Posted: **Fri Dec 18, 2020 3:16 am**

zaval wrote:not that, this one:
Code: Select all
    EFI_PHYSICAL_ADDRESS RdosMemBase = RDOS_MEM;

    MemMapCount = 0;
    MemMapArr = (struct MemMapEntry *)RdosMemBase;
In the function ConvertMemoryMap(), you assign to global pointer MemMapArr the hardcoded address (above). then you use it in other functions and in that function as well. this is wrong. you need to use UEFI memory allocation services for initializing that pointer (see my example above).

PS. I looked at the file you posted, and unlike two other bases:
Code: Select all
#define RDOS_LOADER 0x110000
#define RDOS_MEM  0x400
#define RDOS_BASE 0x121000
where you do use AllocatePages() with the AllocateAddress allocation type, this RDOS_MEM value is just assigned to the global pointer MemMapArr and used straight (as your internal memory map array) as if you are sure you aren't messing up FW structures.
further, before converting from UEFI memory map to your own memory map, you do this:
Code: Select all
        AddMem(0, 0x90000);
which means, you count this range as always free memory. why?

The short answer why I assume that "DOS" memory has a certain layout is a bit of legacy. It's also because the startup memory allocation in the kernel is designed to start using memory from the second page and up. Thus, it's more or less a requirement that FW doesn't use "DOS" memory for the firmware. I think I picked 0x400 for the memory map based on the assumption that the legacy BIOS & DOS data areas would not be relevant for EFI FW. This might be faulty for some systems, but then I simply cannot support those.

There is also a requirement for a fixed below 4G area where the processor can switch between long mode and protected mode. I decided to put this just above the DOS high memory area, and then I also placed the transient loader at this position. Since the 64-bit loader needs to switch back to protected mode, by disabling paging, it must reside at a fixed address below 4G.

The OS image itself can reside at any position below 4G since it is remapped with paging, but it is easier if it has a fixed position. So, if the area is just allocated then it might be mapped above 4G if the computer has more than 4G of memory.

Things would be much easier & less fixed if EFI supported allocating memory below 4G, but it doesn't as far as I know.

An alternative to using fixed memory in the EFI interface would be to let the 64-bit loader find free areas below 4G and then copy the RDOS image to those and fix up the memory map. However, since no EFI BIOS this far has denied the fixed allocations, I've not had reason to do this.

Still, I don't think the placement of the memory structure at 0x400 is causing the GetMemoryMap to fail since it's used after the call to GetMemoryMap to reformat the information, and it is the MemDescrSize that contains 0 after the call to GetMemoryMap. A possible issue might be that EFI expects it to contain the size on entry, but it is not initialized before passing it as a parameter (so should be 0).

OSDev.org

Cannot boot on new Intel i3 laptop

Re: Cannot boot on new Intel i3 laptop

Re: Cannot boot on new Intel i3 laptop

Re: Cannot boot on new Intel i3 laptop

Re: Cannot boot on new Intel i3 laptop

Re: Cannot boot on new Intel i3 laptop

Re: Cannot boot on new Intel i3 laptop

Re: Cannot boot on new Intel i3 laptop

Re: Cannot boot on new Intel i3 laptop

Re: Cannot boot on new Intel i3 laptop

Re: Cannot boot on new Intel i3 laptop

Re: Cannot boot on new Intel i3 laptop

Re: Cannot boot on new Intel i3 laptop

Re: Cannot boot on new Intel i3 laptop

Re: Cannot boot on new Intel i3 laptop

Re: Cannot boot on new Intel i3 laptop