OSDev.org

Posted: **Thu Sep 10, 2020 7:40 am**

Hi,

My boot loader works on all virtual machines and on all real hardware I and my users have tested on, except one. I got a ticket that on a particular real hardware it freezes. Working together with the ticker opener, we've narrowed down the issue to the ExitBootServices call. So the problem is, that on one real machine of many, ExitBootServices freezes and never returns control.

The simplified code:

Code: Select all

map_key = GetMemoryMap();
clear_screen(black);
status = ExitBootServices(map_key);
clear_screen(white);
if(status not success) print error;
clear_screen(blue);

And the result is, the screen is black, no error printed, and neither is the second clear screen executed. (FYI: I've used clear screen because Print() allocates memory and therefore changes the map key required by ExitBootServices).

Again, the code runs on many machines as expected, but ExitBootServices never returns control on one particular machine. Anybody have seen such an issue? Any suggestions why that could be?

Cheers,
bzt

PS: according to the ticket owner, he has flashed the UEFI firmware with a ROM downloaded from the manufacturer's page. I don't have direct access to that machine, but I can proxy any questions or tests to him and let you know the results.

Posted: **Thu Sep 10, 2020 1:49 pm**

do not call anything between GetMemoryMap() and ExitBootServices().

Posted: **Thu Sep 10, 2020 2:04 pm**

I don't know if this is related or not, but your bootloader is misusing the MP Services Protocol. You must return control of the APs to the firmware before calling ExitBootServices().

Posted: **Fri Sep 11, 2020 6:23 am**

zaval wrote:do not call anything between GetMemoryMap() and ExitBootServices().

Nothing is called. And it works on all machines, except one. Besides, ExitBootServices should return with an error code if map_key changed, but surely not freeze.

Octocontrabass wrote:I don't know if this is related or not, but your bootloader is misusing the MP Services Protocol. You must return control of the APs to the firmware before calling ExitBootServices().

What do you mean by "return control to the firmware"? The MP protocol has a StartThisAP function that requires a function pointer to be executed. I call it like this:

Code: Select all

uefi_call_wrapper(mp->StartupThisAP, 7, mp, bootboot_startcore, i, Event, 0, (VOID*)i, NULL);

And then on the BSP, after I finished with everything (including ExitBootServices) I also call

Code: Select all

bootboot_startcore((VOID*)bsp_num);

If that function would return control to the firmware, then how could it execute the kernel on that core? According to the spec, OS Loaders are not supposed to return.

Cheers,
bzt

Posted: **Fri Sep 11, 2020 7:48 am**

bzt wrote:What do you mean by "return control to the firmware"?

The function pointer you pass to StartupThisAP() must be to a function that returns. Said function must return before you call ExitBootServices().

bzt wrote:If that function would return control to the firmware, then how could it execute the kernel on that core?

Using INIT and STARTUP IPIs after calling ExitBootServices().

Posted: **Fri Sep 11, 2020 10:24 am**

Octocontrabass wrote:The function pointer you pass to StartupThisAP() must be to a function that returns. Said function must return before you call ExitBootServices().

Where did you get that? Both the UEFI PI and the EDK header states that StartupThisAP() can be executed in non-blocking mode, when the BSP is not supposed to wait for the function to return.

If WaitEvent is NULL, execution is in blocking mode. The BSP waits until the AP finishes or TimeoutInMicroSecondss expires. Otherwise, execution is in non-blocking mode. BSP proceeds to the next task without waiting for the AP.

There's absolutely nothing about the function must return before ExitBootServices (it just states that MP Services cannot be called after ExitBootServices). UEFI PI section "Non-Blocking Execution Support" on page 2-191 mentions nothing about the function returning. It only mentions using SingalEvent when timeout occurs, that's all (however it also mentions that zero timeout means infinity, aka no signal).

Octocontrabass wrote:Using INIT and STARTUP IPIs after calling ExitBootServices().

The whole point of MP Services is to provide a platform independent way of initializing the APs, from MpService.h:

Code: Select all

  MP Services Protocol is hardware-independent. Most of the logic of this protocol
  is architecturally neutral. It abstracts the multi-processor environment and
  status of processors, and provides interfaces to retrieve information, maintain,
  and dispatch.

INIT and STARTUP IPI are not platform independent, and if a loader must use those, then what's the point of MP Services anyway? Is this yet another epic failure on the Intel engineers part?

Another question, STARTUP IPI requires a code block under 1M for real mode startup. How can you allocate a memory for that, when UEFI is managing the memory? How can you tell AllocPages (or any other UEFI memory allocation function) that you need a page that's below 1M? How can you be sure that there's any free memory left below 1M at all? This makes no sense. From the doc:

Caution: Although the Allocate services allow for specific memory allocation, do not allocate specific addresses in a UEFI driver. Allocating buffers at a specific address could result in errors, including a catastrophic failure on some platforms.

Cheers,
bzt

Posted: **Fri Sep 11, 2020 2:16 pm**

bzt wrote: Another question, STARTUP IPI requires a code block under 1M for real mode startup. How can you allocate a memory for that, when UEFI is managing the memory? How can you tell AllocPages (or any other UEFI memory allocation function) that you need a page that's below 1M? How can you be sure that there's any free memory left below 1M at all? This makes no sense. From the doc:
Caution: Although the Allocate services allow for specific memory allocation, do not allocate specific addresses in a UEFI driver. Allocating buffers at a specific address could result in errors, including a catastrophic failure on some platforms.

Are you writing a UEFI driver? That guide is for driver writers, there is no such a limitation for an OSL - to ask allocation functions about specific address range (or even address), consult the UEFI specification. Of course, the requested address might be not available, but if you decided to spin up secondary CPUs in the OSL (yet - in the UEFI Boot Services stage), it's your problem.

As of the original problem, if your clear_screen() thing isn't calling anything, then the answer why ExitBootServices() hangs on that particular machine could be anything - who knows. it's a bug.

either in your OSL or UEFI of the machine. Could you avoid touching secondary processors? If so, try it, because it looks like the most suspicious culprit.

Posted: **Fri Sep 11, 2020 2:31 pm**

bzt wrote:There's absolutely nothing about the function must return before ExitBootServices (it just states that MP Services cannot be called after ExitBootServices). UEFI PI section "Non-Blocking Execution Support" on page 2-191 mentions nothing about the function returning. It only mentions using SingalEvent when timeout occurs, that's all (however it also mentions that zero timeout means infinity, aka no signal).

UEFI PI section "EFI_MP_SERVICES_PROTOCOL" on page 2-181 explains it.

Modules that use this protocol must guarantee that all non-blocking mode requests on all APs have been completed before the UEFI event EFI_EVENT_LEGACY_BOOT_GUID or EFI_EVENT_GROUP_EXIT_BOOT_SERVICES is signaled.

bzt wrote:Is this yet another epic failure on the Intel engineers part?

Yes!

bzt wrote:Another question, STARTUP IPI requires a code block under 1M for real mode startup. How can you allocate a memory for that, when UEFI is managing the memory?

UEFI isn't managing the memory after ExitBootServices().

bzt wrote:How can you tell AllocPages (or any other UEFI memory allocation function) that you need a page that's below 1M?

I'd try AllocateMaxAddress if I really needed to set up the trampoline before calling ExitBootServices().

bzt wrote:How can you be sure that there's any free memory left below 1M at all?

Linux never seems to have any trouble.

bzt wrote:
Caution: Although the Allocate services allow for specific memory allocation, do not allocate specific addresses in a UEFI driver. Allocating buffers at a specific address could result in errors, including a catastrophic failure on some platforms.

Those guidelines are for UEFI drivers. I don't think a bootloader for one specific platform needs to worry that it won't work on other platforms.

Posted: **Sat Sep 12, 2020 4:43 am**

Octocontrabass wrote:
bzt wrote:Is this yet another epic failure on the Intel engineers part?
Yes!

We should put their hands in gold (so that they can never ever use a keyboard again). That's just sad. Who gave those guys a diploma anyway...?

Octocontrabass wrote:UEFI isn't managing the memory after ExitBootServices().

But it does not release memory for run time services either. UEFI drivers will continue to use memory allocated as run time data.

Octocontrabass wrote:
bzt wrote:How can you be sure that there's any free memory left below 1M at all?
Linux never seems to have any trouble.

Linux is extremely poorly written in this regard. Linux's SMP code isn't UEFI specific, and we know that under BIOS the < 640k memory is indeed free (not fully used by EBDA for sure). Btw, Linux uses a hardcoded value as default, and it does not check if it could really find a free RAM entry below 640k to replace the address in "bios_start" variable. If an UEFI driver allocates all memory below 0x413 << 10 then a crash with Linux is guaranteed. Not just for the AP trampoline code (set here), but for the uncompression too...

Octocontrabass wrote:Those guidelines are for UEFI drivers. I don't think a bootloader for one specific platform needs to worry that it won't work on other platforms.

You are missing the whole point. UEFI drivers are running along with your bootloader, and they are started before your code! Therefore any UEFI driver could "steal" the low memory from you.

Unlike BIOS, UEFI does not guarantee that any memory below 1M going to be usable, which is a big problem for SMP. Yes, you could assume that a certain address is going to be fine, but that will result in exactly the same runs-on-many-except-some-machines problem just like with MP Services

Cheers,
bzt

Posted: **Sat Sep 12, 2020 11:21 am**

bzt, UEFI is CPU independent and that's why it does not guarantee any weird assumptions about particular address range availability. if you are so inclined to start secondary processors in your OSL, then at least do it after you get control over the address map, since it's needed if you want particular address range availabilty on any machine. there is no way, that all lower 1MB of RAM would be occupied with Rintime Services typed memory. but you already know the right answer - the problem is in your strange decision, not in the FW. why you can't start secondary processors from inside your kernel initialization?

Posted: **Sat Sep 12, 2020 11:39 am**

bzt wrote:Linux is extremely poorly written in this regard. Linux's SMP code isn't UEFI specific, and we know that under BIOS the < 640k memory is indeed free (not fully used by EBDA for sure). Btw, Linux uses a hardcoded value as default, and it does not check if it could really find a free RAM entry below 640k to replace the address in "bios_start" variable. If an UEFI driver allocates all memory below 0x413 << 10 then a crash with Linux is guaranteed. Not just for the AP trampoline code (set here), but for the uncompression too...

On x86, SMP is not specific to the firmware. By the way, thank the Intel engineers for not f'ing that up, at least. Other architectures are not so lucky. Today I read the Linux code for starting secondary processors on PowerPC. It calls OpenFirmware. There is so little of a standard of how to do it there that even Linux, which sometimes resembles "NIH the OS" prefers to just call the firmware to do it.

Anyway, the address given is 0x104c00. Pardon my academic elitism, but 6 digits is more than 5 digits, so this is more than 1MB. If a driver did as you proposed, it would allocate all of low memory. I would wager Windows wouldn't work very well with wonky allocations like that, either. Dammit, ran out of w-words. Low memory is simply required for SMP startup on x86, since any AP starts in 16-bit real mode and cannot access any memory beyond 1MB. Plus, the INIT IPI always specifies a start address in low memory.

bzt wrote:Unlike BIOS, UEFI does not guarantee that any memory below 1M going to be usable,

Any such system will be unable to support SMP with any major operating system, and their customers are going to be very upset at such a development. Especially since both Intel and AMD are pushing more and more ridiculous numbers of cores and threads, because apparently we have reached peak single core performance. Having a system have not even a single page of memory free in low memory is simply not workable on x86. UEFI might not guarantee this, but the facts of the matter do.

Posted: **Sat Sep 12, 2020 12:27 pm**

nullplan wrote:On x86, SMP is not specific to the firmware.

Yes, it is. SMP initialization code is very very architecture, platform, and sometimes firmware specific. On x86 you have at least three firmware, BIOS/MP tables, BIOS/ACPI and UEFI/ACPI at least. But for example, with old RaspberryPi firmware, you would have to issue a Mailbox call to awake APs. Then they've changed the firmware to start all APs at once. And with the latest firmware, only BSP runs and now you can release the spinlocks with direct memory writes without touching the firmware.

According to the UEFI PI spec, the whole purpose of MP Services supposed to be hiding all of that and provide a platform independent, common interface for SMP in the firmware.

nullplan wrote:By the way, thank the Intel engineers for not f'ing that up, at least.

What good is MP Services from the OS perspective if the kernel can't use it? It needs all INIT IPI + STARTUP IPIs and real mode and prot mode trampoline implemented in UEFI. Then all that architecture specific sh*t has to implemented *again* in a kernel later when the required memory might be not free, just because APs must be stopped on ExitBootServices? That's just stupid. Why shouldn't a kernel use MP Services in the first place? (Which works on many boards, BTW)

With MP Services, the initialization (and protmode/longmode trampoline) happens *before* any UEFI driver started (in CpuPkg to be precise), therefore any driver can freely use low mem. Then nobody cares what memory the UEFI drivers are allocating, because the APs are already started up and awaiting in native mode. The StartupThisAP method does not actually start up the AP, it just dispatches a native mode function on the AP. (And Hell, even the method name suggests it's for starting up the APs and not do-something-boot-time-only-stuff!!!)

This way MP Services interface is really and truly architecture independent, but what good does that make if you can't use it from your OS?

nullplan wrote:Anyway, the address given is 0x104c00. Pardon my academic elitism, but 6 digits is more than 5 digits, so this is more than 1MB.

That *0x413 << 10 address was copy'n'pasted right from the Linux source. That's why I wrote if the loop in lines 73 - 104 fails to modify the "bios_start" variable, then a crash is guaranteed. BTW, the value of 0x413 is NOT checked in any way either... It could be right in the middle of an UEFI driver allocated page!

nullplan wrote:
bzt wrote:Unlike BIOS, UEFI does not guarantee that any memory below 1M going to be usable,
Any such system will be unable to support SMP with any major operating system

Yeah, because Intel engineers messed this up too. They could easily boot SMP if MP Services were just simply run time services... Or if APs weren't expected to be stopped on ExitBootServices call (many firmware allows that, but it should be in the spec).

Cheers,
bzt

Posted: **Sat Sep 12, 2020 2:11 pm**

All protocols are Boot Service time things and cannot be Runtime Service time things. Drivers and the FW core itself, implementing them, uses Protocol management Boot Services during protocols operation, among other things, belonging to Boot Services. None protocol was intended for usage by an OS, it's all for booting OSs. and honestly, that MP stuff is just a show off thingy. The need of redoing SMP initialization by the OS is not a flaw of the FW, just like resetting the GPU isn't, because the FW does that for its own purposes and what it has done or hasn't, shouldn't be a concern for an OS, since the latter still needs to do way much more and often - the other way, because FW initializes something just as much as needed for loading the OS, the OS does in full. MP isn't needed for OS loading, it was put there probably because of the "oh, look, let's make it SMP, it would be so rad" reason.

Posted: **Sat Sep 12, 2020 10:09 pm**

bzt wrote: On x86 you have at least three firmware, BIOS/MP tables, BIOS/ACPI and UEFI/ACPI at least.

MP tables are of interest only to historians and those OS developers too lazy to integrate ACPICA or similar. I for one do not plan to support multi-processor 486 boards any time soon, so MP tables are about as relevant to me as the river level of the Amazon from last year. And the only difference between BIOS/ACPI and UEFI/ACPI is how to find the XSDT.

bzt wrote:RaspberryPi

Yeah, I did say x86, didn't I? On all other architectures, these things are very flexible, as I tried to point out with the PowerPC example.

bzt wrote:According to the UEFI PI spec, the whole purpose of MP Services supposed to be hiding all of that and provide a platform independent, common interface for SMP in the firmware.

Yup, and then they added a requirement that the APs return to the firmware before calling ExitBootServices(), thereby rendering the whole thing useless. It's almost a good system. I have come to realize that "almost" is just a polite way of saying "not".

Looking at the UEFI membership list, I wonder if this is yet another case of diffusion of responsibility. Can you tell me who broke the spec? No, this way it could have been anyone. And you have no-one to blame.

bzt wrote:What good is MP Services from the OS perspective if the kernel can't use it?

No good, obviously. Apparently they needn't have bothered.

bzt wrote:That *0x413 << 10 address was copy'n'pasted right from the Linux source.

I misread the code. I thought bios_start was assigned an address of 0x413 << 10. No. It is assigned the value at address 0x413, times 1024. Apparently, the BIOS data area contains the RAM size in kB at that location. Of course, Linux assumes the BDA is present unless UEFI is present. The search loop would work great if it could return failure. However, if that particular loop can't find anything, your system is broken anyway, since the kernel cannot be decompressed, then. If they wanted to change that, they'd have to change the architecture of the code.

Posted: **Sun Sep 13, 2020 6:31 am**

zaval wrote:All protocols are Boot Service time things and cannot be Runtime Service time things.

But they could be! For example, you could switch the screen resolution from your OS any time using BIOS/VBE. But you can't do that with GOP (being a boot time service). There's absolutely no technical nor hardware related reasons why those protocols are inaccessible for the OS, except shitty UEFI design and implementations.

nullplan wrote:MP tables are of interest only to historians and those OS developers too lazy to integrate ACPICA or similar. I for one do not plan to support multi-processor 486 boards any time soon, so MP tables are about as relevant to me as the river level of the Amazon from last year. And the only difference between BIOS/ACPI and UEFI/ACPI is how to find the XSDT.

Point being, there are different methods depending on the firmware on x86 too.

nullplan wrote:Yup, and then they added a requirement that the APs return to the firmware before calling ExitBootServices(), thereby rendering the whole thing useless. It's almost a good system. I have come to realize that "almost" is just a polite way of saying "not".

Yeah, so true.

nullplan wrote:Looking at the UEFI membership list, I wonder if this is yet another case of diffusion of responsibility. Can you tell me who broke the spec? No, this way it could have been anyone. And you have no-one to blame.

Correction, you can blame all the UEFI participants equialy

So the best we can do is blindly use some low mem address for the trampoline code and hoping it won't crash the system... What a real progress, well done UEFI!

Cheers,
bzt

ps: BIOS was simpler, (much-much) faster, more stable and safer, cleaner and an OS could use all BIOS services at run-time! It was better in all aspects save it was real-mode. But you could use VM86 mode or libx86emu, so there was a way, while with UEFI you're just screwed. No-one in their sane minds can say UEFI is better! But this is the sh*t forced on all OS developers these days... We have a saying: running around in the d*ckforest with an open mouth, that's what UEFI is!

OSDev.org

UEFI question

UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question