OSDev.org

Posted: **Sun Sep 13, 2020 6:41 am**

bzt wrote:So the best we can do is blindly use some low mem address for the trampoline code and hoping it won't crash the system... What a real progress, well done UEFI!

You could look at the EFI memory map and find a region under 1M that is free. If none is found, then we have a problem.

Posted: **Sun Sep 13, 2020 2:17 pm**

bzt wrote: But they could be! For example, you could switch the screen resolution from your OS any time using BIOS/VBE. But you can't do that with GOP (being a boot time service). There's absolutely no technical nor hardware related reasons why those protocols are inaccessible for the OS, except shitty UEFI design and implementations.

Then you would be unhappy with the way, that "better" UEFI managed a thing for you and complain, that your OS would do it better. The only thing is needed for such an attitude - is your wish to "laugh" at "intel engineers", that, of course, are "dumb". it's not serious.
The point is, the "technical" reason behind the unavailability of all those things for an OS is that, those things ARE FW itself and for the FW and OSL to boot your OS and nothing more and then your OS gets FULL control over the machine, so you can do whatever you want. they just physically go away from memory, freeing space for your OS to manage the machine. what the OS is supposed to do, in fact. If some protocols were made RTS, then all the protocol management had to be RTS too and stay in memory for all runtime. that would mean more memory would be allocated as RTS and it would need to be preserved during the OS runtime. AND protocol managing is a big part of UEFI, so it would require a lot of memory. for what? for freeing the OS from a routine, naturally belonging to it, of starting up secondary processors? maybe then the FW should take care of other OS burdens? GUI? webrowsers?

So the best we can do is blindly use some low mem address for the trampoline code and hoping it won't crash the system... What a real progress, well done UEFI!

Why "blindly"? You can pick the needed place once you took over the system memory - meaning after ExitBootServices(). Definitely, there will be free space below 1MB. Why you ignore this obvious scenario? UEFI IS a single CPU thing, just as any sane FW should be. bringing up secondary processors IS the OS responsibility!

Posted: **Sun Sep 13, 2020 4:45 pm**

nexos wrote:You could look at the EFI memory map and find a region under 1M that is free. If none is found, then we have a problem.

Yeah, but even if you find a region dynamically, you'll have a problem. There are instructions that must use absolute addresses, not to mention system descriptors like gdt_value etc. Since you cannot mix real mode, prot mode and long mode relocations in an object file, this sucks. You cannot rely on the compiler generating the relocation records for you, must be done manually. Not impossible, just lot of unnecessary work.

I think I'll go with a fixed address instead (like 0xF000 for example) and allocate that using AllocateAddress. If not successful, then display a warning and skip SMP alltogether. Should work in almost every machines, does not crash with faulty UEFI drivers, and I don't have to worry about searching the memory map or relocating code manually.

Cheers,
bzt

Posted: **Sat Sep 19, 2020 8:40 am**

Sigh, SMP could in fact be something that can be setup in the bootloader, if not for the UEFI requirements. This is platform initialization after all, and very platform-dependent on top of that, and needs to be done on every machine that has two or more cores. It's nothing like running the init process or starting a web browser. The EFI_MP_SERVICES_PROTOCOL could had been quite useful IMO, but that little detail rendered it useless and I dare say it practically functions as pure bloat at this point, because the UEFI implementers won't bother to test something that's not required to boot Windows or Linux (if they even test the latter), and I doubt any driver uses that.

bzt, I looked a bit through the BOOTBOOT design and implementation yesterday, before I discovered this topic, and also a bit now. It looks interesting to me as an idea, but currently I'd still say that hobby OS developers should write their own bootloader or use a widely tested one. On the upside, though, you seem to have a better design than multiboot (you probably just need to iron it out, same for the implementation), and I suppose it's easier for a hobbyist to contact you here to ask questions about BOOTBOOT (and help you back if they discover an issue, as I'd assume is the case here), than to contact the GRUB developers.

Anyway, in bootboot.c at line 1178, the "0x8000" in CopyMem() should probably be "ap_code", because you might later change the latter and (unlikely but possibly) forget to modify the former, especially if you decide to use the memory map to find a lower address. And at page 20 of the specification, you misleadingly (see note) write under limitations:

Page 20 of BOOTBOOT specification wrote:As it boots in protected mode, it only maps the first 4G of RAM

which probably should instead be:

Page 20 of BOOTBOOT specification wrote:If it boots in protected mode, it only maps the fist 4G of RAM

Note: I hope it's misleadingly written and not the reality, though the code seems to confirm my hopes. On the other side again, in the Hungarian edition you say "mivel" which, according to Wiktionary, means "as" in the sense of "because" and not "if" (I don't know Hungarian).

Posted: **Sat Sep 19, 2020 11:03 am**

testjz wrote:Sigh, SMP could in fact be something that can be setup in the bootloader, if not for the UEFI requirements. This is platform initialization after all, and very platform-dependent on top of that, and needs to be done on every machine that has two or more cores. It's nothing like running the init process or starting a web browser. The EFI_MP_SERVICES_PROTOCOL could had been quite useful IMO, but that little detail rendered it useless and I dare say it practically functions as pure bloat at this point, because the UEFI implementers won't bother to test something that's not required to boot Windows or Linux (if they even test the latter), and I doubt any driver uses that.

I agree!

testjz wrote:bzt, I looked a bit through the BOOTBOOT design and implementation yesterday, before I discovered this topic, and also a bit now. It looks interesting to me as an idea, but currently I'd still say that hobby OS developers should write their own bootloader or use a widely tested one.

Agreed. The problem is, Multiboot is a 32 bit protocol (with a 64 bit hack), and Grub is not available on Raspberry Pi at all. I wanted a clean, 64-bit only, multiplatform-by-design loader.

testjz wrote:On the upside, though, you seem to have a better design than multiboot (you probably just need to iron it out, same for the implementation), and I suppose it's easier for a hobbyist to contact you here to ask questions about BOOTBOOT (and help you back if they discover an issue, as I'd assume is the case here), than to contact the GRUB developers.

Thanks! The thruth is, many have tested BOOTBOOT, and they open issues on the gitlab's issue tracker. I prefer that over contacting me here, but that's fine too. It's already been used for years now, it's ironed out pretty much

testjz wrote:Anyway, in bootboot.c at line 1178, the "0x8000" in CopyMem() should probably be "ap_code", because you might later change the latter and (unlikely but possibly) forget to modify the former, especially if you decide to use the memory map to find a lower address.

Actually no good. The code that's copied to 0x8000 is strictly compiled at that address (see smp.S). The reason for this is because you can't use "org" directive, and without gcc is not smart enough to generate relative offsets. That's why I had to manually calculate all addresses in the relocated trampoline code. If you change the address, then you'll also have to recalculate all those real-mode and prot-mode addresses too in the Assembly. I might have used a define, but loading at an arbitrary address would require run-time relocations for sure.

testjz wrote:And at page 20 of the specification, you misleadingly (see note) write under limitations:
Page 20 of BOOTBOOT specification wrote:As it boots in protected mode, it only maps the first 4G of RAM
which probably should instead be:
Page 20 of BOOTBOOT specification wrote:If it boots in protected mode, it only maps the fist 4G of RAM
Note: I hope it's misleadingly written and not the reality, though the code seems to confirm my hopes. On the other side again, in the Hungarian edition you say "mivel" which, according to Wiktionary, means "as" in the sense of "because" and not "if" (I don't know Hungarian).

Well, it does all boot process in protected mode. There's only a very-very minimal 64 bit code in the loader (sets up segment registers, stack and then jumps to the kernel's long mode entry point). The same stands for the coreboot implementation, because it also runs in prot-mode. UEFI version on the other hand runs entirely in long mode.

Now maybe the phrasing is not the best. Running the loader in protected mode doesn't need maps over 4G, that's why it maps only 4G. Just for the records, the protocol only guarantees that the initrd will be entirely loaded into the identity mapped area, so 4G is enough, since you simply can't load it above 4G in prot-mode. The reason for this identity mapping is, that all kernels tend to set up their own paging tables anyway, so there's no point in putting lot of effort into boot time mappings.

Cheers,
bzt

Posted: **Mon Sep 21, 2020 4:02 pm**

I once again wrote more than expected. I hope it will be helpful to you and possibly anyone reading this topic.

bzt wrote:I wanted a clean, 64-bit only, multiplatform-by-design loader.

Maybe you weren't reading very carefully (it can happen to everyone), but please notice the "write their own bootloader" part.

bzt wrote:Thanks! The thruth is, many have tested BOOTBOOT, and they open issues on the gitlab's issue tracker. I prefer that over contacting me here, but that's fine too. It's already been used for years now, it's ironed out pretty much

If you say so, by our hobbyist standards that could be around 100 people, maybe 200, and that's a very optimistic estimate (I don't know the exact number). GRUB is used in millions of installations of (GNU+Linux) and other OSes. Besides, this topic proves that you still had something to iron out. It's possible that another issue can surface at any time, and I don't think that software can enter a state where nothing can be improved (the same also applies to widely tested software).

bzt wrote:Actually no good. The code that's copied to 0x8000 is strictly compiled at that address (see smp.S). The reason for this is because you can't use "org" directive, and without gcc is not smart enough to generate relative offsets. That's why I had to manually calculate all addresses in the relocated trampoline code. If you change the address, then you'll also have to recalculate all those real-mode and prot-mode addresses too in the Assembly. I might have used a define, but loading at an arbitrary address would require run-time relocations for sure.

My point is simply that you define a (constant) variable with the value 0x8000, then you hardcode 0x8000 for the same purpose instead of using the variable that you defined.

Though now that I'm looking more carefully, you also have other addresses in the 0x8000-0x80FF range, which I recommend you write them as "ap_code + offset", where "offset" is in fact a hardcoded constant.

I looked also at your linker file, and I don't understand why do you start from address 0x00000000000000000. That's usually reserved for the NULL pointer, and thus left unmapped. You might have a problem if you forget to check for a NULL pointer somewhere, and then assume the NULL pointer is valid. And this way it's harder to debug, because you won't get a "nice" page fault together with the address attempting the access (you would find the NULL pointer in O(1)), but you get a vague "something doesn't work after some point" (where you need to search in O(log(n)) using breakpoints). Also, you might overwrite parts of the bootloader, not necessarily only address 0x00000000000000000, but also neighbouring addresses if you use the pointer together with an index or offset (though you put the hash in the first page of the bootloader, so I presume it's not used after integrity has been checked).

Also, I looked a bit more into bootboot.c. I don't know how applicable the advice of not reimplementing security-related functions in your own code is here. Apart from reimplementing cryptographic functions, the best you could do would be to reuse the applicable files from a public domain project like LibTomCrypt or a permissively licensed one, possibly with some adaptations to remove defines and macros irrelevant to your project, and maybe to make the coding style consistent with yours. And then to periodically check for changes in the original code, and also to check the CVE list for possible security issues. But it's probably not very practical, so I'll spare the issue.

I also saw some minor stuff, like assigning a value to "valid" at lines 1117 and 1131 and then not using the variable. If you don't use it, then don't define it. Actually, I suspect that line 1134 might cause gcc to refrain from optimizing this out. And I see that you don't seem to have a very consistent coding style, mostly involving inter-operator spacing, putting multiple statements on the same line (I never do it with the exception of "for" loop headers), and putting single-statement "if" bodies in three different ways (same line, next line without braces, and next line with braces; I always do the last one of these as it also prevents the case where you add statements to such an "if" body and forget to then enclose it in braces; though gcc will see the alignment implying probably what you intended and will fortunately emit a warning), but that doesn't matter too much I'd suppose, except for code readabiliity and maintenance.

That said, you probably are a way more experienced developer than I am. After all you have released so many projects that are hard to do, while I haven't released anything yet. I have written a minimal kernel that booted using GRUB and that didn't even get to userspace. I have attempted some other stuff, like minimal ROM images for QEMU and Bochs, lossless compression of images and audio and toy HTTP servers, but I didn't actually complete any of them and most were abandoned earlier than 10% on the way because I'm bad at sticking with something. Currently, I'm still trying to gather as much knowledge as I can, while helping people here about things I'm familiar with. However, some things like having accessible NULL addresses, having a somewhat inconsistent coding style and (in other projects I've seen) inadequately checking for validity of return values, and (in other projects I've seen) routinely using unsafe functions (though I would classify atoi() as one of them), these things are red flags to me. I tend to switch to verbose mode.

bzt wrote:Well, it does all boot process in protected mode. There's only a very-very minimal 64 bit code in the loader (sets up segment registers, stack and then jumps to the kernel's long mode entry point). The same stands for the coreboot implementation, because it also runs in prot-mode. UEFI version on the other hand runs entirely in long mode.

Now maybe the phrasing is not the best. Running the loader in protected mode doesn't need maps over 4G, that's why it maps only 4G. Just for the records, the protocol only guarantees that the initrd will be entirely loaded into the identity mapped area, so 4G is enough, since you simply can't load it above 4G in prot-mode. The reason for this is, that all kernels tend to set up their own paging tables anyway, so there's no point in putting lot of effort into boot time mappings.
bzt

OK, it seems I misinterpreted the issue. This clears it up, thank you.

But I still don't see any reason to map unused memory because, as you say, "all kernels tend to set up their own paging tables anyway". I also find the mapping of the kernel to specific addresses a bit inflexible, but this can be overcame with some linker magic in rare cases when it would be necessary (e.g. the kernel grows larger than the almost 2M that you allow for it but, if that happens, the kernel is problematic and not the bootloader).

I'm in no way criticizing you, nor I want you to give up. Certainly, you should continue developing and refining BOOTBOOT as I think it has big potential to be one of the generally recommended bootloaders for hobbyist OS development, and it's actually already more than half-way there, maybe even two-thirds-way there. It's much more complete than Multiboot, and it's more flexible in the important stuff. It doesn't require the kernel to be loaded at a fixed physical address defined in advance by the kernel developer (which address may or may not actually exist, though usually it will), and it doesn't leave the system inadequately initialized by refraining from turning on even paging. I'm just pointing out there are many things in BOOTBOOT that could still be improved.

Regarding the original issue of this topic, maybe you would "compromise" with turning the APs on after ExitBootServices()? That would avoid the problem altogether and it probably wouldn't be a big loss (the OS kernels do it themselves after all), and maybe it would be simpler even. Also, would you trust the UEFI firmware more than your own code? (Rhetorical question)

Posted: **Mon Sep 21, 2020 7:06 pm**

testjz wrote:I once again wrote more than expected.

No worries

I like intelligent conversations.

testjz wrote:Maybe you weren't reading very carefully (it can happen to everyone), but please notice the "write their own bootloader" part.

Yes, I was. I've tried to say, this is the reason why I have written my own loader instead of choosing a widely tested one. I'm convinced that the clean design, the flexibility of BOOTBOOT and the wide range of booting environments are beneficial to others too, that's why I split it from the OS project. If someone gets stuck writing their own boot loader, they can use it. Much much simpler than switching to GRUB (both managebility-wise and development-wise).

testjz wrote:If you say so, by our hobbyist standards that could be around 100 people, maybe 200, and that's a very optimistic estimate (I don't know the exact number).

True. I'd say that's a likely estimate: let's say one user of ten takes the effort to give a star to the repo (optimistic, in reality people are lazy). The repo has 15 stars ATM. But 100 users are a damn good start

testjz wrote:GRUB is used in millions of installations of (GNU+Linux) and other OSes.

Read: GRUB is overcomplicated to support all those, and hard to use (just count how many GRUB topics are on this forum, not to mention GRUB's own forum and on stackoverflow). GRUB's config file is almost a programming language on its own right!!! Using both grub-install and grub-mkrescue have hilariously huge number of dependencies. Using mkbootimg is totally dependency-free (save libc), yet knows everything that those other two

Plus I don't care how many millions installations use GRUB, and how widely tested it is, if my target platform isn't one of them, it's no good. You can't install GRUB on a Raspberry Pi. GRUB doesn't support ARM64 officially at all (Multiboot spec is x86 and 32 bit only, and Multiboot2 likewise, talks only about uefi-amd64, like there were no other 64 bit platforms, and does not recognize the possibility of BIOS-amd64 at all).

testjz wrote:Besides, this topic proves that you still had something to iron out. It's possible that another issue can surface at any time

I don't intend to say BOOTBOOT is perfect. You are welcome to open a new issue and I'll try to solve your problem the best I can. I only say that it already runs on all mainstream virtual machines and on numerous real hardware. I won't and I can't give a guarantee that it will run on all possible hardware configurations, but as being a FOSS project, you're welcome to fix it for your machine

testjz wrote:I don't think that software can enter a state where nothing can be improved

On that, we disagree. For three reasons:
1. I don't think that a code that works needs improvement. As long as it gets the job done, don't try to fix it.
2. There's a strict "no new features" policy. The BOOTBOOT Protocol already contains the very minimum to boot a system, you can't take away anything, and there's no reason to add anything if the current feature set is enough to boot a system.
3. You probably have heard, but I like this quote very much: "Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius—and a lot of courage to move in the opposite direction." /E. F. Schuemacher/

testjz wrote:My point is simply that you define a (constant) variable with the value 0x8000, then you hardcode 0x8000 for the same purpose instead of using the variable that you defined.

Yeah, the only reason for that variable is because UEFI does not allow an address to be passed to AllocatePages. It requires an address of a variable holding the address...

testjz wrote:I looked also at your linker file, and I don't understand why do you start from address 0x00000000000000000.

What do you mean? The kernel's linker file starts at 0xffffffffffe00000.
And for the UEFI application, the elf_x86_64_efi.lds is an unmodified, verbatim copy of GNUefi's linker script. UEFI spec mandates that an UEFI PE must be a relocatable binary, therefore must be linked at zero address. It's not something you can change. (Binaries are relocated by UEFI in run-time.)

testjz wrote:Apart from reimplementing cryptographic functions, the best you could do would be to reuse the applicable files from a public domain project like LibTomCrypt or a permissively licensed one, possibly with some adaptations to remove defines and macros irrelevant to your project

Why on earth would I introduce a new dependency? I'll go with the former, thanks. There's only two crypto functions, SHA-512 and CRC-32c, both small enough to be included (UEFI has a third, AES). Besides, there's no library that would support both C and Assembly, therefore some platforms need their own implementation anyway.

testjz wrote:And then to periodically check for changes in the original code

Nope. SHA, CRC (and AES) functions won't change. They are not only standards, but strictly mathematical functions, not subject to change, never.

testjz wrote:I also saw some minor stuff, like assigning a value to "valid" at lines 1117 and 1131 and then not using the variable.

You did not read the code carefully, used at line 1137.

testjz wrote:putting multiple statements on the same line

Yeah, I know this is hard to understand if you haven't learned at my university, and haven't learned about parallel expressions in Fótism (TM). C doesn't have those constructs, so the best I can do to "emulate" that mathematical model is to put those statements in one line.

The most typical example: "free(a); a=NULL;". Freeing a memory and then clearing the pointer should be one statement, but it simply does not worth to put those into a separate function. Writing them on the same line makes sense because they belong together. Putting them into separate lines would suggest that they are separateable, while they're not. This is just like Hungarian notation. Non-hungarian programmers just don't get it how to use HN properly, that linked wiki page is full of bad examples (actually ALL examples are wrong, with one exception, maybe "cApples" could be good, however should be "cntApples", as "c" isn't readable and intuitive enough). My prof would have yell at you and would have given you a negative credit if that were a thing for a name like "u32Identifier". That's just not HN. I still haven't figured out how to explain to non-hungarians that data type and data representation are not the same. Using more statements on the same line is pretty similar.

However you're right that the coding style is not very consistent. That's because I've spent years writing it (my style changed a bit in the meantime), and often I wrote the code late at night after work (the only time I had for my hobby). Coding style is something I should fix.

testjz wrote:But I still don't see any reason to map unused memory because, as you say, "all kernels tend to set up their own paging tables anyway".

What's not clear about it? It's simpler to map the first 16G 1:1 with a simple loop than to create complex algorithms to parse memranges and map only what's needed. It's going to be thrown away and replaced by the kernel's paging table anyway, so why waste boot time on constructing complex mappings?

testjz wrote:I also find the mapping of the kernel to specific addresses a bit inflexible

They are not! Only for BOOTBOOT Protocol level 1 (BIOS version). For level 2 loaders (all the rest), the addresses are read from the kernel's symbol table. They must be within the range -1G..0 and must be page aligned, otherwise the kernel can place them anywhere they want them to be!

testjz wrote:Regarding the original issue of this topic, maybe you would "compromise" with turning the APs on after ExitBootServices()?

Ehem, that's already done... That's where the smp.S and CopyMem(0x8000, ...) comes into play

You can use a simple USE_MP_SERVICES define to switch between the two implementations. The APs are released from the spinlock (efficiently turning them on) after ExitBootServices().

testjz wrote:Also, would you trust the UEFI firmware more than your own code? (Rhetorical question)

Never

I usually trust my own code more than any other, but I don't really trust any code (not even mine) unless it's formally proven to be correct.
(The reason why I trust my own code better than others is not because I'm an egoist, but because I have more than three decades of experience, I've studied to be a programmer-mathematician, and I put lot of effort to make correct code. I usually spend 90% with thinking and figuring out possible scenarios how it could go wrong, and then in only 10% of the time I do the actual coding, and I always test. However I don't have as many test beds as I would like.)

Cheers,
bzt

Posted: **Tue Sep 22, 2020 7:01 am**

Another long post coming...

bzt wrote:
testjz wrote:Maybe you weren't reading very carefully (it can happen to everyone), but please notice the "write their own bootloader" part.
Yes, I was. I've tried to say, this is the reason why I have written my own loader instead of choosing a widely tested one. I'm convinced that the clean design, the flexibility of BOOTBOOT and the wide range of booting environments are beneficial to others too, that's why I split it from the OS project. If someone gets stuck writing their own boot loader, they can use it. Much much simpler than switching to GRUB (both managebility-wise and development-wise).

At the present, if I were writing an OS, I'd also write my own bootloader (I came to hate GRUB). However, I think that I'd wait some years (I don't know how many exactly) before I recommend BOOTBOOT for those who don't want to write their own, that would be the time I roughly estimate BOOTBOOT reaches better stability with greater probability. Of course, I don't see any problem with you recommending it yourself, as more users means obviously more feedback to you, thus better software by you. I never asked you "why did you write your own bootloader?", because I think that's the best choice. I think however that we should not be in a hurry to say that BOOTBOOT is mature enough (I'd classify it as beta), but I predict the clean design and flexibility will work out to be BOOTBOOT's main strength over GRUB and Multiboot (it already is).

bzt wrote:
testjz wrote:GRUB is used in millions of installations of (GNU+Linux) and other OSes.
Read: GRUB is overcomplicated to support all those, and hard to use (just count how many GRUB topics are on this forum, not to mention GRUB's own forum and on stackoverflow). GRUB's config file is almost a programming language on its own right!!! Using both grub-install and grub-mkrescue have hilariously huge number of dependencies. Using mkbootimg is totally dependency-free (save libc), yet knows everything that those other two

I completely agree with this, GRUB is both a bootloader and a boot manager (these two roles should be accomplished by distinct projects), and it tries to be universal in order to support everything, thus it's really too complicated. As for the config file, the dependencies and the other stuff, GRUB as a whole is kind of a disaster I'd say...

bzt wrote:Plus I don't care how many millions installations use GRUB, and how widely tested it is, if my target platform isn't one of them, it's no good. You can't install GRUB on a Raspberry Pi. GRUB doesn't support ARM64 officially at all (Multiboot spec is x86 and 32 bit only, and Multiboot2 likewise, talks only about uefi-amd64, like there were no other 64 bit platforms, and does not recognize the possibility of BIOS-amd64 at all).

My point is that, if there are millions of installations of some software, chances are it's better working (read: patched) than software with 100 of installations, irrespectively of design, code quality and similar factors. On the other things you said, I agree with you.

bzt wrote:
testjz wrote:Besides, this topic proves that you still had something to iron out. It's possible that another issue can surface at any time
I don't intend to say BOOTBOOT is perfect. You are welcome to open a new issue and I'll try to solve your problem the best I can. I only say that it already runs on all mainstream virtual machines and on numerous real hardware. I won't and I can't give a guarantee that it will run on all possible hardware configurations, but as being a FOSS project, you're welcome to fix it for your machine

Sure.

But I was thinking about the rate of stability which, while I think is subjective how good it must be in order to classify a project as mature enough, it usually still depends on some specific factors (in parentheses will be my estimates for BOOTBOOT) like "how big is the project?" (small-to-medium size), "how many bugs were found lately and how many of them were fixed?" (I don't know and I'm not going to count the bugfix commits right now), "how many might use it right now?" (100 to 200, as estimated above), "how much of the users' work is affected by the common uses (i.e. how likely are issues noticed)?" (all of their OS development work, unless they support multiple bootloaders and/or have multiple OS projects), and possibly others I can't think of now and, of course, the intuition of the developer(s) which "estimates" based on these factors.

bzt wrote:
testjz wrote:I don't think that software can enter a state where nothing can be improved
On that, we disagree. For three reasons:
1. I don't think that a code that works needs improvement. As long as it gets the job done, don't try to fix it.

But does it always work? Are all edge cases predicted and taken care of? Is it as simple and as correct as possible? Are all algorithms used there the most optimal ones depending on the situation? Is there guarantee that the dependencies will never change (the only one probably not applicable here)? Is there guarantee that there won't be any new processor architectures and hardware platforms and that they won't invent a new firmware that might be better or worse than UEFI? It's impossible anyone knows any of these before the answer becomes "No". But I'm not saying that nothing is mature, just that improvement is a continuous process and does never stop, unless the software is not used anymore. (Not talking specifically about any project here.)

bzt wrote:2. There's a strict "no new features" policy. The BOOTBOOT Protocol already contains the very minimum to boot a system, you can't take away anything, and there's no reason to add anything if the current feature set is enough to boot a system.

I think you hurried up this a bit, because probably it also prevents removing features that turn out to be unnecessary and/or modifying features that could be done in a better way. That said, I don't know how many years is BOOTBOOT under development (from what I see, at least two years and three months, and I don't know how much time it was before the initial commit and/or also how much time it was for the project this one evolved from). Maybe this is enough time for a feature freeze, it certainly depends on several factors, probably including those iterated above.

bzt wrote:You probably have heard, but I like this quote very much: "Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius—and a lot of courage to move in the opposite direction." /E. F. Schuemacher/

Except that "improvement" doesn't mean "bigger, more complex, and more violent" (certainly not the last one). I also have this other quote:

Ken Thompson wrote:One of my most productive days was throwing away 1000 lines of code.

which I think is more fitting.

bzt wrote:Yeah, the only reason for that variable is because UEFI does not allow an address to be passed to AllocatePages. It requires an address of a variable holding the address...

Alright, but I'd still use it since it's already defined (code maintainability reasons).

bzt wrote:
testjz wrote:I looked also at your linker file, and I don't understand why do you start from address 0x00000000000000000.
What do you mean? The kernel's linker file starts at 0xffffffffffe00000.
And for the UEFI application, the elf_x86_64_efi.lds is an unmodified, verbatim copy of GNUefi's linker script. UEFI spec mandates that an UEFI PE must be a relocatable binary, therefore must be linked at zero address. It's not something you can change. (Binaries are relocated by UEFI in run-time.)

I stand corrected. I indeed meant the UEFI application but, given I never did anything serious with UEFI, I had forgotten the fact it's relocatable.

bzt wrote:
testjz wrote:Apart from reimplementing cryptographic functions, the best you could do would be to reuse the applicable files from a public domain project like LibTomCrypt or a permissively licensed one, possibly with some adaptations to remove defines and macros irrelevant to your project
Why on earth would I introduce a new dependency? I'll go with the former, thanks. There's only two crypto functions, SHA-512 and CRC-32c, both small enough to be included (UEFI has a third, AES). Besides, there's no library that would support both C and Assembly, therefore some platforms need their own implementation anyway.

I obviously never talked about introducing a full dependency, just reusing the one or two files that are needed for the specific purpose. Why would the bootloader need to depend on an entire external hosted library that implements 100 or more functions?

Also, I said previously "I don't know how applicable the advice of not reimplementing security-related functions in your own code is here", implying that the specialized environment the bootloader runs is a special case and non-special advice doesn't always apply, and we can never all agree where to draw the line.

bzt wrote:Nope. SHA, CRC (and AES) functions won't change. They are not only standards, but strictly mathematical functions, not subject to change, never.

That should obviously be always the case, but there might still be edge cases that hadn't been predicted (e.g. some operations might be overflowing with some values), or the code is unportable to some architectures (e.g. unaligned memory accesses are used). I didn't care to check whether in this specific case of these cryptographic functions such or similar situations are practically possible though.

bzt wrote:
testjz wrote:I also saw some minor stuff, like assigning a value to "valid" at lines 1117 and 1131 and then not using the variable.
You did not read the core carefully, used at line 1137.

I stand corrected, and I shouldn't actually review code late in the evening (maybe if there was a space before that "valid", I would have noticed it, but I really should had used Ctrl+F). That said, maybe "valid" could only be defined and assigned to when "GOP_DEBUG" is defined, and this way it would be more correct in my opinion, but probably more complex. You obviously compare the trade-offs.

bzt wrote:
testjz wrote:putting multiple statements on the same line
Yeah, I know this is hard to understand if you haven't learned at my university, and haven't learned about parallel expressions in Fótism (TM). C doesn't have those constructs, so the best I can do to "emulate" that mathematical model is to put those statements in one line.

I think everyone should write code so others do understand it easily. Since this is C, most people will expect one of the styles usually used with C.

By the way, could you tell me more about Fótism? Unfortunately I searched for it using both Startpage and DuckDuckGo (I don't use Google) and I found only one search result with Startpage: this topic. DuckDuckGo didn't return any results with Fótism (yes, I put it in quotes so it's exactly this and not something similar-looking).

bzt wrote:However you're right that the coding style is not very consistent. That's because I've spent years writing it (my style changed a bit in the meantime), and often I wrote the code late at night after work (the only time I had for my hobby). Coding style is something I should fix.

Understandable.

bzt wrote:
testjz wrote:But I still don't see any reason to map unused memory because, as you say, "all kernels tend to set up their own paging tables anyway".
What's not clear about it? It's simpler to map the first 16G 1:1 with a simple loop than to create complex algorithms to parse memranges and map only what's needed. It's going to be thrown away and replaced by the kernel's paging table anyway, so why waste boot time on constructing complex mappings?

In the first post, I indeed talked about mapping the entire RAM. In the previous one, and I should had been more clear, sorry, I was talking about selecting just the kernel, the boot structures, the initial ramdisk and the framebuffer, and map only those. Either way you already map them specially. I wasn't talking anymore about identity mapping the entire RAM.

bzt wrote:
testjz wrote:I also find the mapping of the kernel to specific addresses a bit inflexible
They are not! Only for BOOTBOOT Protocol level 1 (BIOS version). For level 2 loaders (all the rest), the addresses are read from the kernel's symbol table. They must be within the range -1G..0 and must be page aligned, otherwise the kernel can place them anywhere they want them to be!

I see, though it's probably more complexity for the kernel as it needs to check which protocol level it has been loaded with and to use the appropriate addresses. By the way, do you plan making the BIOS version support level 2?

bzt wrote:
testjz wrote:Regarding the original issue of this topic, maybe you would "compromise" with turning the APs on after ExitBootServices()?
Ehem, that's already done... That's where the smp.S and CopyMem(0x8000, ...) comes into play You can use a simple USE_MP_SERVICES define to switch between the two implementations. The APs are released from the spinlock (efficiently turning them on) after ExitBootServices().

I don't know much about how UEFI handles SMP (whatever little I know I learned it in this topic), but is it guaranteed that UEFI won't try to do anything with the cores between the point you send the IPI and SIPIs (shouldn't the second SIPI be sent only in case it the first one didn't work?) and the point where you call ExitBootServices()?

bzt wrote:
testjz wrote:Also, would you trust the UEFI firmware more than your own code? (Rhetorical question)
Never I usually trust my own code more than any other, but I don't really trust any code (not even mine) unless it's formally proven to be correct.

That's good, though I already knew the first part of the answer.

But how do you formally prove your code to be correct? Which methods do you use and what medium (written logic and math, a computer program)? I'm genuinely interested because I also want code to be correct (and it amazes me that so many widely used projects have very low quality code).

Posted: **Tue Sep 22, 2020 10:38 am**

testjz wrote:I think that I'd wait some years (I don't know how many exactly) before I recommend BOOTBOOT for those who don't want to write their own, that would be the time I roughly estimate BOOTBOOT reaches better stability with greater probability. ... I think however that we should not be in a hurry to say that BOOTBOOT is mature enough (I'd classify it as beta)

Thanks, but I don't think I called it mature in a hurry. The feature set got frozen more than a year and a half ago. I'm regularly testing it, as well as many users, and just because you've asked, I've done a through testing with the following combinations:
bochs, BIOS, ROM
bochs, BIOS, disk
bochs, BIOS, cdrom
qemu, BIOS, ROM (-option-rom)
qemu, BIOS, Linux boot protocol (-kernel)
qemu, BIOS, disk
qemu, BIOS, cdrom
qemu, GRUB, Multiboot (via grub-mkrescue)
qemu, UEFI, disk (TianoCore)
qemu, UEFI, cdrom (via UEFI El Torito)
qemu, RPI3, sdcard
qemu, RPI3, serial
qemu, coreboot, ROM (using cbfstool to add initrd to ROM)
qemu, coreboot, disk
qemu, coreboot, USB disk*
VB, BIOS, disk
VB, UEFI, disk
real hardware, BIOS, USB disk
real hardware, BIOS, SATA disk
real hardware, UEFI, USB disk

* - this one didn't work, but the problem is not in BOOTBOOT, libpayload's usb_poll() function does not return for some reason. I guess I've compiled coreboot with incorrect config or I've messed up the qemu arguments to emulate an USB storage. Further investigation required.

testjz wrote:But I was thinking about the rate of stability which, while I think is subjective how good it must be in order to classify a project as mature enough, it usually still depends on some specific factors (in parentheses will be my estimates for BOOTBOOT) like "how big is the project?" (small-to-medium size),

I don't think so. I think testing matters more. However that's true that smaller projects are easier to be tested correct, that's why I have created several, separated loaders all implementing the same protocol instead of a single, bloated code base.

testjz wrote:"how many bugs were found lately and how many of them were fixed?" (I don't know and I'm not going to count the bugfix commits right now)

This tells absolutely nothing. A better designed and implemented software will get fewer bugs. All of my projects in general have very few bug reports, because I put lot of effort in them. Btw, BOOTBOOT right now has about 20 issues, all has been fixed and closed. As a comparition, one of my most popular projects with nearly 1000 stars (and probably 10000 users) has 60 issues, not all of them are bug reports (there are many feature requests), and only 2 are open ATM.

testjz wrote:"how many might use it right now?" (100 to 200, as estimated above), "how much of the users' work is affected by the common uses (i.e. how likely are issues noticed)?"

Now these are very good questions. It is true, that the more users are using it, the more the chances are that a bug pops onto the surface.

testjz wrote:But does it always work?

It doesn't have to always work, it is enough if it works for all the users

No one can guarantee that a software will always work unless it's mathematically proven correct, and even then there could be errors caused by faulty hardware.

testjz wrote:Are all edge cases predicted and taken care of?

I think so! SMP was the last feature to be added, and there was only one issue on real hardware in the last year.

testjz wrote:Is it as simple and as correct as possible?

Yes!

testjz wrote:Are all algorithms used there the most optimal ones depending on the situation?

No. They are not always the most optimal, but always a correct one! I don't care about optimization if the code loads the system in less than a second. There's simply no point in optimizing.

testjz wrote:Is there guarantee that the dependencies will never change (the only one probably not applicable here)?

Yes! If the dependencies (which is BIOS int, UEFI protocols, and hardwired SoC for RPi) change, they will break their standards, and it would affect all boot loaders, not just BOOTBOOT. (Take a look at the 2nd part of the PDF specification, it lists all firmware functions that's been used for a particular implementation)

testjz wrote:Is there guarantee that there won't be any new processor architectures and hardware platforms and that they won't invent a new firmware that might be better or worse than UEFI?

No, but who cares? BOOTBOOT is a protocol, and a new loader can be added (independently to the existing ones) any time that would implement those new hardware and firmware. The BOOTBOOT protocols requirements are so slim, that I'm absolutely sure they can be implemented. There's one exception, BOOTBOOT is 64 bit only, for 128 bit ISA there'll be a need for a new protocol, but that's fine and calculated.

testjz wrote:But I'm not saying that nothing is mature, just that improvement is a continuous process and does never stop

Wrong! You are thinking about one big bloated project! BOOTBOOT is a standard, with many little implementations.

testjz wrote:I think you hurried up this a bit, because probably it also prevents removing features that turn out to be unnecessary

You *think*. On the other hand, I *know* for sure, because I've put several decades of research in this topic.

testjz wrote:I don't know how many years is BOOTBOOT under development (from what I see, at least two years and three months, and I don't know how much time it was before the initial commit and/or also how much time it was for the project this one evolved from).

Decades. The first version of my loaders that was called BOOTBOOT is more than 3 years old, and the main concept hasn't changed a bit. That initial commit 2 years ago just marks the time when I moved the project from github to gitlab.

testjz wrote:I also have this other quote:
Ken Thompson wrote:One of my most productive days was throwing away 1000 lines of code.
which I think is more fitting.

I like this one! K.I.S.S. was always my favourite principle

testjz wrote:Why would the bootloader need to depend on an entire external hosted library that implements 100 or more functions?

Exactly!

testjz wrote:That should obviously be always the case, but there might still be edge cases that hadn't been predicted (e.g. some operations might be overflowing with some values)

Not possible, there are standards for crypto functions. If overflow happens, then that's an implementation problem, and it surely won't pass the test vectors.

testjz wrote:or the code is unportable to some architectures (e.g. unaligned memory accesses are used).

This is why I prefer a separate implementation of those on all platforms.

testjz wrote:That said, maybe "valid" could only be defined and assigned to when "GOP_DEBUG" is defined

I'm already ahead of you, this has been done!

testjz wrote:I think everyone should write code so others do understand it easily. Since this is C, most people will expect one of the styles usually used with C.

Yeah, using a consistent coding style is a high priority on the TO DO list, no question about it.

testjz wrote:By the way, could you tell me more about Fótism? Unfortunately I searched for it using both Startpage and DuckDuckGo (I don't use Google) and I found only one search result with Startpage: this topic. DuckDuckGo didn't return any results with Fótism (yes, I put it in quotes so it's exactly this and not something similar-looking).

Not surprising

This is a local slang at my university, also called the Zen-of-Fóthi (sorry, Hungarian only). Long story short, my prof, Dr. Ákos Fóthi created a programming methodology based on mathematics in the early 1980's at ELTE. I couldn't find that book on-line (I've a printed version), but I could find a link to another book, called "Prelude to Programming" (sorry, Hungarian only). Note this book is not about the methodology, rather it's a prerequisite that explains the background, a glossary if you'd like.

testjz wrote:I should had been more clear, sorry, I was talking about selecting just the kernel, the boot structures, the initial ramdisk and the framebuffer, and map only those. Either way you already map them specially. I wasn't talking anymore about identity mapping the entire RAM.

That's different. Those are mapped in KERNEL SPACE, that's totally different than identity mapping RAM in USER SPACE.

testjz wrote:I see, though it's probably more complexity for the kernel as it needs to check which protocol level it has been loaded with and to use the appropriate addresses.

No, it is the other way around! The kernel has to check nothing. Level 1 compliant loaders can load kernels that comply with BOOTBOOT Protocol Level 1, while Level 2 compliant loaders can boot both Level 1 and Level 2 kernels. To check which version a kernel complies to, use

Code: Select all

./mkbootimg check yourkernel.elf

testjz wrote:By the way, do you plan making the BIOS version support level 2?

I'm not sure. Maybe, but TBH I'm good with supporting Level 1 only, and at least there's an example Level 1 loader.

testjz wrote:is it guaranteed that UEFI won't try to do anything with the cores between the point you send the IPI and SIPIs (shouldn't the second SIPI be sent only in case it the first one didn't work?) and the point where you call ExitBootServices()?

Yes. UEFI runs on BSP, and ExitBootServices(0 must be called on the BSP.

testjz wrote:But how do you formally prove your code to be correct?

For example Fóthism

Here's a list of things required for a master's degree, called "Theory of Programming Methodology" (again, sorry, Hungarian only). This is a non-official paper written by one of PhD students.

testjz wrote:Which methods do you use and what medium (written logic and math, a computer program)?

None! It is extremely hard to create proof of correctness! What Fóthism teaches you is that you don't need proof. It is enough if you learn mathematically proven algorithms, then you do a thing called "backtrack" (however this is confusing, because there's a search algorithm also called backtrack, I'm not talking about that). The point of backtrack method is, that you use bidirectional relations to transfer the state-space of your problem until you can find a (proven) algorithm that fits and solves your original problem. Then you transfer back the result to the original state-space, and you're good. Hope this makes sense to you, there's an entire semester on this topic.

testjz wrote:I'm genuinely interested because I also want code to be correct (and it amazes me that so many widely used projects have very low quality code).

I would say the easiest way is practicing. Do lot of things. See your code fail deliberately. Fix your code. After a couple of decades, you'll feel it in your guts what solution is going to be good and which isn't. Using mathematics for that might help, but it is hard and it isn't the only way.

Cheers,
bzt

Posted: **Tue Sep 22, 2020 6:43 pm**

bzt wrote:The APs are released from the spinlock (efficiently turning them on) after ExitBootServices().

The firmware still owns the APs until after the call to ExitBootServices() returns. You can't send the IPIs to start the APs before calling ExitBootServices().

Implementations of this protocol may use the UEFI event EFI_EVENT_LEGACY_BOOT_GUID or EFI_EVENT_GROUP_EXIT_BOOT_SERVICES to force APs into the OS compatible state as defined by the UEFI Specification.

Posted: **Wed Sep 23, 2020 8:21 am**

Octocontrabass wrote:You can't send the IPIs to start the APs before calling ExitBootServices().

Actually you can. It works on all VMs and on all real hardware. Why? Because...

Implementations of this protocol may use the UEFI event EFI_EVENT_LEGACY_BOOT_GUID or EFI_EVENT_GROUP_EXIT_BOOT_SERVICES to force APs into the OS compatible state as defined by the UEFI Specification.

...is a complete bullshit in the spec, nobody cared to define what "OS compatible state" actually is. Neither the UEFI nor the UEFI PI spec contains that definition, therefore most firmware manufacturers don't give a sh*t about it (specially because it's a "may" and not a "must" nor "should"). Furthermore TianoCore (which serves as a base for real hw implementations) allows not only SIPI, but starting kernel from EFI_MP_SERVICES (although it shouldn't allow that latter, that's pretty clear from the UEFI PI spec, but it does).

This is yet another epic failure on UEFI designer's part. I'm not suprised. But I've moved SIPI after ExitBootServices, which is kinda sucks, because no UEFI Delay functions are available after ExitBootServices. Seriously, what a moron these UEFI guys had to be to make such huge mistakes? TianoCore eats up at least 3M of RAM without providing any useful services to the OS. It's just purely waste of RAM.

Cheers,
bzt

Posted: **Sun Sep 27, 2020 9:15 am**

bzt, By "mature", I didn't mean the range of supported platforms or the amount of testing. I meant that testing won't reveal many issues anymore because there isn't left much to improve (though there will always be something still left).

bzt wrote:
testjz wrote:"how many bugs were found lately and how many of them were fixed?" (I don't know and I'm not going to count the bugfix commits right now)
This tells absolutely nothing. A better designed and implemented software will get fewer bugs. All of my projects in general have very few bug reports, because I put lot of effort in them. Btw, BOOTBOOT right now has about 20 issues, all has been fixed and closed. As a comparition, one of my most popular projects with nearly 1000 stars (and probably 10000 users) has 60 issues, not all of them are bug reports (there are many feature requests), and only 2 are open ATM.

But you probably also discovered some of them yourself, and you didn't open them formally on the issue tracker (why would you open an issue just to mark it as fixed and close it?). You just did a commit when fixing an issue, or you maybe integrated the fix into another commit (but that's bad style as it's harder to review the commit history). I don't know how many issues you discover yourself, or how many issues were reported to you through somewhere else than the issue tracker, but the number of issues on the tracker (open or close) doesn't tell me much, thus I would need to count the commits.

But you said:

bzt wrote:A better designed and implemented software will get fewer bugs

which was precisely (part of) my point. But if there are 5 discovered bugs left unfixed for several years, then it's probably worse than if there were 8 discovered bugs, of which 6 fixed.

However, if BOOTBOOT really is more than 10 years under development in total (excluding breaks), then I was probably wrong when I assumed that you hurried up, though I see that the code could be at least cleaned up (at least the immediately obvious inconsistency in the coding style). I also think that I disagree with some aspects of the design, namely having multiple protocol levels and identity mapping (parts of) the RAM, though these ones probably can't be changed anymore as they are part of the specification and not the rest of the design or the implementation.

On another note, you seem a bit too overenthusiastic and a bit too overoptimistic about the software you write and about other things, including the way firmware developers might interpret the UEFI spec. It's good to have enthusiasm and to be an optimist, but too much is not that good either. We also have to be modest and, in the case of depending on the firmware developers, we should rather be prepared for the worst case(s) that do(es)n't break the specification.

bzt wrote:
testjz wrote:Is it as simple and as correct as possible?
Yes!

bzt wrote:
testjz wrote:That said, maybe "valid" could only be defined and assigned to when "GOP_DEBUG" is defined
I'm already ahead of you, this has been done!

Contradiction! So it wasn't as simple and as correct as possible previously, and it probably isn't now either, unless you don't make further commits because there isn't anything left anymore to fix!

These are the overoptimism and overenthusiasm I'm talking about!

bzt wrote:
testjz wrote:I should had been more clear, sorry, I was talking about selecting just the kernel, the boot structures, the initial ramdisk and the framebuffer, and map only those. Either way you already map them specially. I wasn't talking anymore about identity mapping the entire RAM.
That's different. Those are mapped in KERNEL SPACE, that's totally different than identity mapping RAM in USER SPACE.

I mean, it is different, and that's why it makes a difference of why the one is necessary and why I think the other is superfluous.

bzt wrote:
testjz wrote:is it guaranteed that UEFI won't try to do anything with the cores between the point you send the IPI and SIPIs (shouldn't the second SIPI be sent only in case it the first one didn't work?) and the point where you call ExitBootServices()?
Yes. UEFI runs on BSP, and ExitBootServices(0 must be called on the BSP.

I don't know anything about this topic (I can merely suspect something), so I yield it to the more knowledgeable of these forums.

bzt wrote:
testjz wrote:By the way, could you tell me more about Fótism? Unfortunately I searched for it using both Startpage and DuckDuckGo (I don't use Google) and I found only one search result with Startpage: this topic. DuckDuckGo didn't return any results with Fótism (yes, I put it in quotes so it's exactly this and not something similar-looking).
Not surprising This is a local slang at my university, also called the Zen-of-Fóthi (sorry, Hungarian only). Long story short, my prof, Dr. Ákos Fóthi created a programming methodology based on mathematics in the early 1980's at ELTE. I couldn't find that book on-line (I've a printed version), but I could find a link to another book, called "Prelude to Programming" (sorry, Hungarian only). Note this book is not about the methodology, rather it's a prerequisite that explains the background, a glossary if you'd like.

That's unfortunate, I'd be interested to understand what are the main concepts of this methodology.

bzt wrote:
testjz wrote:But how do you formally prove your code to be correct?
For example Fóthism Here's a list of things required for a master's degree, called "Theory of Programming Methodology" (again, sorry, Hungarian only). This is a non-official paper written by one of PhD students.

How does Fóthism achieve that though, if not to prove correctness as you say below, but to effectively ensure it?

bzt wrote:
testjz wrote:Which methods do you use and what medium (written logic and math, a computer program)?
None! It is extremely hard to create proof of correctness! What Fóthism teaches you is that you don't need proof. It is enough if you learn mathematically proven algorithms, then you do a thing called "backtrack" (however this is confusing, because there's a search algorithm also called backtrack, I'm not talking about that). The point of backtrack method is, that you use bidirectional relations to transfer the state-space of your problem until you can find a (proven) algorithm that fits and solves your original problem. Then you transfer back the result to the original state-space, and you're good. Hope this makes sense to you, there's an entire semester on this topic.

I don't know whether I understand it correctly, but from your description it looks like mathematical reduction or transformation, depending on the case. Am I correct?

bzt wrote:
testjz wrote:I'm genuinely interested because I also want code to be correct (and it amazes me that so many widely used projects have very low quality code).
I would say the easiest way is practicing. Do lot of things. See your code fail deliberately. Fix your code. After a couple of decades, you'll feel it in your guts what solution is going to be good and which isn't. Using mathematics for that might help, but it is hard and it isn't the only way.

I'm not sure I understand this one. Could you please elaborate? What kind of things should I purposely break and on what scale, say single statements, entire functions, or entire programs and how big ones?

But, as this branch of the discussion is offtopic in this topic (no pun intended), could we maybe start a new topic so we can talk more about Fóthism, software correctness and related concepts?

Posted: **Sun Sep 27, 2020 9:47 am**

I opened a new topic here:

viewtopic.php?f=13&t=37287

Posted: **Sun Sep 27, 2020 10:48 am**

testjz wrote:But if there are 5 discovered bugs left unfixed for several years, then it's probably worse than if there were 8 discovered bugs, of which 6 fixed.

Yes, the number of open bug issues is closer to the truth. BOOTBOOT actually has zero. (BTW: another very important measure is that how long a bug issue is open. It's much worse to have 2 issues being open for years, than to have 20 closed within days.)

testjz wrote:However, if BOOTBOOT really is more than 10 years under development in total

Well, I do loader development for more than 20 years. This particular project is about 4-5 years old (meaning I already had more than a decade of experience when I started it), and it was given the name "BOOTBOOT" and got its own repository 3 years ago. To be precise

testjz wrote:I disagree with some aspects of the design, namely having multiple protocol levels

The only difference between levels is that Level 1 uses compile time specified fixed addresses, while Level 2 has run-time parsed, variable addresses. Would you be more happy with the "static / dynamic addressing" terminology?

testjz wrote:and identity mapping (parts of) the RAM,

Not parts! Not plural, singular. Strictly one single part of RAM, starting from address zero. I'm just curious, what would you rather do?

testjz wrote:On another note, you seem a bit too overenthusiastic and a bit too overoptimistic about the software you write

That's not true. I can see why you might think of that, but you're mistaken. Look, I know exactly what I'm capable of, and all my statements are backed by facts and tests. Lots of tests. I've never said that BOOTBOOT is guaranteed to run on all computers, it isn't.

I used to be modest. But then a bunch of brainwashed Scientologists took advantage of that and tried to force their stupid lies on me. I've stopped being modest when it comes to things that are backed up by facts and tests. So should you and everybody else if we ever want to stop this fake-news age.

testjz wrote:Contradiction! So it wasn't as simple and as correct as possible previously

What are you talking about? It is a variable used by debug output, has absolutely nothing to do with the boot process, therefore it doesn't influence correctness either.

testjz wrote:I mean, it is different, and that's why it makes a difference of why the one is necessary and why I think the other is superfluous.

Please elaborate! Is the higher-half kernel mapping superfluous? Or is mapping the initrd in user space superfluous?

Neither! Both are needed, and there's a practical reason for the separation:
1. higher-half kernel space: is where the kernel is linked at, and which is going to be needed in every tasks' address space. You can't go without it.
2. identity mapping (including things like initrd and ACPI tables), not part of the kernel space because only needed during boot. There's no point in mapping the initrd in every single task's address space.

testjz wrote:I'd be interested to understand what are the main concepts of this methodology.

Then read that book I linked! It is especially about the main concepts. Try Google Translate.

testjz wrote:I don't know whether I understand it correctly, but from your description it looks like mathematical reduction or transformation, depending on the case. Am I correct?

Hmm, yes and no. It is a transformation, indeed, but not any. Reduction definitely does not play. There are strict restrictions what relations can be used. It must be bijective and surjective and inversable. This means, R = AxB, where every element of A can be assigned to one, and only one element of B, no elements left unassigned, and furthermore R-1 = BxA, and R-1(R(a)) = a. Failing in one of these restrictions, and you cannot guarantee that your backtrack is correct (it still could be, but you can't be sure).

testjz wrote:I'm not sure I understand this one. Could you please elaborate? What kind of things should I purposely break and on what scale, say single statements, entire functions, or entire programs and how big ones?

Everything! You need to gain experience how programs can break, how they behave on different inputs, the more situations you experience the better. Übung macht den Meister!

Cheers,
bzt

Posted: **Fri Oct 09, 2020 1:22 pm**

It took me days to respond but here we go...

bzt wrote:Yes, the number of open bug issues is closer to the truth. BOOTBOOT actually has zero.

Closer to the truth than what? You don't know how many bugs are still undiscovered! And I don't know either! That said, of course, those bugs haven't mattered yet and might never matter, but that's how all non-intentional bugs start.

testjz wrote:But if there are 5 discovered bugs left unfixed for several years, then it's probably worse than if there were 8 discovered bugs, of which 6 fixed.

bzt wrote:(BTW: another very important measure is that how long a bug issue is open. It's much worse to have 2 issues being open for years, than to have 20 closed within days.)

Aren't we actually saying the same thing here? Or are you just confirming what I said?

Here is a tip: Usually, when I want to confirm something, I prefix the sentence with "indeed" or a synonym to signify that I agree with the person I reply to. If I don't, it's an oversight from my part.

bzt wrote:Would you be more happy with the "static / dynamic addressing" terminology?

I don't think it actually matters, though it would be a little more precise. The existence of multiple levels usually implies there are several sets of features supported or not supported depending on the level. In this case, you say there are only static and dynamic addresses.

bzt wrote:
testjz wrote:and identity mapping (parts of) the RAM,
Not parts! Not plural, singular. Strictly one single part of RAM, starting from address zero. I'm just curious, what would you rather do?

I would map just the initrd there and whatever else is needed (which depends on whom you ask). For example, I think the ACPI tables can be left unmapped at this point and mapped later permanently by the kernel, but this doesn't obviously apply in case the kernel uses ACPI before memory management is enabled and, of course, the bootloader cannot know that. That said, most kernels setup some initial mappings themselves anyway, depending on what they need. There is, in my opinion, no reason to map everything from address zero up to 4G or 16G. But we can just disagree and it will be still fine.

bzt wrote:What are you talking about? It is a variable used by debug output, has absolutely nothing to do with the boot process, therefore it doesn't influence correctness either.

I stand corrected. It was a bad choice of words from my part. I should had said (code) quality and not correctness, though simplicity still applies I think. I agree with what you said in the child topic of this one, that quality is not the same as correctness.

bzt wrote:I used to be modest. But then a bunch of brainwashed Scientologists took advantage of that and tried to force their stupid lies on me. I've stopped being modest when it comes to things that are backed up by facts and tests. So should you and everybody else if we ever want to stop this fake-news age.

I have heard about the tactics of Scientologists (they have even tried to censor Wikipedia), so it must have been quite traumatic for you. I thus won't try to talk about it.

But, modesty is different than self-defence against those that want to exploit you. Let me make an analogy with software. Simple software is modest in some sense, it doesn't pretend it has thousands of features. But simple software can also be more difficult to exploit (i.e. more secure), more so than complex software. Now with people. Someone who is not modest and pursuits to be a multi-millionaire (presumably for self pleasure) can of course exploit others without regrets in order to advance their own goal, granted, but is also themselves more easily exploited when accepting deals that wouldn't be accepted if not for their greed and might also need several lawyers to defend their own case. Thus, modesty and self-defence show at most just a little correlation, if any (or even negative correlation).

bzt wrote:Try Google Translate.

I try to avoid Google as much as possible, for various reasons. Furthermore, their translator isn't perfect, and often you can't even deduct what it should be saying. Of course, it might be not the case with Hungarian to English, but I haven't tried it for the other reasons.

For individual words or phrases, I can look them up in the Wiktionary, which not only does not track visitors in thousands of ways (at most server logs are kept and some very rudimentary statistics are collected), but it usually has all of the most common meanings of a word, several of those along with related words and the respective translations to multiple languages. I can look up the derivation of a word if I'm curious about it and/or can't figure it out, that is, trace the word to Greek, Polish, or Latin (which is the 80% of English and the 90% of French). Often, even the complete declination or conjugation is present. In other words, the Wiktionary is a quite complete dictionary and grammar, but not an automatic translator. That is, it's completely useless for translating texts longer than maybe a paragraph.

But I opened the book in Hungarian and, interestingly, understood a little more than I predicted (traced the derivation of some words to Greek or Latin, Polish doesn't apply). But the maths can still be useful, I'd just need to follow the concepts right from the start. I mean, I know sets and logic, but there are in the book specific forms that I don't really understand and I need to also practice them after understanding them.

bzt wrote:Everything! You need to gain experience how programs can break, how they behave on different inputs, the more situations you experience the better.

Yes, that seems like good advice. Thank you!

Indeed (as you also said in the child topic of this one), a program must be correct for all valid input for it to be complete. So, for example, if I have a program that's defined for a range of values, I need to test how the program behaves for a good sample of these values. Furthermore, in case the program itself is responsible for checking whether the input is valid, I also need test whether the program correctly errors out. It just makes sense.

OSDev.org

UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question

Re: UEFI question