OSDev.org

Posted: **Mon Jul 13, 2020 1:37 am**

zaval wrote:just for the note, despite bzt scares you, that you are required to mark a FAT partition as ESP and use only GPT

Using GPT is not a requirement for EFI, but you would be a fool not to use it anyway. GPT is mandatory for BOOTBOOT. There's a good reason why I made the legacy implementation of BOOTBOOT and the Raspberry Pi version GPT-aware. I don't now why are you afraid of GPT. All mainstream OS supports it, what's more, some demand it (using GPT is the default for recent Windows versions, as well as in Linux, and in MacOS, supported by BSDs and Solaris et cetera, et cetera, et cetera).

I'm glad that finally there's only one partitioning scheme that all OS supports, so why shouldn't your OS support it too?

zaval wrote:you can create MBR with an ordinary FAT partition and be sure

Nope, you must mark that FAT partition in MBR as 0xEF. The UEFI spec is pretty clear about this one. Read sections 5.2.1 Legacy Master Boot Record (MBR) and 12.3.2 Partition Discovery. If your firmware recognizes ESP without a valid 0xEF type, then good for you, but most EFI machines out there won't (and according to the spec they shouldn't). And if you have a GPT, then MBR shouldn't be used at all.

zaval wrote:your OSL may access your OS boot volume (where your OS resides)

Storing an entire OS on the ESP boot volume is terrible idea. Boot volumes are (as the name suggests) to store files required for booting. I would recommend having an ESP boot volume with the loader and an initrd, and a system partition with the rest of the OS files using your OS's native filesystem (that's what all mainstream OS does btw).

zaval wrote:(because, see, bzt, FAT is a required part by the standard, for easening interoperability and osdevers lives and not for what you fantasize)

I have absolutely no idea who you are talking to. My mkbootimg generates a FAT16/FAT32 file system for the ESP on the fly.

zaval wrote:on this stubborn placement in \efi\boot\bootx64.efi. this is not a normal way

No, it is the simplest and easiest way, no NVRAM required. By default, qemu does not use an NVRAM storage, you have to add extra parameters on command line to enable it. Using the default OS loader name saves you from that.

zaval wrote:it's more to bzt with his bootboot. the first time user directs the Boot Manager to load your bootboot, then, at the start, bootboot checks how it was started (by analyzing BootCurrent variable) and if it's not from its own Load Option, which is the case for the first start, asks a user if they want to install bootboot to this machine.

Never gonna happen, the philosophy of BOOTBOOT is to boot the OS without prompting the user for anything. Besides, you can run BOOTBOOT interactively from an EFI shell without making any changes to your system. Not to mention if it would rely on NVRAM then you couldn't boot it on machines without one. Right now you can use BOOTBOOT on the simplest qemu configuration too, and I want to keep it that way.

What you describe is a job for an OS installer or a boot manager, not for a boot loader.

Cheers,
bzt

Posted: **Mon Jul 13, 2020 2:53 pm**

bzt wrote: Using GPT is not a requirement for EFI, but you would be a fool not to use it anyway.
GPT is mandatory for BOOTBOOT. There's a good reason why I made the legacy implementation
of BOOTBOOT and the Raspberry Pi version GPT-aware. I don't now why are you afraid of GPT.
All mainstream OS supports it, what's more, some demand it (using GPT is the default for
recent Windows versions, as well as in Linux, and in MacOS, supported by BSDs and Solaris
et cetera, et cetera, et cetera).

I'm glad that finally there's only one partitioning scheme that all OS supports, so why
shouldn't your OS support it too?

I said nothing against using GPT, moreover not expressed any fear about using it, what I said,
that you still perfectly can use the MBR scheme if there is reason (you denied it as a possibility,
what's not true). and there might be such a reason, for example, as mentioned a million times over here, it's the only standard option for SD cards - its standard not only requires FAT as a FS, but also - MBR as a partitioning scheme. Also, for a beginner, it could be easier to start with. see, the OP already has some troubles with partitions. starting without ESP may be easier.

bzt wrote:
zaval wrote: you can create MBR with an ordinary FAT partition and be sure
Nope, you must mark that FAT partition in MBR as 0xEF. The UEFI spec is pretty clear about
this one. Read sections 5.2.1 Legacy Master Boot Record (MBR) and 12.3.2 Partition Discovery.
If your firmware recognizes ESP without a valid 0xEF type, then good for you, but most EFI
machines out there won't (and according to the spec they shouldn't). And if you have a GPT,
then MBR shouldn't be used at all.

No, you shouldn't. Again, for especially gifted - UEFI, by the specification must and does
recognize any FAT volume attached to the system. "recognize" means - lets you do anything
with this volume your OSL might need - read, write etc. it was exactly the point of my suggestion
to the OP - not use ESP at the beginning, to avoid complications, since different tools
noticing an ESP, even on "virtual" disks, may start to behave whimsy - not something you
want to deal with at the beginning of your OS creation. an ordinary FAT volume for such
cases is more than optimal. try it by yourself instead of needlessly arguing - it works as it should.

bzt wrote:
zaval wrote: your OSL may access your OS boot volume (where your OS resides)
Storing an entire OS on the ESP boot volume is terrible idea. Boot volumes are (as the name suggests)
to store files required for booting. I would recommend having an ESP boot volume with the loader and
an initrd, and a system partition with the rest of the OS files using your OS's native filesystem
(that's what all mainstream OS does btw).

this is all cool, but Boot Volume != System Partition. I specifically clarified in parenthesis,
that Boot Volume in a FW traditional nomenclature is where your OS resides, OS installation target
in other words.
whatever you imply by "initrd", this unixy thing shouldn't reside in the ESP too, with all the honesty,
but this is for normal OSs, that don't tend to put everything in a monolithic piece of it.

bzt wrote:
zaval wrote: it's more to bzt with his bootboot. the first time user directs the Boot Manager to load your bootboot,
then, at the start, bootboot checks how it was started (by analyzing BootCurrent variable) and if it's
not from its own Load Option, which is the case for the first start, asks a user if they want to install
bootboot to this machine.
Never gonna happen, the philosophy of BOOTBOOT is to boot the OS without
prompting the user for anything. Besides, you can run BOOTBOOT interactively from an EFI shell without making
any changes to your system. Not to mention if it would rely on NVRAM then you couldn't boot it on machines
without one. Right now you can use BOOTBOOT on the simplest qemu configuration too, and I want to keep it that way.

What you describe is a job for an OS installer or a boot manager, not for a boot loader.

whatever. you are free to make it as lame as you wish.

your "philosophy" avoids
prompting the user for some incomprehensible reasons, but puts stupid limitations on the
same user. woohoo!

what you called NVRAM, is a required part of any UEFI implementation and qemu btw, is not
an exception here. I guess, a normal way of working with an emulated environment, would be
doing as close to the real HW as possible, instead of following some crippled configurations,
some dude cluelessly decided are "better" for the user. it's your choice.

what I described is a normal behavior for a UEFI OSL, that is trying to play good, taking into
account it may run in a multi OS machine and having a FW, that provides it a mechanism for reliably
register itself on the system for the user benefit. OS installer may do this - for the OSL. again,
it doesn't change the fact, that it is the normal way, not that lame \efi\boot\bootx64.efi one.

bzt wrote:
zaval wrote: on this stubborn placement in \efi\boot\bootx64.efi. this is not a normal way
No, it is the simplest and easiest way, no NVRAM required. By default, qemu does not use an NVRAM
storage, you have to add extra parameters on command line to enable it. Using the default OS loader
name saves you from that.

it's the lamest way for sure. another lame loader, that goes to the same disk, will wipe your bootboot
into oblivion. very "simple". for whom? not for the user. again - NVRAM varibales of UEFI are a requirement.
qemu has no defaults what you are talking about - just supply it with a proper OVMF and options and you are well.
it's not qemu business at all, it's OVMF actually and what you supply. if you provide a crippled set, then yeah,
there is no NVRAM. but the same way, you may not provide even the FW at all! what your bootboot will do then?

this is not for a plain arguing. look. this "default", actually - "last resort" option is heavily overused, so
you easily expose your potential users to needless troubles - literally every lame thing overwrites this "resort"
paths. but why you, as a OSL provider do not use UEFI mechanisms? it's plainly unwise. if you want that this is done
by your user OS installer, good, tell this in your documentation and provide the appropriate help.

let's consider this much saner scenario. you make your bootboot. the user creates their .iso for example, and
puts there bootboot, say in \efi\bzt\bootboot. also, they create OS installer, that will create a Load Option
for it. at the first run, their users, run your loader from the Boot Manager "load from file" option, then it starts
their OS. then their users, when doing installation, run their installer, and that, calls UEFI RS SetVariable(),
creating a Load Option for bootboot, creates ESP if needed and copies it over there. this way, your bootboot will
be seen in the Boot Manager menu as an option, it wouldn't get accidently wiped out by some grub or whatever lamery.
but if you think this is "meh" and abusing that "last chance path" is cool, then I have nothing to add here.

Posted: **Tue Jul 14, 2020 5:31 am**

Dear zaval,

It is hard not to notice that your suggestions would cripple BOOTBOOT. So no.

zaval wrote:you still perfectly can use the MBR scheme

No you can't. this topic is about BOOTBOOT, and BOOTBOOT uses (and forward-supports) GPT. Period.

Besides, your beloved Windows can't boot using MBR tables on UEFI machines either. From Microsoft docs: "Windows cannot be installed to this disk. The selected disk is not of the GPT partition style".

zaval wrote:Also, for a beginner, it could be easier to start with. see, the OP already has some troubles with partitions.

That's a lie, I'm the Original Poster and I have no troubles with partitions. Creating a GPT is not harder than creating an MBR table, see our wiki with fdisk command examples. If someone has problems using fdisk, then they would equally have problems creating MBR tables too.

zaval wrote: starting without ESP may be easier.

ESP is required by the spec. Tell me, how do you boot under UEFI without ESP?

zaval wrote:UEFI, by the specification must and does recognize any FAT volume attached to the system.

No, the UEFI spec says that firmware must recognize volumes only in the GPT if there's a GPT. Read the spec "Partition Discovery", MBR partitions should not be and are not recognized when there's a GPT.

zaval wrote:your "philosophy" avoids prompting the user for some incomprehensible reasons, but puts stupid limitations on the
same user. woohoo!

Actually your suggestion would impose a stupid limitation, not the current implementation. It would be a nightmare to use your suggestion on dual-boot machines and on embedded systems without user interface.

Whether you can comprehend or not, the thing is, the BOOTBOOT loader is not allowed to make any modifications on the system. And it is not allowed to prompt the user, because it must work in a remote / embedded environment, when there's no user at all to prompt. Deal with it!

zaval wrote:taking into account it may run in a multi OS machine

Exactly. It is easy to see there would be a mess if every OS loader would set itself as default in a multi OS environment.

zaval wrote:goes to the same disk, will wipe your bootboot into oblivion. very "simple". for whom?

You are obviously just trolling. Nobody ever said, you can't rename the loader to FS0:/BOOTBOOT/LOADER.EFI and use LoadFile from the boot manager IF you have a boot manager. But if you don't have a configured boot manager, then you MUST use the default name, defined by the UEFI spec. Pointless to argue about this.

zaval wrote:if you want that this is done by your user OS installer, good, tell this in your documentation

I did. You just haven't read the README. Quote: "...or adding bootboot.efi to UEFI Boot Manager's menu for example."

You're also misleading forum readers and you're trolling. Just because you call it lame without understanding the reasons behind certain choices, actually makes YOU lame. If you get money to deliberately undermine this community then you're not worthy of your wage.

Cheers,
bzt

Posted: **Tue Jul 14, 2020 4:51 pm**

bzt, like I didn't know you are hopeless. ignorant and proud. did you really think I expect you to listen to my suggestions? or even react on them? I just wrote my opinion on the matter for other readers to face another point of view for not thinking your wrong understanding of the specification is really something good. needless to say, I am not gonna point you in your numerous wrong claims again. it's proven many times on this forum by many different people - you are a troublesome person, not being able to listen.

Or should I? for helping newcomers not to trap into your twisted visions.

That's a lie, I'm the Original Poster and I have no troubles with partitions.
Creating a GPT is not harder than creating an MBR table, see our wiki with fdisk
command examples. If someone has problems using fdisk, then they would equally
have problems creating MBR tables too.

you caught me! the OP is you and you surely don't have troubles messing up with your bootboot.
but I meant the guy, asking questions about his troubles getting your creature to work if you didn't get.

harder may be that fact, that modifying ESP in some circumstances is harder than an ordinary FAT volume. what I said.
GPT may be unavailable for some (virtual) disk sizes. SD cards cannot be formatted with it, without breaking their standard.

ESP is required by the spec. Tell me, how do you boot under UEFI without ESP?

should I make a video for you how I am doing this or draw pictures? you don't get
words? TRY IT BY YOURSELF! create an .vhd with VBox, make there an ordinary FAT volume,
put bootboot there and run it from the UEFI shell, not mounting to your VM any other storage. the machine is without ESP, still perfectly
deals with the FAT volume and lets you run your efi application. both from shell, Boot Manager
and even lets you create a Load Option.

No, the UEFI spec says that firmware must recognize volumes only in the GPT if there's a GPT.
Read the spec "Partition Discovery", MBR partitions should not be and are not recognized when there's a GPT.

wrong! it's such a BS, that I even don't know what to say against. so you say if you attach two disks,
one with GPT and other with MBR, UEFI won't recognize the latter? suure. tell me, when you read this,
did you smoke something, that is prohibited in yet a significant number of countries?

you misunderstood that section. it shows the order in which every storage is scanned. it's first
is checked for GPT, then el torito and then for legacy MBR partitions. you had to read a little farther
the beginning and notice this:

The following is the order in which a block device must be scanned to determine if it contains partitions. When a check for a valid partitioning scheme succeeds, the search terminates.
1. Check for GUID Partition Table Headers.
2. Follow ISO-9660 specification to search for ISO-9660 volume structures on the magic LBA.
3. Check for an “El Torito” volume extension and follow the “El Torito” CD-ROM specification.
4. If none of the above, check LBA 0 for a legacy MBR partition table.
5. No partition found on device.

that doesn't of course mean, that having somewhere GPT, you won't be able to use MBR in another place at the same machine.

and here is the quote, where it's clearly said about all FAT volumes "automatically handled" by the FW, not just
ESP ones. it's in "Simple File System Protocol" section, nearby "Partition discovery". I repeated this several times,
and I suspect you will argue again, I won't respond - wanna see, just go and try it!

UEFI Spec. Simple File System Protocol wrote: The firmware automatically creates handles for any block device that supports the following file system formats:
• FAT12
• FAT16
• FAT32

Actually your suggestion would impose a stupid limitation, not the current implementation.
It would be a nightmare to use your suggestion on dual-boot machines and on embedded systems
without user interface.

Whether you can comprehend or not, the thing is, the BOOTBOOT loader is not allowed to make
any modifications on the system. And it is not allowed to prompt the user, because it must
work in a remote / embedded environment, when there's no user at all to prompt. Deal with it!
...
Exactly. It is easy to see there would be a mess if every OS loader would set itself as default in a multi OS environment.
...
You are obviously just trolling. Nobody ever said, you can't rename the loader to FS0:/BOOTBOOT/LOADER.EFI and use
LoadFile from the boot manager IF you have a boot manager. But if you don't have a configured boot manager, then
you MUST use the default name, defined by the UEFI spec. Pointless to argue about this.

you are so comprehensive, yet fail to understand such an elementary thing - no matter interactive the system is or not,
if it's gonna be multi OS, or even not, the user chooses what the default would be. by setting the freaking boot order while
setting the machine up! not by the lame demanding to put the loader into \efi\boot\bootx64.efi
btw, if the same user will install a normal OS after your bootboot, the latter never will be run. because
the way you chose to put it with your "philosophy", it's not registered with the system, so it's absent in Load Options.
and no matter what the type of the system is, every loader installed, is supposed to create a Load Option for itself.
that's basically the main point. what you do is trickery to avoid that because you think on embedded systems it
would be cool.

no, it wouldn't!

what your bootboot is not allowed is not a proof of the truth, it's a proof of poor design desicions,
not understanding the specification, not wanting to listen to criticism and not understanding even the
fact, that your embedded system still must be interacted with at least at the set up time. so this "not allowed"
is just a nonsense, not a virtue.

"configured boot manager" wtf does it mean? you either have a BootXXXX variable for your loader, as it's supposed to be or,
for lamers, demand your poor users using \efi\boot\bootx64.efi

with the OS installer, one just sets up BootXXXX variable, the user still can go and change the order in which to arrange OSs for
booting. non-interactive systems still need to be set up and their embedded nature, doesn't prevent from using the normal way
of installing the UEFI OSL. \efi\boot\bootXXX.efi was created as a last resort for just one shot run - you put the installer UEFI
application on the USB thumb drive, run it and wipe out after it's done. using it on non removable storages is LAME AF. it was put into the
specification for the sake of completeness - for the Boot Manager to let the last chance to run something. and that "something", won't be run if there is at least one Load Option present at the system and your users won't be able to change that, because your "something" is not in Load Options and EVEN if they go and create such a Load Option manually for you, pointing to \efi\boot\bootx64.efi, where your philosophy dictated to put it, it will be f&cked up instantly, once next lamery that demands putting there arrives. cheers.

Posted: **Wed Jul 15, 2020 5:13 am**

For other forum members, please don't mind zaval, he's clearly just trolling.

Dear zaval,

Your arguments against GPT and "special" boot volumes is entirely off-topic, and your dual-boot related argument is just rubbish.

zaval wrote:no matter interactive the system is or not, if it's gonna be multi OS, or even not, the user chooses what the default would be. by setting the freaking boot order

Exactly. It is the user who chooses the boot order in the boot manager. BOOTBOOT has been tested on dual-boot environments, and it is working perfectly. Should I implement your suggestion to set the default boot order in the loader, and all would go havoc.

Besides, on embedded devices you can put BOOTBOOT in a ROM, what kind of filename do you suggest to put in the boot order in that case? Or are suggesting to cripple BOOTBOOT not to be ROM-compatible?

zaval wrote:not by the lame demanding to put the loader into \efi\boot\bootx64.efi

Yes, a sad troll, no doubt. I've already told you, that path is not demanded by BOOTBOOT, it is required by the UEFI spec when there's no configured boot order. Rename the loader, and use your boot manager to add that to the list, it will work perfectly. The documentation suggests to use the name FS0:\BOOTBOOT\LOADER.EFI.

zaval wrote:with the OS installer, one just sets up BootXXXX variable, the user still can go and change the order

Exactly. Why do you want to take that away from the users? A few sentence earlier you were saying "the user chooses what the default would be".

I'm going to discard your further posts.

Cheers,
bzt

Posted: **Sat Jul 25, 2020 3:30 pm**

Hi bzt,

I decided to tear apart one of your images, the x86 image from https://gitlab.com/bztsrc/bootboot/tree/master/images. I have to admit, I was surprised at the image and was disappointed in myself for not thinking of this as well. :-)

My early conclusions is that you have an image that can be booted on a hard drive "type" device as well as a CDROM both on Legacy BIOS and UEFI all with one image.

LBA 0: Legacy MBR for hard drive "type" devices
LBA 1: UEFI GPT (with two entries)
LBA 16: ISO 9660 for CDROM's (LBA 16 = 512-byte LBA 64)
LBA 128: bootable partition

Legacy:
- The MBR at LBA 0 has two of the four entries used, the first as a legacy BIOS bootable entry pointing to LBA 128.
- The second entry is the 0xEE entry. (See note 2 below)

UEFI:
- The GPT has two entries:
- - the first points to LBA 128 (FAT16 partition)
- - the second is an empty entry

ISO 9660:
- LBA 16 has a Primary Volume Descriptor for ISO 9660 CD ROM's
- I didn't check, but the Boot Descriptor probably points to LBA 128--in 512-byte sectors--LBA 32 in CDROM sized sectors.

This way, this single image can be written to a hard drive, USB thumb drive, or CDROM, and be booted by a Legacy or a UEFI machine with no modifications.

Well done. Again, I am disappointed in myself for not thinking of this as well. :-) Though I am going to have to "steal" this idea and create one myself...

Anyway, I only have a few questions/comments.

The first entry in the MBR (LBA 0) has a Starting CHS Sector value of 0x81 and an Ending CHS Sector value of 0x80. This must be an error, yes? Sector values cannot be more than 63. Do you need to check your LBAtoCHS code or am I missing something?
Why have the 0xEE (EFI GPT) entry as the second entry? I would think that you need to swap the first and second entries. Have the 0xEE entry as the first and the Legacy FAT16 entry as the second. Any (valid) BIOS will see the first entry as a non-bootable entry and move to the next entry to see if it is bootable. This would more closely follow the EFI GPT specs, yes?
The 0xEE entry should have a Size field value of 0xFFFFFFFF or at least the size of all of the media containing partitions within the GPT.
Taking note 3 above into consideration, your MBR shows that it encompasses only the first 0x8080 sectors of the disk. Entry 0 having 0x8000 sectors from sector 0x80 and the second entry only having 63 sectors from sector 1 (The culprit). Therefore, if a non-GPT aware utility was to try to partition the media, it "legally" could start partitioning from LBA 0x8080, which will destroy anything on the second GPT entry's partition, if anyone ever marked it as used. (Currently, the case is moot since nothing is currently allocated past LBA 0x8080)

I also tried to load the image with my tool (http://www.fysnet.net/ultimate/index.htm) and it crashed. I will take a look.

At the same time, I need to place a selection value somewhere. Currently, simply because I check for ISO first, my utility will only see that it is a CDROM image, even though it has a valid MBR *and* a valid GPT. I will have to create something to allow the user to select which to parse...

Thanks. It has been (will be) an interesting little project today.

Ben

Edit:
I am still investigating, but I think your Root ISO Directory is corrupt. You have a correct '.' entry, but an incorrect '..' entry.
In your FAT partition, your FAT directories are also corrupt. For example, your '.' entries are:
'.\0\32\32\32\32\32\32' '\32\32\32'
In other words, you are null terminating the dot. This is not a valid entry. The dot and dotdot entries should be as follows:
'.\32\32\32\32\32\32\32' '\32\32\32'
'.\.\32\32\32\32\32\32' '\32\32\32'

Please verify.

Thanks,
Ben

Posted: **Sat Jul 25, 2020 4:29 pm**

bzt wrote:For other forum members, please don't mind zaval, he's clearly just trolling.

Dear zaval,

Your arguments against GPT and "special" boot volumes is entirely off-topic, and your dual-boot related argument is just rubbish.

But you feed the troll. Why?

Greetings
Peter

Posted: **Sun Jul 26, 2020 5:47 am**

Hi,

@PeterX: I did, but not any more.

BenLunt wrote:Hi bzt,

I decided to tear apart one of your images, the x86 image from https://gitlab.com/bztsrc/bootboot/tree/master/images. I have to admit, I was surprised at the image and was disappointed in myself for not thinking of this as well.

Coming from you means a lot! Thanks! And don't feel disappointed, I've many decades of experience and K.I.S.S. is my superpower

BenLunt wrote:My early conclusions is that you have an image that can be booted on a hard drive "type" device as well as a CDROM both on Legacy BIOS and UEFI all with one image.
...
This way, this single image can be written to a hard drive, USB thumb drive, or CDROM, and be booted by a Legacy or a UEFI machine with no modifications.

Correct! Plus if you configure for AArch64 too, then you can also boot the same image on a RaspPi3/4. (I've created separate images, but one could create a multiplatform one just by using an array in the JSON instead of a string for initrd directory.)

BenLunt wrote:Well done. Again, I am disappointed in myself for not thinking of this as well. Though I am going to have to "steal" this idea and create one myself...

Please do so! I believe great ideas do not belong to anybody, they belong to all of us. That's why I've used MIT license so that you can study the code and learn from it.

BenLunt wrote:Anyway, I only have a few questions/comments.
The first entry in the MBR (LBA 0) has a Starting CHS Sector value of 0x81 and an Ending CHS Sector value of 0x80. This must be an error, yes? Sector values cannot be more than 63. Do you need to check your LBAtoCHS code or am I missing something?

The MBR partition for the system partition uses type 0xC or 0xE, both being LBA. The type 0xEE is by definition LBA-only, so I didn't bother too much to calculate correct CHS values. They are just garbage since nobody uses them any more (not the partitioning tools, neither UEFI firmware, and most definitely not my BOOTBOOT loader). Using zero CHS on the other hand did not pass some verifications, so I just put some random values there.

BenLunt wrote:[*]Why have the 0xEE (EFI GPT) entry as the second entry? I would think that you need to swap the first and second entries. Have the 0xEE entry as the first and the Legacy FAT16 entry as the second. Any (valid) BIOS will see the first entry as a non-bootable entry and move to the next entry to see if it is bootable. This would more closely follow the EFI GPT specs, yes?

Actually the ordering doesn't matter for BIOS and neither for UEFI. But for RaspPi it does, that loads firmware files only from them first entry (more precisely it's set up in the PWM, but the first entry is the default and Raspberry OS never changes that).

BenLunt wrote:[*]The 0xEE entry should have a Size field value of 0xFFFFFFFF or at least the size of all of the media containing partitions within the GPT.
[*]Taking note 3 above into consideration, your MBR shows that it encompasses only the first 0x8080 sectors of the disk. Entry 0 having 0x8000 sectors from sector 0x80 and the second entry only having 63 sectors from sector 1 (The culprit). Therefore, if a non-GPT aware utility was to try to partition the media, it "legally" could start partitioning from LBA 0x8080, which will destroy anything on the second GPT entry's partition, if anyone ever marked it as used. (Currently, the case is moot since nothing is currently allocated past LBA 0x8080)[/list]

Yeah, the whole 0xEE entry is a bad idea and not specified well. It is not clear if it should cover the entire GPT disk or just the GPT describing the entire disk. After many trial-and-error itrations, I've finally settled at a value that passes gdisk's verification successfully. If you think about it, having an MBR entry that covers the entire GPT disk is a nonsense, because one of the reasons GPT was introduced is to cover larger disks that MBR can't cover...

My advice is to use GPT partitioning tool only, and forget about MBR.

BenLunt wrote:I also tried to load the image with my tool (http://www.fysnet.net/ultimate/index.htm) and it crashed. I will take a look.

I've tested my images using fdisk (it switches automatically to GPT-mode), gdisk (which is a GPT-only) and MacOSX's diskutil.

BenLunt wrote:At the same time, I need to place a selection value somewhere. Currently, simply because I check for ISO first, my utility will only see that it is a CDROM image, even though it has a valid MBR *and* a valid GPT. I will have to create something to allow the user to select which to parse...

Yes, this is going to be useful for any hybrid image.

BenLunt wrote:Thanks. It has been (will be) an interesting little project today.

Sounds great! Let us know your results.

BenLunt wrote:I am still investigating, but I think your Root ISO Directory is corrupt. You have a correct '.' entry, but an incorrect '..' entry.
In your FAT partition, your FAT directories are also corrupt. For example, your '.' entries are:
'.\0\32\32\32\32\32\32' '\32\32\32'
In other words, you are null terminating the dot. This is not a valid entry. The dot and dotdot entries should be as follows:
'.\32\32\32\32\32\32\32' '\32\32\32'
'.\.\32\32\32\32\32\32' '\32\32\32'

Please verify.

Could be! I've tried to boot the images as CDROM under bochs, qemu, BIOS and UEFI, and since all worked well, I haven't gave much more thought about it (I don't have IRL CDROM devices any more, so I can't test it on real machines). Thank you for reporting this, I'll take a look at it. Fix will be out today, or tomorrow in worst case.

Cheers,
bzt

Posted: **Sun Jul 26, 2020 9:45 am**

Hi BenLunt,

I've uploaded a new version of mkbootimg. I've also regenerated the images. I was able to mount the image under Linux (with "-t iso9660"), and I could read the bootboot.txt just fine. What's more, I've downloaded some FOSS ISO9660 validators and analyzers that I could find, all seems green. Just for the records, the Boot Catalog was correct, that's why I haven't noticed this (the loader does not use the cdrom direntries), so thank you very much for letting me know about the parent dir issue!

Cheers,
bzt

Posted: **Sun Jul 26, 2020 2:23 pm**

bzt,

I think you still have the dot and dotdot entries wrong in both your ISO 9660 partitions and your FAT16 partition.

In the ISO9660 partition, the ISO 9660 specification, section 6.8.2.2, states that the first entry must be a single byte of '\0' representing the dot entry and the second entry must be a single byte of '\1' representing the dotdot entry. The two entries must not be '.' and '..' respectively as you have it in your image(s). The specification specifically states that a directory name must not include a '.' in the name, therefore the zero and one bytes.

The first entry must have a fi_len field of 1, with an ident value of zero (0x00).
The second entry must have a fi_len field of 1, with an ident value of one (0x01).
These two entries represent the dot and dotdot entries.

In the FAT16 partition, you still have the dot and dotdot entries null terminated.

The dot entry must be
2E 20 20 20 20 20 20 20 20 20 20
while the dotdot entry must be
2E 2E 20 20 20 20 20 20 20 20 20

You currently have it as
2E 00 20 20 20 20 20 20 20 20 20
and
2E 2E 00 20 20 20 20 20 20 20 20

For my sake, this was a good thing since my utility didn't check to see if I was recursively parsing the same directory (dot entry) and ran out of stack space quite quickly. (My code checked for "valid" dot and dotdot entries for the recursion check, not "invalid" entries, though '.' and and '..' were still considered valid after the parser extracted the name. This has been fixed.)

Thanks,
Ben

Posted: **Mon Jul 27, 2020 8:46 am**

Ah, I hate fixing problems that I cannot see... I mean I can mount and use the image both as ISO and FAT without problems, and neither does fsck nor the ISO validators warn about anything. I've quickly put together an iso dumper to see at least what you're saying (attached).

Using that dumper, I've fixed the images, please check it out. If you say it's now okay, then I'll update the mkbootimg binaries too. And many thanks for your help!

Cheers,
bzt

Posted: **Mon Jul 27, 2020 6:43 pm**

Yes, both the CDROM's image looks correct and the FAT image looks correct.

Thanks,
Ben

Posted: **Tue Jul 28, 2020 6:59 am**

BenLunt wrote:Yes, both the CDROM's image looks correct and the FAT image looks correct.

Thanks,
Ben

Thank you! I've uploaded the fixed mkbootimg binaries.

Cheers,
bzt

Posted: **Wed Sep 16, 2020 9:30 pm**

Hi,

And from now on, you can use BOOTBOOT as a coreboot primary payload! This means you don't need UEFI, you don't need BIOS, you can flash the boot loader into the ROM directly. For those who doesn't know, coreboot is an OpenSource firmware used on Chromebooks for example.

Features:

Although coreboot runs in protected mode, it maps 16G of RAM for the long mode kernel
Supports loading recovery initrd over serial line, just like the Raspberry Pi version (speaks raspbootcom protocol, compatible with USBImager -S)
Supports initrd in ROM (both as a Flashmap partition and as a CBFS file)
Supports libpayload's storage (ata, sata, atapi, ahci) and USB disk interface, and a plain simple IDE fallback
If libpayload compiled with support for it, then beside of gzip, BOOTBOOT can use LZMA (xz) and LZ4 compressed initrds too
Implements the level 2 protocol (meaning it parses the kernel for symbols and maps buffers accordingly)

Disadvantages:

You cannot simply compile the x86_64-cb version as the others. You have to patch coreboot (by adding some files under payloads/external), configure it to use BOOTBOOT payload, and then let the build environment do its thing. The resulting build/coreboot.rom will be BOOTBOOT Protocol compatible. Complicated, but this is how the coreboot build environment works.
There's no mode setting interface in libpayload. You have to set up the screen's dimensions at compile time with coreboot's "make menuconfig".

Cheers,
bzt

Posted: **Fri Mar 19, 2021 1:11 am**

Hi,

Little update, BOOTBOOT's disk image creator, mkbootimg can now generate partitions out of a directory in the following formats:

tar - simplest, tape archive format
echidnaFS - used by limine (lot simpler to use than echfs-utils, you just specify the source directory and the rest is taken care for you)
FAT16/32 with long file name support
Minix (V3 with full POSIX permissions and device file support. I couldn't find any doc on it (the link is for V2), so I've used the Minix3 source)
ext2 (revision 1, with file types in directory entries, no up-to-date docs. One would have thought that ext2 is already well-documented, but it's not)
LeanFS (FreeDOS-32's file system, with a lot of help from BenLunt, thanks!)
FS/Z my OS's file system

You can now also specify exactly which driver to use for the generation. This is necessary if partition type allows multiple file system formats.

Code: Select all

{
    "diskguid": "00000000-0000-0000-0000-000000000000",
    "disksize": 128,
    "align": 1024,
    "iso9660": true,
    "config": "boot/sys/config",
    "initrd": { "type": "tar", "gzip": true, "directory": "boot" },
    "partitions": [
        { "type": "boot", "size": 16 },
        { "type": "Microsoft basic data", "size": 32, "name": "MyOS", "directory": "contents/usr", "driver": "fat" }
    ]
}

Here for example partition type "Microsoft basic data" would allow ntfs as well as FAT, so I've added "driver": "fat". This driver switches between FAT16 and 32 depending on the size of the partition. (The boot ESP partition is generated with a specialized code that uses special cluster alignment so that it can be used on CDROMs with 2048 sectors too, able to force fat entry size and only supports 8+3 names. Now additional FAT partitions doesn't have this limitation any more, and the switch is automatic.)

Cheers,
bzt

OSDev.org

BOOTBOOT Multi-platform Micro-kernel Loader

Re: BOOTBOOT Multi-platform Micro-kernel Loader

Re: BOOTBOOT Multi-platform Micro-kernel Loader

Re: BOOTBOOT Multi-platform Micro-kernel Loader

Re: BOOTBOOT Multi-platform Micro-kernel Loader

Re: BOOTBOOT Multi-platform Micro-kernel Loader

Re: BOOTBOOT Multi-platform Micro-kernel Loader

Re: BOOTBOOT Multi-platform Micro-kernel Loader

Re: BOOTBOOT Multi-platform Micro-kernel Loader

Re: BOOTBOOT Multi-platform Micro-kernel Loader

Re: BOOTBOOT Multi-platform Micro-kernel Loader

Re: BOOTBOOT Multi-platform Micro-kernel Loader

Re: BOOTBOOT Multi-platform Micro-kernel Loader

Re: BOOTBOOT Multi-platform Micro-kernel Loader

Re: BOOTBOOT Multi-platform Micro-kernel Loader

Re: BOOTBOOT Multi-platform Micro-kernel Loader