efi application doesn't appear on iso image

[CobaltEthon]

I am following the https://wiki.osdev.org/EFI page instead of the https://wiki.osdev.org/UEFI_App_Bare_Bones since it's more complete
I've got the "hello world" code building succesfully for x64_64 with clang.

When I start qemu, I can see the FS0 drive and go there.
I expect the BOOTX64.EFI application to be in there so I can boot it, but it's not there.
So I can't run it.

I'm losing my mind a little over it.
the commands I'm executing to build the image, together with (some of) the buildcommands:

Code: Select all

add_executable(BOOTX64.EFI src/hello.c)

target_link_options(BOOTX64.EFI PRIVATE -nostdlib)
target_link_options(BOOTX64.EFI PRIVATE -shared)
target_link_options(BOOTX64.EFI PRIVATE)

target_include_directories(BOOTX64.EFI PRIVATE /usr/include/efi)
target_include_directories(BOOTX64.EFI PRIVATE /usr/include/efi/x86_64)
target_include_directories(BOOTX64.EFI PRIVATE /usr/include/efi/protocol)
target_link_libraries(BOOTX64.EFI PRIVATE /usr/lib/libefi.a)
target_link_libraries(BOOTX64.EFI PRIVATE /usr/lib/libgnuefi.a)

target_compile_options(BOOTX64.EFI PRIVATE -Wl,-subsystem:efi_application)
target_compile_options(BOOTX64.EFI PRIVATE -Wl,-entry:efi_main)
target_compile_options(BOOTX64.EFI PRIVATE -fPIC)
target_compile_options(BOOTX64.EFI PRIVATE -Wno-error=unused-command-line-argument)

add_custom_command( # Make empty image
    COMMAND dd if=/dev/zero of=empty_uefi_partition.img bs=512 count=93750
    OUTPUT empty_uefi_partition.img
)


add_custom_command( # Make EFI partition from empty image
    COMMAND cp --reflink=auto -rf empty_uefi_partition.img uefi_partition.img && parted uefi_partition.img -s -a minimal mklabel gpt && parted uefi_partition.img -s -a minimal mkpart EFI FAT16 2048s 93716s && parted uefi_partition.img -s -a minimal toggle 1 boot
    DEPENDS empty_uefi_partition.img
    OUTPUT uefi_partition.img
)

add_custom_command( # Format temporary EFI partition
    COMMAND dd if=/dev/zero of=temporary_partition.img bs=512 count=91669 && mformat -i temporary_partition.img -h 32 -t 321 -n 64 -c 1
    OUTPUT temporary_partition.img
)

add_custom_command( # CP the kernel to EFI partition
    COMMAND cp --reflink=auto -rf temporary_partition.img filled_partition.img && mcopy -i filled_partition.img BOOTX64.EFI ::
    DEPENDS temporary_partition.img BOOTX64.EFI
    OUTPUT filled_partition.img
)

add_custom_command( # Write filled EFI partition to an image.
    COMMAND cp --reflink=auto -rf temporary_partition.img BOOTX64.ISO && dd if=filled_partition.img of=BOOTX64.ISO bs=512 count=91669 seek=2048 conv=notrunc
    DEPENDS uefi_partition.img filled_partition.img
    OUTPUT BOOTX64.ISO
)

add_custom_target( lauch_BOOTX64.ISO
    COMMAND qemu-system-x86_64 -cpu qemu64 -bios /usr/share/ovmf/OVMF.fd -drive file=BOOTX64.ISO,if=ide -vga std -usb -serial stdio -no-reboot -net none
    DEPENDS BOOTX64.ISO
)

Relevant parts of my stack are:
gnu-efi,
clang,
cmake,
qemu-system-x86_64,

The md5 hashes of all my inbetween files are different, so they are being changed by every step.
I don't know of any way how to see if there is no BOOTX64.EFI file in there or not, if I don't trust my setup (yet).
Is it possible that the file isn't being copied into the image?
Is it possible that the image itsself is malformed?
Or is it possible that the application isn't being recognised as some runnable kernel?

Any suggestions on next steps on how to find out where I made a mistake?
Thanks in advance!

[xeyes]

I don't know of any way how to see if there is no BOOTX64.EFI file in there or not

Why don't you mount it on the host and take a look?

(7zip,winrar or most other archive tool can also open iso and check the contents if 'mount' isn't a valid program/cmd on your host)

[Octocontrabass]

Code: Select all

target_compile_options(BOOTX64.EFI PRIVATE -Wl,-subsystem:efi_application)
target_compile_options(BOOTX64.EFI PRIVATE -Wl,-entry:efi_main)

Are you telling Clang to directly create a PE executable? GNU-EFI requires an ELF executable to work correctly. (This doesn't explain why OVMF can't find your file, though.)

[CobaltEthon]

I don't know of any way how to see if there is no BOOTX64.EFI file in there or not

Why don't you mount it on the host and take a look?

Now i'm wondering why I didn't think of that myself.
It seems like the image is empty, except for an NvVars file.
Which seems expected.


Then that leaves me with the question, why doesn't the following line copy the image into the filled_partition.img file, which then gets further processed into a ISO file?

Code: Select all

mcopy -i filled_partition.img BOOTX64.EFI ::

Code: Select all

target_compile_options(BOOTX64.EFI PRIVATE -Wl,-subsystem:efi_application)
target_compile_options(BOOTX64.EFI PRIVATE -Wl,-entry:efi_main)

Are you telling Clang to directly create a PE executable? GNU-EFI requires an ELF executable to work correctly. (This doesn't explain why OVMF can't find your file, though.)

No I'm using clang with -fshort-wchar -mno-red-zone -ffreestanding -fno-builtin -nostdlib -nostdinc++ in my top level cmake right now.
So if I'm correct, this would create a freestanding EFI application, which then gets linked and booted by GNU-EFI.
I want that, because then I'm able to use GCC without changing my booting process in the future.
I've got secondary reasons for using LLVM, and that' s ehh... mainly me wanting to explore new things.
I probably won't change compilers, but it seems like GNU-EFI is the best described system so I'll stick with that.

It seems like OVMF can't find my file because it's not there.
I don't have time to look into how mcopy is supposed to work right now, but I'll look into it in the evening.
Thank you for your help, I'll post an update by then.

[bzt]

Then that leaves me with the question, why doesn't the following line copy the image into the filled_partition.img file, which then gets further processed into a ISO file?

Well, I don't know about that. You seem to create a lots of unnecessary files:

empty_uefi_partition.img - what's the point of this? Why don't you just create uefi_partition.img with dd?
uefi_partition.img - is the one you create GPT on. I don't see where you use this afterwards...
temporary_partition.img - the one you format as FAT
filled_partition.img - the one you use mcopy on
BOOTX64.ISO - the one you copy from temporary_partition.img and not from filled_partition.img...

Are you absolutely sure you're not lost in this myriad of images? I think you are. You'd only need one, see bootable disks tutorial. You'll need
1. "dd" to create an empty file,
2. "parted"/"fdisk" to create the GPT (doesn't matter which one),
3. "losetup" to create a loop back device that points to the partition inside the image file,
4. "mformat"/"mkfs.vfat" to create a FAT filesystem on that loop back device (doesn't matter which one),
5. "mcopy"/"mount + cp" to copy files to FS0: (you should probably use "mcopy" if you've used "mformat" in dostools)
That's about it.

FYI: if your image does not contain an ISO9660 file system, then you should call it ".img" and not ".iso", that's misleading. Otherwise doesn't matter.

Cheers,
bzt

[Octocontrabass]

No I'm using clang with -fshort-wchar -mno-red-zone -ffreestanding -fno-builtin -nostdlib -nostdinc++ in my top level cmake right now.
So if I'm correct, this would create a freestanding EFI application, which then gets linked and booted by GNU-EFI.

The "-Wl,-subsystem:efi_application" and "-Wl,-entry:efi_main" flags must not be passed to the compiler when you're using GNU-EFI, but it looks like that's exactly what you're doing. Am I misunderstanding something about your build system?

I probably won't change compilers, but it seems like GNU-EFI is the best described system so I'll stick with that.

GNU-EFI is an ugly hack. You'll get results just as good using a proper cross-compiler with a UEFI-compatible (Windows) target, and you won't need things like uefi_call_wrapper(). The only trick is that the most convenient source of UEFI headers is GNU-EFI, but you can find an example of how to use just the UEFI headers without the rest of GNU-EFI on the wiki. (Or better yet, find a different set of UEFI headers.)

[CobaltEthon]

Turns out that the second dd command on the wiki doesn't work out of the box.
I haven't really the filled_partition.img has the BOOTX64.EFI file in there, but when using dd to write that intermediate image to the end result image it seems that the result is not an image with the runnable executable.
So the following chunk of code doesn't result in an image with a program in there.

Code: Select all

 dd if=filled_partition.img of=BOOTX64.ISO bs=512 count=91669 seek=2048 conv=notrunc

Which is my interpretation of the following chunk of code on the wiki page https://wiki.osdev.org/UEFI#Emulation_w ... U_and_OVMF.

Code: Select all

$ dd if=/tmp/part.img of=/path/to/uefi.img bs=512 count=91669 seek=2048 conv=notrunc

(it also doesn't work if I rename the .iso file to an .img file.

Now I never really use dd, so I can't really tell why it doesn't work of the bet.
And unfortunately the wiki page doesn't explain why one image has to be copied over the other, only that it should be done.
I was hoping getting a bootable image was a little less work and figure out what exactly happened after that (and adjust to my liking), but it seems to be a bit more difficult than I hoped.

Does anyone know why dd-ing one image which contains a file over an image that is gpt formatted and has an EFI partition with boot flag enabled is a step that must be taken?
Can't I just do it simpler with one dd command?
Also, why am I doing it wrong then?

I seem I could simplify the cmake a bit into the following for building an image.

Code: Select all

add_custom_command( # Make bootable FAT16 EFI partition with GPT partition scheme.
    COMMAND parted uefi_partition.img -s -a minimal mklabel gpt && parted uefi_partition.img -s -a minimal mkpart EFI FAT16 2048s 93716s && parted uefi_partition.img -s -a minimal toggle 1 boot
    OUTPUT uefi_partition.img
)

add_custom_command( # Format empty EFI partition
    COMMAND dd if=/dev/zero of=empty_partition.img bs=512 count=91669 && mformat -i empty_partition.img -h 32 -t 321 -n 64 -c 1
    OUTPUT empty_partition.img
)

add_custom_command( # CP the kernel to EFI partition
    COMMAND cp --reflink=auto -rf empty_partition.img filled_partition.img && mcopy -i filled_partition.img BOOTX64.EFI ::
    DEPENDS empty_partition.img BOOTX64.EFI
    OUTPUT filled_partition.img
)

add_custom_command( # Write filled EFI partition to an image.
    COMMAND cp --reflink=auto -rf empty_partition.img uefi.img && dd if=filled_partition.img of=uefi.img bs=512 count=91669 seek=2048 conv=notrunc
    DEPENDS uefi_partition.img filled_partition.img
    OUTPUT uefi.img
)

[zaval]

"simpler" is just create a virtual disk* (no matter GPT or MBR) with a FAT volume and put there your loader file(s), say into \efi\cobalt\x64 and start it either through the UEFI shell or a Load Option. why on earth one would want to regenerate 49MB iso file everytime or use GNU-EFI or use ELF or fPIC ... - I have no idea. I just described (yet once) how I do. Octocontrabass already explained well on the matter.

* - this is for emulators, for real hardware, using a USB stick the same way is the easiest way.

[bzt]

it also doesn't work if I rename the .iso file to an .img file.

Yep, it's for the people who download your image. The .iso would suggest them to use it as a CDROM image, and .img to use it as a disk image. Otherwise the file extension is irrelevant.

Does anyone know why dd-ing one image which contains a file over an image that is gpt formatted and has an EFI partition with boot flag enabled is a step that must be taken?

Well, the GPT requires space at the beginning of the image, so the file system can't start at the beginning of the file, as mformat/mount expects (it must start at offset 2048*512). So you create a disk image with the GPT and after that you have two options:
1. create another image, a one partition-only image with the file system, and then you copy that image into the first disk image at the partition's position (requires two dd commands, the second with a "seek" and "notrunc")
2. you create a loop back device that points somewhere in the middle of the disk image at the partition's position, then you use that loop device to create a file system (one dd is enough)

The advantage of 2. is that don't need two images, as the loop device is just a pseudo-file which points into the image, setting it's offset 0 to the image file's offset 2048*512. For example when you write to that loop device at offset 10, then you would actually writing to the image file at offset 2048*512+10. This little trick allows mformat/mount to work properly.

Also, why am I doing it wrong then?

I've told you, you're mixing up the images. You create the GPT in one image, and then copy the partition into another.
You create the GPT and the ESP partition entry in uefi_partition.img

Code: Select all

COMMAND parted uefi_partition.img -s -a minimal mklabel gpt && parted uefi_partition.img -s -a minimal mkpart EFI FAT16 2048s 93716s && parted uefi_partition.img -s -a minimal toggle 1 boot

but you copy and place the partition into empty_partition.img (previously was temporary_partition.img, but that wasn't correct either)

Code: Select all

COMMAND cp --reflink=auto -rf empty_partition.img uefi.img && dd if=filled_partition.img of=uefi.img bs=512 count=91669 seek=2048 conv=notrunc

You need to dd the file system image (filled_partition.img) into the disk image with the partitioning table (uefi_partition.img).

To put you in perspective, if your image file were a real device, then for example uefi_partition.img would be /dev/sda (entire disk), and filled_partition.img would be /dev/sda1 (one partition only). With Windows terminology, uefi_partition.img would be \\.\PhysicalDisk\0 and filled_partitition.img would be C:.

Cheers,
bzt

[CobaltEthon]

why on earth one would want to regenerate 49MB iso file everytime or use GNU-EFI or use ELF or fPIC ... - I have no idea.

Just making my way down the tutorial.

"simpler" is just create a virtual disk* (no matter GPT or MBR) with a FAT volume and put there your loader file(s), say into \efi\cobalt\x64 and start it either through the UEFI shell or a Load Option.

That's the next step. I just wanted something I could use for development and maybe even for deployment to start, then simplify things for development.

You need to dd the file system image (filled_partition.img) into the disk image with the partitioning table (uefi_partition.img)

Thank you! I knew I was overseeing something dumb.
Now I've "only" got the problem of the file not running, but that's something I should be able to solve on my own.

The "-Wl,-subsystem:efi_application" and "-Wl,-entry:efi_main" flags must not be passed to the compiler when you're using GNU-EFI, but it looks like that's exactly what you're doing. Am I misunderstanding something about your build system?

No you're right. That's a remainder from using UEFI_App_Bare_Bones before realizing the UEFI article contained a much simpler setup.
There's also a top level cmake that I plan to use to strap all the libraries together and have compilation flags that should be present everywhere else, and those were the ones I listed.
But yes, these flags are added to that.
So this won't boot, and I need to fix that.

GNU-EFI is an ugly hack. You'll get results just as good using a proper cross-compiler with a UEFI-compatible (Windows) target, and you won't need things like uefi_call_wrapper(). The only trick is that the most convenient source of UEFI headers is GNU-EFI, but you can find an example of how to use just the UEFI headers without the rest of GNU-EFI on the wiki. (Or better yet, find a different set of UEFI headers.)

I will do that.


Thank you all for the help, I should be able to get by from here on my own.