UEFI bare bones bootable usb stick

[jamesread]

Hi,

I am working my way through the https://wiki.osdev.org/UEFI_Bare_Bones tutorial. I haven't been able to install the OVFM firmware to test on Qemu so I wanted to skip to test directly on my spare hardware.

To this end how do you make a bootable USB. I tried sudo dd if=cdimage.iso of=/dev/sdb && sync but the resulting USB stick is unbootable. In the file browser I get two files on the USB stick boot.catalog and fat.img

[bzt]

I haven't been able to install the OVFM firmware to test on Qemu so I wanted to skip to test directly on my spare hardware.

Bad idea. Always try your code in an emulator first. What was the problem? Do you have any error messages? Those could be helpful.

To this end how do you make a bootable USB.

Read https://wiki.osdev.org/Bootable_Disk, that may help. The https://wiki.osdev.org/UEFI#Creating_disk_images page also has examples on how to create such image. For creating a bootable image programatically, take a look at my mkbootimg tool.

I tried sudo dd if=cdimage.iso of=/dev/sdb && sync but the resulting USB stick is unbootable.

Of course. The ISO is an image of an ISO9660 file system, but for booting from an USB stick under UEFI you'll need an image partitioned with GPT, having an ESP partition formatted with FAT file system.

For the records, qemu is capable of creating such an image on-the-fly. You just specify a directory on the command line, and the guest VM will see that as a perfectly valid ESP.

Code: Select all

qemu-system-x86_64 -bios $(OMVF) -m 64 -hda fat:$(ESPdir) ...

Note that OVMF requires at least 64Mb of RAM, otherwise it won't work. You only need the ".fd" file, nothing else.
If you want to use EFI NVRAM variables too, then read https://wiki.osdev.org/UEFI#Emulation_w ... U_and_OVMF.

The quickest and dirtiest, but also simplest solution, dd my ready-to-use image to your USB stick, mount it, then replace EFI/BOOT/BOOTX64.EFI with your own loader.

Cheers,
bzt

[jamesread]

Regarding OVMF I ran

Code: Select all

git clone https://github.com/tianocore/edk2.git

Then in the absence of any installation instructions I ran:

Code: Select all

cd edk2/
./edksetup.sh 

This gave me the following output:

Code: Select all

WORKSPACE: /home/yaakov/osdev002/edk2
EDK_TOOLS_PATH: /home/yaakov/osdev002/edk2/BaseTools
CONF_PATH: /home/yaakov/osdev002/edk2/Conf
Copying $EDK_TOOLS_PATH/Conf/build_rule.template
     to /home/yaakov/osdev002/edk2/Conf/build_rule.txt
Copying $EDK_TOOLS_PATH/Conf/tools_def.template
     to /home/yaakov/osdev002/edk2/Conf/tools_def.txt
Copying $EDK_TOOLS_PATH/Conf/target.template
     to /home/yaakov/osdev002/edk2/Conf/target.txt
./edksetup.sh: line 214: return: can only `return' from a function or sourced script

I then tried:

Code: Select all

cd OvmfPkg/
./build.sh

This gave me the following error:

Code: Select all

make -C BrotliCompress
make[2]: Entering directory '/home/yaakov/osdev002/edk2/BaseTools/Source/C/BrotliCompress'
gcc  -c -I ./brotli/c/include -I .. -I ../Include/Common -I ../Include/ -I ../Include/IndustryStandard -I ../Common/ -I .. -I . -I ../Include/X64/ -MD -fshort-wchar -fno-strict-aliasing -fwrapv -fno-delete-null-pointer-checks -Wall -Werror -Wno-deprecated-declarations -Wno-stringop-truncation -Wno-restrict -Wno-unused-result -nostdlib -g -O2  BrotliCompress.c -o BrotliCompress.o
BrotliCompress.c:20:10: fatal error: ./brotli/c/common/constants.h: No such file or directory
   20 | #include "./brotli/c/common/constants.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [../Makefiles/footer.makefile:21: BrotliCompress.o] Error 1
make[2]: Leaving directory '/home/yaakov/osdev002/edk2/BaseTools/Source/C/BrotliCompress'
make[1]: *** [GNUmakefile:77: BrotliCompress] Error 2
make[1]: Leaving directory '/home/yaakov/osdev002/edk2/BaseTools/Source/C'
make: *** [GNUmakefile:19: Source/C] Error 2
make: Leaving directory '/home/yaakov/osdev002/edk2/BaseTools'

Given that the information in the tutorial seems outdated and that OVMF seems to lack installation instructions I gave up. I do agree though. It was a bad idea to skip testing in Qemu.

[bzt]

Regarding OVMF I ran

Just download a pre-compiled OVMF.fd from the official site:
https://github.com/tianocore/tianocore. ... /wiki/OVMF
or try one of these:
https://github.com/BlankOn/ovmf-blobs

Cheers,
bzt

[jamesread]

Success. Thanks. OVMF is now working. Still working on getting a bootable USB working though.

[bzt]

Success. Thanks. OVMF is now working. Still working on getting a bootable USB working though.

Well done! Try to dd my known-to-work image to an USB stick, mount it and replace EFI/BOOT/BOOTX64.EFI, that's the simplest way. Or you can create a loop device on the image directly and mount that.

Code: Select all

# gzip -d disk-x86.img.gz
# losetup -o65536 /dev/loop0 disk-x86.img
# mount /dev/loop0 somedir
# cp yourloader.efi somedir/EFI/BOOT/BOOTX64.EFI
# umount somedir
# losetup -d /dev/loop0

and you can boot that disk-x86.img with qemu too. Otherwise follow the instructions on Bootable Disk wiki page if you want to create everything from scratch.

Cheers,
bzt

[jamesread]

What's the correct command to test with Qemu and the new image I have created? I tried

Code: Select all

qemu-system-x86_64 -L OVMF_dir/ -bios osdev002/OVMF-pure-efi.fd -cdrom osdev005/disk-x86.img

I also dd'd the image to my USB but no joy trying to boot from it.

[isaacwoods]

This gave me the following error:

You need to fetch the submodules. Try:

Code: Select all

git submodule update --init

[zaval]

I tried sudo dd if=cdimage.iso of=/dev/sdb && sync but the resulting USB stick is unbootable.

Of course. The ISO is an image of an ISO9660 file system, but for booting from an USB stick under UEFI you'll need an image partitioned with GPT, having an ESP partition formatted with FAT file system.

here we go again... despite I've shown even the screenshots, proving it's a wrong and misleading statement, bzt keeps repeating it, further confusing new people. This is not true! Not for OVMF nor for real hardware.

james, you say, you dd'd image, now try this. during FW init, break into it - either Boot Manager or UEFI shell and if the first, look for "boot from the file" option in the FW Boot Manager, if the second, launch your .efi file from the shell. In the former case, you might locate it, knowing it sits on the USB and is CD file system, and, then move through the file system tree until you find your .efi file, and launch it, in the second case - just type it's path in the shell, the common way. Assuming, that, of course, that tutorial recommends using the "last attempt" option, the file system part of the path would be something like:
\efi\boot\bootia32.efi

if you really made the image properly, it will work. unlike what bzt says, it doesn't have to be GPT nor ESP. even better, you just can test your loader without burning the whole USB stick with an iso image, just make an "efi" folder in the root of the stick, then make there your folder, say "james" (\efi\james), and then put there your files and launch them the way I described. later, you'll create a Load Option and FW will start it automatically.

[bzt]

What's the correct command to test with Qemu and the new image I have created?

Take a look at the Makefile on the link I've provided. It has several qemu invocations. For example "make efi" will boot it from disk with EFI, and "make eficdrom" will boot from cdrom using EFI.

Code: Select all

efi:
	qemu-system-x86_64 -bios $(OVMF) -m 64 -drive file=disk-x86.img,format=raw -serial stdio

eficdrom:
	qemu-system-x86_64 -bios $(OVMF) -m 64 -cdrom disk-x86.img -serial stdio

The image has been tested with these commands and known to work.

I tried

Code: Select all

qemu-system-x86_64 -L OVMF_dir/ -bios osdev002/OVMF-pure-efi.fd -cdrom osdev005/disk-x86.img

With "-cdrom" you'll need an ISO9660 image with a Boot Catalog. My mkbootimg tool created a special image that has such catalog with both BIOS boot and EFI boot entries. It shouldn't matter if you load it from disk or a cdrom. But for that to work, you need a special hybrid layout and an ISO9660 El Torito Boot Catalog in the image.

I also dd'd the image to my USB but no joy trying to boot from it.

You must have used a bad image. Can you boot the original, unmodified image? Have you made any other modifications to it other than overwriting BOOTX64.EFI? Have you unmounted and sync'd properly? Do you have any error messages?

But first of all, can you boot the unmodified image?

here we go again... despite I've shown even the screenshots, proving it's a wrong and misleading statement, bzt keeps repeating it, further confusing new people. This is not true! Not for OVMF nor for real hardware.

Nobody cares what screenshots you have shown (proves nothing, plus you've already represented us photoshoped "screenshots"), and it is not me who is saying dd'ing a cdrom image to an USB stick doesn't work. It is the OP who has boot problems with that configuration! Just saying I'm wrong when they experience the opposite won't help them, not even a bit! Why don't you try to make yourself useful and figure out why "dd if=iso of=stick" is NOT working for them?

Cheers,
bzt

[zaval]

bzt, I wasn't and am not going to argue with you, you are wrong and I've proven that, I just wanted to help the OP. that's all. I never posted here "photoshopped" screenshots. we don't know, why his stick isn't working. your conclusion, that his problem is because of it's not GPT and ESP is just as hilarious as many things from you when it comes to understanding UEFI (somehow ).

but it's really getting tiresome. for being purely on topic. james, let's do it easier - just create a folder on a FAT USB stick, as I outlined in the above post, put there your OSL and its stuff and be sure - it will work. I did this a million times, both on real hardware and emulators and it always worked. it's the easiest for the beginning. why would you need to burn a USB stick as a "CDROM" anyway? maybe for bzt, creating an iso and burning it everytime is easier, than updating the folder, but for most of people, I'd guess, the latter is easier.

[Doctor5555]

I'm going to assume that you have a valid 64-bit PE executable built and linked using clang or a gcc cross compiler (not the .img file).
In order to run this in QEMU in the simplest way possible, start with an empty folder. Make sure the executable is named "bootx64.efi", and place it:

Code: Select all

<your folder>/efi/boot/bootx64.efi

Next, replace the 'path/to' s with the correct paths and then run the following command:

Code: Select all

qemu-system-x86_64 -drive if=pflash,format=raw,file=path/to/OVMF.fd -drive format=raw,file=fat:rw:path/to/<your folder> -M accel=kvm:tcg -serial stdio

This tells QEMU to emulate the folder as a fat-formatted drive and allows it to be booted from, without needing to make an image every time.

[bzt]

bzt, I wasn't and am not going to argue with you, you are wrong and I've proven that

Yes, you're arguing, you're saying I'm wrong while you've proven nothing.

Writing an ISO9660 image on an USB stick does not work for the OP. Period.

ps: it does not work on my testbeds either. Neither in UEFI, nor in legacy CSM mode. The UEFI spec does not mention that ISO9660 should work on non-CDROM media, where did you get that?
- Page 320 section 10.3.5.1 there's no El Torito option in the Hard Disk Media Device Path
- Page 321 section 10.3.5.2 says that for CDROM Media Device Path implicitly El Torito ISO-9660 file system assumed (it does not say other paths should recognize that file system)
- Page 502 section 13.3.2.1 says that partition discovery for CDROMs and DVDROMs use El Torito ISO-9660 (it does not say this file system should be recognized on non-CDROM paths)

Conclusion: whether ISO9660 can be used on an USB stick or not is entirely firmware implementation specific. You have a firmware that can, while me and the OP have such a firmware which can't.

Cheers,
bzt

[jamesread]

Code: Select all

efi:
	qemu-system-x86_64 -bios $(OVMF) -m 64 -drive file=disk-x86.img,format=raw -serial stdio

I tried with this command and got the following output

Code: Select all

BdsDxe: failed to load Boot0001 "UEFI QEMU DVD-ROM QM00003 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Secondary,Master,0x0): Not Found
BdsDxe: loading Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Primary,Master,0x0)
BdsDxe: starting Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Primary,Master,0x0)
Hello, world!
BdsDxe: loading Boot0000 "UiApp" from Fv(7CB8BDC9-F8EB-4F34-AAEA-3EE4AF6516A1)/FvFile(462CAA21-7614-4503-836E-8AB6F4662331)
BdsDxe: starting Boot0000 "UiApp" from Fv(7CB8BDC9-F8EB-4F34-AAEA-3EE4AF6516A1)/FvFile(462CAA21-7614-4503-836E-8AB6F4662331)

So, I guess it's working. It doesn't seem to work when I boot a machine from USB though.

[jamesread]

OK, I just tried on another device (lenovo ideapad 320). It seems to be stuck in a never ending loop of booting and rebooting. It prints some stuff to screen but no Hello World message. The output doesn't stay on screen long enough for me to be able to copy it and post it here.