OSDev.org • Making a bootable image - Page 2

Page 2 of 7

Re: Making a bootable image

Posted: Sat Oct 31, 2020 2:05 am

by Bonfra

BenLunt wrote: It looks like you are using a *nix platform to do this, since you are using DD and other *nix utilities. I have a utility (sadly for your sake it is Windows only) that can manipulate image files with ease. http://www.fysnet.net/ultimate/index.htm

Luckily I use windows, I'm able to use *nix utils thanks to WSL(windows subsystem linux) witch is a natively integrate linux terminal in windows.

Ok so to add a MBR to my bootloader reading trough this I can add at the end of my bootloader, before the magic numer and with the right padding, this:

Code: Select all

UID times 10 db 0             ; Unique Disk ID
PT1 times 16 db 0             ; First Partition Entry
PT2 times 16 db 0             ; Second Partition Entry
PT3 times 16 db 0             ; Third Partition Entry
PT4 times 16 db 0             ; Fourth Partition Entry

Can I leave PT2 PT3 and PT4 as 0 if I want only one partition?
Then I can implement the partition table as a struct:

Code: Select all

struc PartitionTable
    .Drive   resb 1
    .CHSf    resb 3
    .PType   resb 1
    .CHSl    resb 3
    .LBA     resb 4
    .NSec    resb 4
endstruc

And substitute PT1 with the implementation of the struct.
Finally i can write the MBR to the first sector of the hard disk image and it should work right?

Some final questions:
How do I create the partition in the image? the wiki creates a GPT, witch is not what I want as far as i understood. Can i just create the file system on the image with mkfs and as so create only one partition?
Do i really have to poulate the partition table entry my self or can i make them be populated by the code above and so burn a bootloader only in the first part of the MBR leaving the table there?

Re: Making a bootable image

Posted: Sat Oct 31, 2020 6:42 am

by bzt

Bonfra wrote:Luckily I use windows, I'm able to use *nix utils thanks to WSL(windows subsystem linux) witch is a natively integrate linux terminal in windows.

BenLunt's tool is really cool. For writing the image to an USB stick, you can also use USBImager.

Bonfra wrote:Can I leave PT2 PT3 and PT4 as 0 if I want only one partition?

Yes, of course!

Bonfra wrote:Finally i can write the MBR to the first sector of the hard disk image and it should work right?

Yes

Bonfra wrote:How do I create the partition in the image? the wiki creates a GPT, witch is not what I want as far as i understood.

That depends. GPT is the de facto standard so I suggest to use it. You can also create a legacy MBR with fdisk (don't use the "g" command).

Bonfra wrote:Can i just create the file system on the image with mkfs and as so create only one partition?

No. You have to create the file system at the starting LBA where the partitioning table (either MBR or GPT) tells to. That's why the wiki creates a /dev/loop device. For example, if that single partition starts at sector 2048, then you have to create a file system starting from offset 1M (2048*512).

Bonfra wrote:Do i really have to poulate the partition table entry my self or can i make them be populated by the code above and so burn a bootloader only in the first part of the MBR leaving the table there?

You don't need to create it manually. You can use fdisk to create a GPT and take a look at my tool that I've linked which maps the ESP in GPT into the MBR. (Or as I've already said, you can use fdisk to create an MBR table in the first place.) Filling the partitioning table from code also possible and works perfectly. As a matter of fact, you can create both the MBR and the GPT table from code (however the CRC calculation requires some very crafted macro assembler like fasm).

I've created a simple tool that writes disk images (with both MBR and GPT, and also creates a FAT16/32 partition). You can use that as a skeleton for your own disk image creator if you want to. See mkbootimg. I recommend this approach as it takes into consideration how large the disk image is.

Cheers,
bzt

Re: Making a bootable image

Posted: Sat Oct 31, 2020 8:20 am

by Bonfra

bzt wrote: No. You have to create the file system at the starting LBA where the partitioning table (either MBR or GPT) tells to. That's why the wiki creates a /dev/loop device. For example, if that single partition starts at sector 2048, then you have to create a file system starting from offset 1M (2048*512).

Ok, is there another way of doing it? WSL does not have loop module and can't mount devices, so I must find another way...

Re: Making a bootable image

Posted: Sat Oct 31, 2020 8:26 am

by PeterX

Bonfra wrote:
bzt wrote: No. You have to create the file system at the starting LBA where the partitioning table (either MBR or GPT) tells to. That's why the wiki creates a /dev/loop device. For example, if that single partition starts at sector 2048, then you have to create a file system starting from offset 1M (2048*512).
Ok, is there another way of doing it? WSL does not have loop module and can't mount devices, so I must find another way...

Why not double-click on the image? Then AFAIK Windows mounts all partitions. I don't use Windows anymore so I don't know Win10 very well.

Re: Making a bootable image

Posted: Sat Oct 31, 2020 9:27 am

by BenLunt

Bonfra wrote:
bzt wrote: No. You have to create the file system at the starting LBA where the partitioning table (either MBR or GPT) tells to. That's why the wiki creates a /dev/loop device. For example, if that single partition starts at sector 2048, then you have to create a file system starting from offset 1M (2048*512).
Ok, is there another way of doing it? WSL does not have loop module and can't mount devices, so I must find another way...

Since you are using Windows, why not give my utility a try? Creating a new image is very simple. (I just updated the instructions to be more detailed)

Once the image is created, you can easily modify it for your needs.
1) You can adjust the MBR partition entries
2) You can update the boot code for each partition
3) You can insert/delete files on the partition (Currently only FAT partitions are guaranteed to work. Other file systems are experimental)

For example, you only need to create the image once. If you use my utility, it will create the MBR and place the specified image at a LBA of your choosing.
You will need to update the MBR code, but there is an option to do that.

Once you have the image file created, if you need to update a file on the bootable partition, you simply delete the old file and insert the new file.

This utility will allow you to create MBR or GPT style images. I have a detailed example for GPT images.

A few notes:
1) The examples shown at the URLs above all show a Windows XP GUI. This is simply because I think the WinXP theme is much more elegant than the new Win10 theme. However, the utility comes in both WinXP (32-bit) and Win10 (64-bit) versions.
2) Whether you choose a MBR or GPT style image, you will need to place code within the first few sectors to boot and parse the partitions. If you have a Legacy BIOS boot, both styles require some form of code to parse these partitions. If you use a UEFI boot, only the MBR requires code, though the UEFI may not recognize the partitions since it will be looking for a GPT.
3) It is perfectly acceptable, and I think it was bzt that mentioned this elsewhere, to have an image that has both a MBR and a GPT allowing for both firmware styles to boot the same image. Simply have the partition tables in the MBR point to the same partitions the GPT points to. However, this has the one assumption that all partitions are within the 32-bit sector range.

Ben

Update:

I have added detailed instructions on how to create an image file.

Start at the top, then scroll down to the type you wish to create. For the purpose of this thread, you want to scroll down to the Master Boot Record type. Once you follow those instructions, scroll down to the end of the page for "Ready to Create Image" final instructions.

Then once you have done that, go to the instructions about what to do with that MBR image.

Re: Making a bootable image

Posted: Sat Oct 31, 2020 4:24 pm

by Bonfra

PeterX wrote: Why not double-click on the image? Then AFAIK Windows mounts all partitions. I don't use Windows anymore so I don't know Win10 very well.

I thought about this but somehow the image created via dd and modified with fdisk or mkfs is not a valid image, It can't be right clicked so I tried with this powershell line:

Code: Select all

 Mount-DiskImage -ImagePath <Full path to your ISO file>

but is still not recognized.

BenLunt wrote: Since you are using Windows, why not give my utility a try?

I'm studying it right now, I've used it before to inspect the image to study the MBR, now I'm struggling a bit on creating the filesystem on the first partition but I'm confident this will work.
It's an awesome solution for now but in the future I'd like to automate te build process with something like a batch file or a Makefile, does your utility provide some terminal (non gui) interactions?

Re: Making a bootable image

Posted: Sat Oct 31, 2020 4:43 pm

by bzt

Bonfra wrote:I thought about this but somehow the image created via dd

Hmm, I feel disturbance in the force

The dd command does not create images. It just copies sectors one after another. It is the fdisk tool that creates a partitioning table (either MBR or GPT), and mkfs tool that creates an empty file system on a partition.

Bonfra wrote:It's an awesome solution for now but in the future I'd like to automate te build process with something like a batch file or a Makefile, does your utility provide some terminal (non gui) interactions?

Unfortunately I don't know that (sorry I don't use Win except for a mingw vm), but I've written mkbootimg with executing in batch mode from a Makefile in mind.

If you don't want to create a disk image creator, then you can simply echo the commands into fdisk's stdin, that will work (but that's just the partitioning table). For the filesystem, I believe you could use mtools (mformat, mcopy etc.), those work without a loop device.

If everything else fails, you can create a partition image file with the file system, and dd it into the disk image at the partition position or just concatenate. For example:

Code: Select all

dd if=/dev/zero of=partition.dd bs=512 count=X
mkfs.vfat partition.dd
...mount partition.dd and copy files...

(Of course replace X with the required number of sectors) This creates partition.dd without the tables, containing only the file system (no need for file offset).

Code: Select all

dd if=/dev/zero of=table.dd bs=512 count=2048
echo -e "n p\n1\n2048\nX\nt 1\n1\nw\n" | fdisk table.dd
cat table.dd partition.dd > disk.dd

Please note I haven't tested the above, I just wrote it as a guide from memory. Here table.dd is just 2048 sectors long and contains only the paritioning table.

The second variant would be

Code: Select all

dd if=/dev/zero of=disk.dd bs=512 count=$[2048+X]
echo -e "n p\n1\n2048\nX\nt 1\n1\nw\n" | fdisk disk.dd
dd if=partition.dd of=disk.dd conv=notrunc bs=512 seek=$[2048*512]

This writes the partition image (properly prepared with the file system) into the middle of the disk image, which is created with the final size. This has an advantage that this can be used with GPT too (the GPT table is repeated at the end of the disk).

Cheers,
bzt

Re: Making a bootable image

Posted: Sat Oct 31, 2020 5:27 pm

by PeterX

I've thought a lot about a way to format the partition inside of the image in a scriptable way.

I checked fdisk, cfdisk, sfidsk, gparted, parted, mtools, mkfs. But they all rely on the OS they are running on to handle the sector offset stuff. For example gparted simply adds "1" to the image name for the first partition. That doesn't work on a disk image file, off course. Very similar with all tools I named.

Bzt's suggestion works, but unfortunately you have to "dd"/"cat" large amounts of bytes for that.

a) Maybe you install a real Linux besides your Windows and use a loop device.

b) An alternative would be to break with conventions and use an unpartitioned FAT32 volumn on your USB stick. Then mkfs can handle it, and Windows probably, too. But if that boots on real hardware depends on your BIOS.

Re: Making a bootable image

Posted: Sun Nov 01, 2020 2:57 am

by Bonfra

Using bzt's solution I've done:

Code: Select all

dd if=/dev/zero of=partition.dd bs=512 count=65536 # count = [ K = megabyte; K*(1024)^2/512 ]
mkfs.vfat partition.dd

# copy stuffs to partition.dd with mcopy
# mcopy -i partition.dd path/to/file.eg ::/

dd if=/dev/zero of=disk.dd bs=512 count=$[2048+65536]
echo -e "n \n p \n \n \n \n t \n 1\n w" | fdisk disk.dd

dd if=partition.dd of=disk.dd conv=notrunc bs=512 seek=$[2048*512]

This creates the partition file with FAT12 (Also tried FAT16) and copies files in it (not the bootloader yet).
Then it creates the disk image with the MBR first entry.
This far inspecting the images with various tools shows that the partition file has a FAT12 file sysyem with my files inside and the disk file has an MBR which first entry is a FAT12 whit all right address.
Finally when I dd the partition file in the disk image, it is kinda written but is all zeros, the right amount of zeros, but still zeros. BenLunt's tools marks this area as 'unknown'.

Re: Making a bootable image

Posted: Sun Nov 01, 2020 3:58 am

by PeterX

You use:

Code: Select all

dd if=partition.dd of=disk.dd conv=notrunc bs=512 seek=$[2048*512]

but it's kind of "double doing" because you set the block size to 512 AND multiply with 512.
Use instead:

Code: Select all

dd if=partition.dd of=disk.dd conv=notrunc bs=512 seek=2048

Re: Making a bootable image

Posted: Sun Nov 01, 2020 5:41 am

by Bonfra

PeterX wrote: Use instead:
Code: Select all
dd if=partition.dd of=disk.dd conv=notrunc bs=512 seek=2048

Ok it works, the image is valid. Now I "just" neet to add the bootloader.
I think I'll follow the wiki example and make the MBR load the bootloader in the first partition, then it should work from there.

With a few tweaks I can use the wiki's example file and use my old ReadSecotrs function from here.
Honestly this is not my code, I wrote it following a tutorial. I understood how it works but I can't adapt it.
It has this "header:

Code: Select all

Parameters:
    cx => Number of sectors to read
    ax => Starting sector
    es:ebx => Buffer to read to

Instead the wiki assume this:

Code: Select all

EBX - 32-bit LBA Address
CX - Sector Count
ES:DI - Buffer
BYTE [bootDrive] - Drive Number

The drive number I think can be ignored and it'll use the boot one. CX register is fine too. The others (in mine ax and es:ebx) I don't know how to adapt them.

This will load the partition specific bootloader in memory and it will load the second stage bootloader.
I used this (which relies on this) to interact with the FAT12 fs assuming was only it in the image, how do I have to change it to make it work with the file system shifted to the firt partition?

Re: Making a bootable image

Posted: Sun Nov 01, 2020 6:28 am

by PeterX

Try this:
http://ctyme.com/rbrown.htm
Use Int 13/AH=02h for cylinder/head/sector (CHS) addressing
or
Int 13/AH=42h for for linear base adress (LBA) addressing

Re: Making a bootable image

Posted: Sun Nov 01, 2020 9:09 am

by Bonfra

I think I've done the MBR part.
Now I tried adding the bootloader to the partition image but if I add the image became invalid. This is what I've done:

Code: Select all

dd if=/dev/zero of=partition.dd bs=512 count=65536 # count = [ K = megabyte; K*(1024)^2/512 ]
mkfs.vfat -F 16 partition.dd

mcopy -i partition.dd bin/kernel/kernel.sys ::/

dd if=bin/boot/boot.bin of=partition.dd seek=0 count=1 conv=notrunc

At this point partidion.dd is invalid maybe because of the bootloader.
The after creating the disk.dd file and adding the partition to it as so:

Code: Select all

dd if=/dev/zero of=disk.dd bs=512 count=$[2048+65536]
echo -e "n \n p \n \n \n \n t \n 6\n a \n w" | fdisk disk.dd
dd if=bin/boot/mbr.bin of=disk.dd seek=0 count=1 conv=notrunc bs=436

dd if=partition.dd of=disk.dd conv=notrunc bs=512 seek=2048

The partition is still unvalid and marked as unknown by the tool (The partition contains a hello worlds bootloader)

Re: Making a bootable image

Posted: Sun Nov 01, 2020 9:42 am

by BenLunt

Bonfra wrote:The partition is still unvalid and marked as unknown by the tool (The partition contains a hello worlds bootloader)

It marks it as unknown because it did not find a valid file system.

Do you have a FAT file system in that partition, or do you simply have code to print "Hello, World!" ?

Disassembling the bytes shown gives:

Code: Select all

  cli
  xor  ax,ax
  mov  es,ax
  mov  es,ax
  mov  ss,ax
  mov  sp,ax
  ...

This means you don't have a valid BPB, so it is not a FAT file system. My utility looks for file systems. If it doesn't find one, it marks it as unknown.

Also, the second "mov es,ax" should probably be "mov ds,ax"? However, this isn't the elephant in the room, making SS:SP = 0x0000:0x0000 is.

Ben

Re: Making a bootable image

Posted: Sun Nov 01, 2020 9:49 am

by Bonfra

BenLunt wrote: Do you have a FAT file system in that partition, or do you simply have code to print "Hello, World!" ?

The partition has a FAT12 file system created with:

Code: Select all

mkfs.vfat -F 16 partition.dd

Then, after adding some files to the partition i write the bootloader with:

Code: Select all

dd if=bin/boot/boot.bin of=partition.dd seek=0 count=1 conv=notrunc

BenLunt wrote: Disassembling the bytes shown gives:

Yes infact the bootloader is the one fromt this post. (With some slight differences)

In any case even if alter only the first byte of the partition it will be marked as Unknown. Why? Do I have to put the bootloader in a different position?