USB Compatible BPB Issue

[clns]

I have been struggling with USB deployment for a week. I really need some advice on how to forge a valid BPB that will deploy gracefully on Floppy, CDROM, and USB, both in a VM and on the bare metal (I don't mind if there are differences for each build).

My setup works fine in bochs and emulates the drive as a floppy. But when I test on all hardware I have I'm getting HDD emulation. I've exhausted the following BPB's they all behave the same on my hardware:
https://github.com/alexfru/BootProg
https://board.flatassembler.net/topic.php?t=17130

I'm developing on windows and I've been using BootProg and Rufus. I've analyzed the contents of the disk through 010 editor and Rufus is working fine and the image does boot. The problem occurs when I read more than 18 sectors.

On bochs things look like this:
DX = 0000 on entry to 7C00
LBA = 18 -> reads sector 18
LBA = 19 -> reads sector 19
LBA = 1A -> reads sector 1A

On the hardware I get this
DX = 0080 on entry to 7C00
LBA = 18 -> reads sector 18
LBA = 19 -> reads sector 3F
LBA = 1A -> reads sector 40

Here is the current BPB that I'm using (lifted straight from Dex), can someone see any issues? I want to avoid developing a boot loader if at all possible, and wish to continue using BootProg or some other convenient alternative under windows.

Code: Select all

 jmp     short init
 nop
 bsOemName               DB      "MSDOS5.0"      ; 0x03
 bpbBytesPerSector       DW      512             ; 0x0B
 bpbSectorsPerCluster    DB      1               ; 0x0D
 bpbReservedSectors      DW      1               ; 0x0E
 bpbNumberOfFATs         DB      2               ; 0x10
 bpbRootEntries          DW      224             ; 0x11
 bpbTotalSectors         DW      2880            ; 0x13
 bpbMedia                DB      0F0h            ; 0x15
 bpbSectorsPerFAT        DW      9               ; 0x16
 bpbSectorsPerTrack      DW      18              ; 0x18
 bpbHeadsPerCylinder     DW      2               ; 0x1A
 bpbHiddenSectors        DD      0               ; 0x1C
 bpbTotalSectorsBig      DD      0               ; 0x20
 bsDriveNumber           DB      0               ; 0x24
 bsUnused                DB      0               ; 0x25
 bsExtBootSignature      DB      29H             ; 0x26
 bsSerialNumber          DD      1                     ; 0x27
 bsVolumeLabel           DB      "BOOT FLOPPY"   ; 0x2B
 bsFileSystem            DB      "FAT12   "      ; 0x36

full boot sector is attached, it dumps each sector and records the LBA, CHS, and DX in the first 6 bytes of each sector

[MichaelPetch]

Just an observation. I didn't look thoroughly through this code but it appears you put the stack at 0x800 and that grows downward. The BDA ends at about 0x500 which pretty much leaves 768 byte (0x800-0x500) on most modern hardware. That's about 768 bytes for stack space before you start overwriting the BDA and 1024 before you start over writing the interrupt table. I've encountered BIOSes that will use considerable stack space for things like disk read etc. I might caution you that 768 bytes might cause issues on some hardware. Not saying that has anything to do with your issues, just an observation.

[Octocontrabass]

Your boot code assumes there will always be 2 heads and 18 sectors per track. This is only true for 1.44MB floppy disks. For your code to work elsewhere, you will need to determine the appropriate parameters to translate between LBA and CHS using INT 0x13 AH=0x08.

(There are some rare situations involving very old floppy disks where INT 0x13 AH=0x08 won't work correctly, but you probably don't care about those.)

[clns]

This is only a test I wrote to isolate the issues I'm having with BootProg on a USB drive. I'm not writing a bootloader. But now that you mention it BootProg depends directly on the BPB for its disk geometry.

It was my understanding that if the BPB looks sane, then Floppy emulation may be used, which would suit me cos I can just use BootProg with no adjustments except to the BPB. This is not what I'm seeing though, I'm getting the same results on 6 different systems. Not one of them uses Floppy emulation

Thanks for the advice. I think I'm unblocked.

[alexfru]

I really need some advice on how to forge a valid BPB that will deploy gracefully on Floppy, CDROM, and USB, both in a VM and on the bare metal (I don't mind if there are differences for each build).

My setup works fine in bochs and emulates the drive as a floppy. But when I test on all hardware I have I'm getting HDD emulation. I've exhausted the following BPB's they all behave the same on my hardware:
https://github.com/alexfru/BootProg

Thanks for using BootProg!

What I did with USB was kinda simple. I FAT32-formatted my 2GB stick using standard Windows tools. I then just used the BPB they created and only changed the code in the boot sector. HDD emulation worked just fine on a few PCs I tried. Perhaps, that's what you should be doing as well. I mean, formatting for HDD emulation.

[clns]

Yeah I think your right there. I'll format it and grab a capture. BTW great tool, much more comfortable for me to develop under windows and it fits seamlessly into the build process.

[Octocontrabass]

But now that you mention it BootProg depends directly on the BPB for its disk geometry.

There's no guarantee that the BIOS will use any particular geometry for a USB drive. You can't rely on the BPB having correct values for anything other than a real floppy disk. You must use INT 0x13 AH=0x08.

It was my understanding that if the BPB looks sane, then Floppy emulation may be used, which would suit me cos I can just use BootProg with no adjustments except to the BPB.

Having a valid BPB is not a guarantee that the BIOS will choose floppy disk emulation. I suspect the BIOS doesn't consider 2 heads and 18 sectors per track a valid geometry for a USB drive anyway. You might have better luck with 255 (not 256) heads and 63 sectors per track.

[Brendan]

Hi,

My setup works fine in bochs and emulates the drive as a floppy. But when I test on all hardware I have I'm getting HDD emulation.

You might like the Problems Booting From USB Flash wiki page.


Cheers,

Brendan

[clns]

I've read that a few times, thanks though Brendan, but there's a big issues with that article: It's full of folklore and superstition. The "must list" towards the bottom is a shaman's ritual. This USB boots just fine on ancient Pentiums with nothing more than the boot signature, all else is ignored. My problem is the cross-over to the second track, it reads wrong sectors. The advice Octocontrabass gave is exactly what I need to do.

[Octocontrabass]

The "must list" towards the bottom is a shaman's ritual.

It's way more conservative than necessary, but it is not superstition. In order to reliably boot from USB on the greatest number of computers, you must have either a valid partition table or a valid BPB, and the criteria for "valid" are complicated and vary significantly between firmware. (The only foolproof way to appease every BIOS is to generate exactly the same partition table or BPB as Windows. Doing this without Windows is a huge challenge.)

This USB boots just fine on ancient Pentiums with nothing more than the boot signature, all else is ignored.

Older BIOSes are not smart enough to validate the boot sector. It's newer BIOSes that will cause the most trouble.

[BenLunt]

I decided to do a little testing and of the computers I have and tried on, here are the requirements to get the same formatted USB thumb drive to boot on all machines:

• Must have four partition entries at offset 0x1BE within the first sector.
• All Boot_ID fields (first field in each entry) must be either 00h or 80h. (only one can be non-zero (80h))
• The Sys_ID field must be non-zero for at least one entry.
• The last two bytes in the first sector (512 byte sectors) must be 0xAA55 (55 AA).

All other fields are ignored. For example, it doesn't matter if none of the entries are marked active. It also doesn't matter what the starting and ending CHS values are, nor what the Starting LBA and Sector Count fields are.

I have a valid DOS 5.0 1.44Meg Floppy BPB with 2 heads, 18 SPT, 2880 sectors, etc., and it still did not matter.

All machines emulated the USB thumb drive as a hard drive. I removed the partition entries and replaced with NOP's and all but one machine would no longer recognize the thumb drive as bootable. That one machine would still boot it, though it still emulated it as a hard drive, even with a floppy BPB.

Therefore, for all the machines I tried, as long as there is a Partition Table with four valid entries, three of them may be NULL's, but one must be a valid entry, all machines booted my code just fine.

There are a few things you must remember though. You must use BIOS service 08h (DL < 80h) or service 48h (DL >= 80h and verified this service is available) to get the parameters, *or* you must check to see that the BIOS emulates the Extended Read Service and use LBA addressing only, period. If you do this, it doesn't matter if the emulation is floppy or hard drive, as long as you do one of two things:

• if DL < 80h, call service 08h, then use the CHS read service (02h)
• if DL >= 80h, use service 42h and LBA addressing (making sure to check for the availability of service 42h).

If you follow those two rules, it won't matter if the emulation is floppy or hard drive emulation.

Anyway, these are my findings.
Ben
- http://www.fysnet.net/the_universal_serial_bus.htm

P.S. A little more digging and it looks like some of the machines I have use the Head count and Sectors per Track count from the BPB, if it exists, and then tries to calculate a cylinder count from that and the total sectors of the USB drive. However, it never uses a value larger than 0x400 (1024) cylinders. If the BPB does not exist, it uses a Head count of 0xFF and a SPT count of 0x3F. If I give it a BPB for a 1.44Meg floppy, it uses a head count of 2 and a SPT count of 18. However, DL is always 80h and the BIOS Disk Extension services are always supported, even with a 1.44Meg floppy BPB and a head count of 2 and a SPT count of 18. :-)

[BenLunt]

For those whom may be interested, I have posted the assembly source code for my few tests reported in the previous post.

https://github.com/fysnet/FYSOS/tree/master/boot

The two files are usbboot.asm and usbboot.bin.

The .asm file is the file used to create the .bin file, assembled with NBASM. Write it to a USB thumb drive and then using the various forms of BIOS settings, see if it is recognized and then try to boot it.

If it is recognized and booted, you should first see a string that states you have made it this far. Then it will print various values from the BIOS "get disk attributes" service(s). This will tell you if your BIOS emulated it as a 1.44Meg floppy or a hard drive.

The source is written so that you can change a few things. For example, clear the USE_FLOPPY equate to use hard drive parameters, or clear the USE_BPB equate to not use a BPB at all.

Feel free to add code to it as well to get a few more bits of information.

All the machines I ran it on emulated it at a hard drive, even with SPT at 18 and Heads at 2. Some of the machines would not even recognize it without the valid Partition Entries. Only one would boot it without the Partition Entries and it even emulated as hard drive even with the 1.44Meg BPB.

Ben

[Antti]

Ben, this may be nitpicking but if you test for 386+ CPU, it would be a good idea to use 8086 compatible instructions when doing it or have a comment line that explains the decision. To me this is like a "do it well or don't do it at all" issue because your code is likely to be more referenced in general. Things like these may give a bad first impression about code quality and that is way more relevant than the test itself.

Push imm16 (opcode 0x68) was not supported or it was undocumented at least.

[clns]

Thanks BenLunt, Your research corresponds to my own. Tested on 6 seperate systems dating from the late 90's, early and late 00's, laptop from 2012 and a brand new AMD just bought a few months ago. All have the exact same behaviour. All emulate USB as Hard Drive, due to a partition table. None of them inspect the BPB from first partition and will boot a USB with nothing more than a boot signature (in the first partition). Haven't got a partitionless set up working, it seems that this is where the head aches come in. Thanks for the code but I ended up adapting the bootloader diskpart uses. That bootloader is compatible with modern windows (BOOTMGR bootloader). It uses INT 13h ah=41h bx=55AAh to test if ah=42h is supported and falls back to ah=8. It does all the stuff you suggested, I only modified it to relocate itself so I could get a larger contiguous free block to read sectors into.

Antti, given that this is a USB boot and that the 386 (1985) predates USB (1996) by a decade, I think it's safe to assume you are not executing on a 8086 (1978). Even the Microsoft boot loader uses 32bit register extensions making it 386 as well. Why on earth would you self impose such a restriction? There is no such thing as an 8086 which boots USB (outside of hobby hardware hacking). I've got a Pentium from the mid 1990's which does't have USB.

[MichaelPetch]

Antti, given that this is a USB boot and that the 386 (1985) predates USB (1996) by a decade, I think it's safe to assume you are not executing on a 8086 (1978). Even the Microsoft boot loader uses 32bit register extensions making it 386 as well. Why on earth would you self impose such a restriction? There is no such thing as an 8086 which boots USB (outside of hobby hardware hacking). I've got a Pentium from the mid 1990's which does't have USB.

Maybe because the OP said this " I really need some advice on how to forge a valid BPB that will deploy gracefully on Floppy, CDROM, and USB" . He was looking for something that deploys on all of them including real floppies.