Booting from a USB and BIOS faults

[BenLunt]

Hi guys,

It has been a while, but I have found a bit of time to come back and participate. I have been following the "Bootloaderis not consistent when booting in real hardware" thread, particularly with the start of BIOS fault conversation.

I can confirm that that particular BIOS, the one that Octocontrabass mentions, does in fact modify your code/data depending on the BPB and/or Partition Table entry it finds.

For example, if you go to my bit of code (which has a 1.44Meg binary file if you don't want to assemble it) will show that offset 0x012F of the returned data is corrupt. After the read, whether I use CHS or Disk Extentions, the word at 0x12F of the returned read buffer is corrupt.

At first I thought it was my code. However, there are a few modifications to the code that will change this.

First Example: (With IS_HARDDRIVE == 0)
Instead of a valid partition entry, clear all sixteen bytes to zero, then run again. The word at 0x12F is no longer corrupt.

Code: Select all

; modify existing code as so:
    ;       db  0  ; boot id
    ;         db  0 ; start head
    ;         db  1 ; start sector
    ;         db  0 ; start cyl 
    ;       db  1 ; sys id   1 = FAT12
    ;         db  1 ; end head
    ;         db 18 ; end sector
    ;         db 79 ; end cyl 
    ;       dd     0  ; start lba
    ;       dd  2880  ; size in sectors

           dup 16,0

Second Example: (With IS_HARDDRIVE == 0)
Keep the partition entry as is, a 1.44m entry, and instead of writing zeros to the remainder of the disk, write 0xFF's. The word at 0x12F is no longer corrupt.

Code: Select all

; modify existing code as so:
 ; temp_buff  dup ((2880 * 512) - $),00h

temp_buff  dup ((2880 * 512) - $),0FFh

Third Example: (With IS_HARDDRIVE == 1)
Set IS_HARDDRIVE to 1 so that it uses a FAT16 BPB and a "larger" Partition Table entry and the word at 0x12F is no longer corrupt.

Code: Select all

; modify existing code as so:
 ; IS_HARDDRIVE     equ  0            ; 0 = use 1.44 floppy emulation, 1 = use hard drive emulation

IS_HARDDRIVE     equ  1            ; 0 = use 1.44 floppy emulation, 1 = use hard drive emulation

This proves to me that the BIOS does in fact look for a valid BPB and/or MBR Partition entry. However, I do not know why it is modifying the word at offset 0x12F of the returned read buffer.

Granted, why would I have a MBR Partition Entry if I was going to try to emulate a 1.44Meg floppy?

I wouldn't, however, the reason for it here, is to prove that indeed, this particular BIOS does check for a BPB and/or Partition Table, and corrupts the data if the data found is not "the normal way of doing it".

I can only speculate here, but I am guessing with the 2nd example above is that the BIOS is checking the actual FAT and/or ROOT of the (non-existent) partition. I am guessing more on the FAT than the ROOT, since a value of 0xFF will be "no more entries". Though this is only speculation.

Anyway, I have spent the past couple of hours modifying the code slightly each time to see what results I got. I checked two different BIOSes, both of the same manufacture, one considerably newer than the other, and both give the exact results.

Also, since I don't have a valid FAT File System on the USB drive, both BIOSes return 0x01CA4000 sectors, which is the whole 16-gig drive, no matter if the IS_HARDDRIVE equate is zero or one.

My conclusion is, at least for this particular BIOS, if you want to emulate a 1.44Meg floppy, you must have a valid BPB, no Partition Table, and a valid FAT, at least the first two entries (0xFFF and 0xFF0). I am not 100% confident that this is the case, but currently I am thinking this. More tests will be in order.

Anyway, I just thought I would share. I am not trying to say that one person is right and the other is wrong, I am just showing proof of a bug in the BIOS. Maybe not a "bug", but an irregularity if the data on the USB disk is not the "normal expected type". Which then means, a lot of newbies might be pulling their hair out asking why their code doesn't work when in fact it is a irregularity in the BIOS emulation, not their code.

Feel free to take my code and modify it to see what results you get. I would be very interested in seeing what you find.

Ben
- http://www.fysnet.net/the_universal_serial_bus.htm

[Octocontrabass]

I can confirm that that particular BIOS, the one that Octocontrabass mentions, does in fact modify your code/data depending on the BPB and/or Partition Table entry it finds.

Which one are you talking about? I've never actually seen a BIOS modify my code/data, I only know about it from reading threads here where someone found their PC doing it.

After the read, whether I use CHS or Disk Extentions, the word at 0x12F of the returned read buffer is corrupt.

When you say corrupt, what is it returning in place of the expected data?

emulate a 1.44Meg floppy

Or a 16GB floppy. Every BIOS I've seen gives you a way to access the whole capacity of the disk. (But then, I don't usually use a floppy disk BPB, since that isn't what Windows writes to the disk during formatting.)

[bzt]

By modifying data do you mean writing sector data back to disk? It's hard to imagine. Otherwise if I understand correctly, your boot sector code should just avoid to use the byte at 7D2Fh and you're good even on that buggy BIOS.

And, as I've asked in the other topic as well, does this particular buggy, non-BSS compliant BIOS produce the same error when "floppy emulation" is turned off, and the boot sector code is not using CHS at all but LBA only?

Cheers,
bzt

[Octocontrabass]

By modifying data do you mean writing sector data back to disk?

No, only in memory. (Unless Ben is talking about something different from what I've seen.)

[BenLunt]

This particular BIOS actually changes two bytes in the 512-byte block read, after placing it in memory at 0x7E00 and before returning from the INT 13h call.

The current code (and binary file) I linked to, modifies the word at offset 0x12F within that buffer. Offset 0x7F2F. This happens to be the NULL and SPACE chars just after the 'SPT:' string literal.

Code: Select all

small_params_spt   db  ' SPT: ',0
small_params_total db  ' Total Sectors: ',0

The zero at the end of the first line and the space before the T in Total in the second line.

So, with this in mind, I made a few modifications, which changed the position of small_params_spt in the code expecting two different chars to be modified.

Two different chars were modified, but not as I expected. This time the same two chars were modified with the same new value but at:

Code: Select all

small_params_cyls  db  ' Cylinders: ',0
small_params_heads db  ' Heads: ',0

The zero at the end of the first line and the space before the H in Heads in the second line.

I thought maybe it was coincidence that I modified the code enough to move the data to happen to be at this point. Not that case.

The BIOS actually found a NULL and a SPACE char and modified it.

Now, what is interesting is that the value found is:

Code: Select all

   00 20   

and the new value it writes to that memory is:

Code: Select all

   FF 2F   

Note that the first two bytes are actually an EMPTY FAT ENTRY. i.e.: In the buffer at 0x07E00, the first sequence of a 12-bit value of zero is changed to the second set of bytes.
The BIOS is actually marking an entry in the FAT (or what it thinks is a FAT) as used.

Here's my proof: (Well at least what I have so far. I haven't investigated further)
The second sector of the disk, of a 1.44 meg floppy, is the first FAT. The BIOS evaluated my "image" as a FAT-12 volume and has stored a cache of (at least) the second sector in memory. For some reason, it has found and marked the first free FAT entry as used (0xFFF). Then when I read the sector of the disk, it returns the cached sector it thinks is the first sector of the FAT.

This tells me that the BIOS in question is actually emulating the disk as its own version of a FAT12 volume and returns its modified FAT instead of the actual sector physically residing on the disk.

In theory, this will work just fine *IF* the volume is indeed an actual valid FAT12 volume. However, this leaves two questions:
1) Why is it finding an empty FAT entry and marking it as used (0xFFF)
2) What is it actually writing to the (cached) volume at that found cluster.

To prove my findings even more, if I move the first instance of the 12-bit value of 0x000 to not align on a 12-bit cluster entry, the BIOS finds the *next* aligned 12-bit value of 0x000. This tells me it is actually looking for a free cluster entry in a FAT12 FAT.

If I want to investigate further, I could find out which entry this actually was (which I have), and read the 512-byte sector from the disk. Would the BIOS return the actual data from the (USB) drive or would it return a cached sector (cluster, since clusters are one-sector in size)? If it returns a cached sector, what now occupies this cached sector? Definitely not what I, as a newbie would think it would be if I am at the first stages of my project and have a multi-sector boot sector which includes this found (cached) sector.

Now, before you get too far into this, yes, most of the time, I would have a completely valid FAT12 volume on the (USB) disk and this would not matter. However, I want to make two points:

1) A newbie that has yet to create a file system, solely wants to create a multi-sector boot sector (and most newbies do exactly this) to see if they can boot their new project. What happens when this BIOS modifies the first three-nibble null value it finds? (P.S. This three-nibble value has to be aligned on a 12-bit cluster entry as well). For example, what if the second sector of my multi-sector boot sector contains the following:

Code: Select all

SomeString     db  'A String to Print',0
AnotherString  db  ' This string starts with a space char which has a value of 0x20',0

When the newbie tells the BIOS to print the first string, the BIOS will actually print both strings with a value of 0xFF 0x2F in-between each string. Exactly what is happening in my example I link to above.

2) The BIOS is actually emulating a FAT 12 volume, not a 1.44M floppy disk.

To make a long story short, ha ha, this is actually a rare case where the user has placed a valid FAT12 BPB at the start of the disk, yet doesn't have a valid FAT12 volume on the disk. When would this happen in real life? I merely show this for two reasons, one that this little "bug" caught me, and two, this proves that the BIOS (at least the one in question here) can and does modify your boot sector/volume data when emulating the USB as a disk.

Again, this is a rare occasion; you have to have the (almost) exact sequence I have shown; you have to be booting as a legacy BIOS boot; and you have to be using a non-conforming FAT volume to get this to happen.

Though my whole point is......It did happen. :-)

Ben

P.S. Please note that I am not trying to start/continue a fight of any kind--a fight of who is right and who is wrong. I am merely trying to prove, for a newbie's sake, that if you wish to boot your code from a USB disk from a Legacy BIOS boot, *and* use a FAT12 formatted volume, you *must* have a valid FAT12 volume and BPB, or it will not work...

I removed both the BPB and the partition table entry from my example and this particular BIOS booted my code without the (erroneous) modification mentioned above.

[Octocontrabass]

Well... now I know the BPB alone isn't enough to satisfy some firmware.

However, this leaves two questions:
1) Why is it finding an empty FAT entry and marking it as used (0xFFF)
2) What is it actually writing to the (cached) volume at that found cluster.

I have a third question: what is the name of the file occupying this cluster?

[BenLunt]

Well... now I know the BPB alone isn't enough to satisfy some firmware.

However, this leaves two questions:
1) Why is it finding an empty FAT entry and marking it as used (0xFFF)
2) What is it actually writing to the (cached) volume at that found cluster.

I have a third question: what is the name of the file occupying this cluster?

Three of many questions. I wish I knew.

Remember that in the example I give, everything including the third sector to the end of the "disk" is nulls, which includes the root directory, which means the whole directory is empty.

Anyway, to conclude this example, to make sure your USB thumb drive is bootable, my recommendations are to use one of the following two scenarios, with the first one to be more true than the second.

1. Use a properly formatted partitioning system, with a properly formatted FAT-12, -16, or -32 volume.
   • If you use a FAT-12 volume, you may skip the partitioning scheme, but may only have access to the single volume.
   • If you have a BPB, you must have a properly formatted FAT and Root Directory as well.
   • Do not use both a BPB and a MBR style partitioning scheme in the first sector.
   • It is recommended that you use a FAT-32 formatted volume, with a cluster size of 32k. (Please see the note below)
2. Do not use any form of a BPB within the first sector of the disk.
   • Do not use a jump as the first instruction in the first sector.
   • Do not place any data that could presumably be valid BPB data within the first 64 bytes of the first sector.
   • If using a MBR partitioning scheme, be sure to follow either of, without mixing, these two scenarios for the volumes within.

It has been my impression that if you use any other form of formatting, the firmware (BIOS) may not, and is not guaranteed to successfully boot your USB thumb drive.

Why use FAT-32? The following is from my notes, from a page that now reports 404, but can be found using the WayBack Machine.

Most portable flash media come preformatted as with a FAT32 file system. This is not only done because there is support for this file system in all operating systems, it is actually a reasonably good choice for the media: The data on a FAT32 file system is always written in clusters of e.g. 32 KB, and the media are normally formatted with a cluster size matching the optimum write size, as well as aligning the clusters to the start of internal units, and the access patterns on a FAT32 file system are relatively predictable, alternating between data blocks, file allocation table (FAT) and directories.

The cards take advantage of this knowledge by optimizing for the access patterns that are observed on FAT32, which unfortunately can lead to worst-case access patterns when using ext3 or other Linux file systems. In particular, the following (mis-)optimizations are commonly seen on flash media:

1) Most allocation groups are in linear write mode by default, only the first one or two allocation groups allow efficient write patterns in smaller units at all times. Since the FAT is known to be in the beginning of the partition, the controller only needs to allow writing small updates there, while it can expect other parts of the medium to be used for large image or video files.

[ See the URL page linked above for the remaining items ]

- Ben
http://www.fysnet.net/the_universal_serial_bus.htm

[Octocontrabass]

Remember that in the example I give, everything including the third sector to the end of the "disk" is nulls, which includes the root directory, which means the whole directory is empty.

Is the directory empty even when you read it with INT 0x13?

[BenLunt]

Remember that in the example I give, everything including the third sector to the end of the "disk" is nulls, which includes the root directory, which means the whole directory is empty.

Is the directory empty even when you read it with INT 0x13?

Surprisingly, no.

When I read the root directory, there is a Long File Name compatible entry named: "System Volume Information" (SFN: "SYSTEM~1")
It has an attribute of 0x16 (Sub, Sys, Hidden), the current date and time (in real time), and a cluster number of 0xE8.
Can anyone say 'Windows'?

It looks like the emulation actually stores some information about the volume in its own file system, all cached within the BIOS.

Interesting. I hadn't intended to go this far into this issue.

Ben

[Octocontrabass]

Are you sure that entry is being added by your firmware and not by Windows?

[BenLunt]

Are you sure that entry is being added by your firmware and not by Windows?

Well, since it is booting from the USB, my code is actually running, I can't image it being Windows. However, I have learned in my many years to not be sure of everything. Always question everything?

I pulled the ribbon from the (Windows XP formatted) hard drive and booted the USB again. Same results. It must be the firmware.

With that said, is it the firmware of the BIOS on the Host machine, or is it the firmware on the Sandisk Thumb drive? It could be either.

Either way, my point stands. If you don't follow specific rules when booting from the USB drive, your code/data will be changed, and this is proof.

Thanks,
Ben

[Octocontrabass]

Are you sure that directory doesn't exist on the disk when you read it afterwards?

[BenLunt]

Are you sure that directory doesn't exist on the disk when you read it afterwards?

Do you mean, was it there (physically) at boot time? Yes, I am sure that it was not there at boot time.

Do you mean, did the BIOS or USB Disk firmware (physically) write it to the media after boot? No, I did not check if it had.

If it did, then it physically modified the media, which in my opinion, is a complete No-No. In my opinion, the firmware does not have the right to modify the data on my volume. Though this is just my opinion.

Ben
- http://www.fysnet.net/osdesign_book_series.htm

[Octocontrabass]

I don't think the firmware is modifying the contents of the disk.

I do think Windows is modifying the contents of the disk. You do most of your development work on Windows, right?

[BenLunt]

I don't think the firmware is modifying the contents of the disk.

I do think Windows is modifying the contents of the disk. You do most of your development work on Windows, right?

Hi guys, sorry for the late reply.

Every once in a while, I come back and think about the issue shown in this thread. The other day, the thought come to mind that maybe Windows was writing the "System Volume Information" to the thumb drive at the short time between the time Rufus releases the drive and I pull the drive from the USB socket on my host machine.

As far as I know, Rufus does not restore the USB drive so that Windows can access it. The indicator light on the thumb drive itself goes out, the instant Rufus is done with it, and it never comes back on. I also don't get the usual "beep beep" when you unplug a USB device.

However, there is a slight chance that Windows is doing just that. Maybe one of the service calls Rufus makes, creates the "System Volume Information".

The way to check would to be to remove the thumb drive, then insert in another non-Windows host and read the volume back. However, this comes back to the reason why I posted this thread to begin with. How do I know it isn't the BIOS doing this.

Anyway, the reason I posted and the point I would like to get across is that if you have a non-FAT compliant file system on the thumb drive, in one way or another, the sector data--either physically read as is from the media or physically read and then modified in memory--when executed, may not be what you thought it was. If memory serves, I spent some time trying to figure out what was wrong with my code, when it was not me at all.

If you boot your OS from a USB thumb drive, it is (IMHO) highly recommended to have a FAT compliant file system and/or MBR, so that somewhere from the time you write your code to the thumb drive to the point of execution, something (BIOS, Host OS, other) doesn't modify it.

Anyway, again, I apologize for the late reply to this thread. I just seem to not have the time/interest I use to to play with this hobby. With that being said, I do want to express my appreciation to those that do spend time here helping others. Way long ago, I was in their shoes and appreciated the help.

Ben
- http://www.fysnet.net/osdesign_book_series.htm