Migrating OS from floppy to USB flash and USB hard drives

[DavidCooper]

Up until now I've done everything with floppy disks, but the small capacities are becoming highly limiting when it comes to data storage (even though there's still more than enough room for the OS itself) and I need to move on to something better. Because I'll need to go on using the BIOS for loading and saving for a while yet, I have to be sure that what I want to do will actually turn out to be possible. I don't want to use the hard drive in the machine as I wan't to keep Windows on there. I'm looking at MicroSD cards with a simple USB reader as the ideal way forward as this would hopefully allow 16GB cards to be switched like floppy disks without confusing the BIOS, and this matters because the tools I use to write my OS are all part of my OS: I've always worked by booting my OS, modifying it, changing disk, and saving the new version, so if the new version doesn't boot (this occasionally happens), I can simply boot up the old version and use it to load and modify the new version until it does boot properly. I also have a long series of previous versions sitting around on other disks which I can go back to if necessary to fix anything that's become corrupted and gone unnoticed for a long time. I want to go on working the same way, though a viable alternative may simply be to use ordinary USB flash drives (or USB hard drives) and have multiple versions of my OS stored on each (in a single partition) with a multiboot option in my own MBR which allows different versions to be booted, though that would have the little disadvantage of needing to be backed up regularly within Windows using dd in order to get a copy into safe storage on a different device. I've decided to go for the latter option first and to try out MicroSD cards later on where I can simply retain the option of booting different versions of my OS on each card too (which I might as well do as it'll only take up about 15MB for ten versions - each one living in 1.44MB of space and being possible to copy to a floppy disk at any time such that it'll always still be able to boot from one of those on an ancient machine).

One of the barriers to me doing this is getting my code from floppy disk to a USB flash drive in the first place, written in the right place on it, and done safely (without damaging anything on the machine I work on). Rawwrite makes the first step of that easy - I can make bits and pieces on floppy disks and get them into Windows as binary files without difficulty. The tool I need for the second half of the process appears to be dd, but the instructions that I've managed to find for using it are not sufficient for me to be confident about what I'm doing (and it isn't clear to me that the commands are going to be exactly the same for different versions of dd either). I can't afford to wipe out any drives by making mistakes. Even more importantly, I want other people to be able to get my OS onto a flash drive too after downloading the image, and that means that people who may know even less than I do are going to have to be able to do it safely too.

Anyway, there are a few questions that need to be resolved, so if anyone thinks they can help out, please do:-

1. I've read a lot of confusing stuff about how the BIOS boots up a machine, and one of those confusions relates to partitions. I've heard more than once that the BIOS will look for the bootable partition, load its boot sector into 7c00 and then treat that partition as if it's a drive in its own right, so that makes it sound as if you would access the second sector of that partition by treating it as the second sector of a drive, but that appears to conflict with how MBRs are supposed to work so I think it must be wrong, but I need to know for certain - it could only be right if the MBR code was completely sidelined by a BIOS, because if the MBR is allowed to run, its code must be able to access the entire drive, and when it loads in the boot sector from the right partition, there's no way the BIOS could recognise the point when that boot sector takes over from the MBR to start loading further sectors from within its partition - the BIOS isn't suddenly going to switch the way it works to restrict loads and saves to within the partition and with the first sector of that partition suddenly becoming LBA 0 (or sector 1 of track 0, head 0). So, what I need to know is, is there any truth in the idea that a BIOS might make a partition behave as if it's a drive or can I just assume that the BIOS always gives access to the entire drive?

2. How important is it for the CHS values to match up with the LBA values in the MBR's partition table? I've had a look at the hard drive in the machine I usually run my OS on, and what I've found is this: two entries, the first partition (MSDOS5.0) starting at the 1MB point and stopping 1MB short of the second partition (NTFS) if you go by the LBA values, but there is no such gap between the two partitions if you go by the CHS values. Is there some reason why the two systems shouldn't always match up (obviously not counting the point where CHS hits the FE FF FF limit and can't go any higher anyway)?

3. When copying the image of my OS to a flash drive, the image is only going to be about 16MB at the most (although it could be a lot smaller in the downloadable-from-Web version as it only really needs to hold a single copy of the 1.44MB OS - the other nine copies can be generated from that later on so that the multiboot options won't cause a crash if the wrong number's typed in when booting [while the copies will diverge later still as the user modifies them, always keeping a history of older versions in case they make a mistake at some point]), but the important point here is that the image isn't actually going to contain any part of the second partition specified in the MBR - the image will only hold everything from the MBR to the end of the first partition (or just to the end of the first copy of the OS within that first partition). Copying that to the flash drive should be easy enough using dd (provided that the command is formulated correctly), but I don't know how easy it would then be to use the rest of the disk. Assuming that the second partition is defined correctly in the MBR, is it simply a matter of asking Windows to format the second partition as FAT32, or is it more awkward than that? Also, could Windows mess up my MBR in the process, or would it limit itself to changing the partition type byte in the partition table? (That second question isn't so important as the MBR can simply be written to the disk again afterwards if it gets corrupted.)

[The size of the drive that my OS could be copied to could obviously come from a limitless range of sizes, so if someone follows simple instructions to copy the disk image onto a drive, the end location stated in the MBR's partition table for the second partition is invariably going to be wrong, but it can be adjusted afterwards from within my OS to fit the actual size of the drive, so that won't be a problem, just so long as the 2nd partition can be formatted later on as FAT32 within Windows.]

4. When building my OS for the first time on a flash drive, I'm going to need to copy the floppy image into the first partition location (with my OS modified to boot using LBA if DL!=0 and from the place where the first partition starts), and then I'm going to have to copy the MBR in from another floppy image containing just that one sector, so I need two versions of dd commands for these things, and they have to work with a Windows version of dd. I have seen a command for doing this kind of thing somewhere, but it had a great long serial number as part of it, presumably to identify the flash drive, and I don't know how you're supposed to get hold of that number if it's required. For the MBR part of it, dd if=mbr.bin of=???? bs=512 count=1 should be on the right lines, but I don't know what to put in place of the ????. Can anyone suggest what the ???? should be. For the OS part, I'd need something similar with a seek=???? added to it, but I think I can probably work that out myself. Once I've built my OS on the flash drive and got it working, I'll be able to copy multiple versions of it through the rest of the first partition from within my OS and I can work on getting the MBR to boot them all (pressing a number key to select any one of ten versions - each one taking up 1.44MB of space), so the other things I'll need to be able to do with dd should be simple enough as it's just a matter of copying the first 16MB of the drive into a file using dd, and then it can be copied back onto any drive using dd again with "if" and "of" switched over.

5. Where should the first partition start? Are there any limitations on where it should be, and how much space is it worth leaving in front of it? I could certainly use an extra sector or two for the multiboot code and could probably store MBR copies in the gap too, but I don't know if I want to leave a megabyte of empty space.

6. On another issue (not USB drives, but internal hard disk), I notice that there's a 3GB gap at the end of the hard drive in my machine, so I'm wondering if that's used by Windows XP for hibernating or some other vital purpose. Does anyone know if that's what this space is for? Given that the machine only has 1GB of RAM in it at the moment, I'm just wondering if it might be possible to put my OS into the end of that space without getting in the way of Windows, and whether it would be safe to mess with the MBR in order to make the machine multiboot for this? It looks as if it should be easy enough to chainload the original MBR from a new location in the first gap to boot Windows in the normal way, but I'd like to know if anyone's out there has done that. One complication may be how the system boots up if it's been hibernated, because the BIOS only boots my OS from USB floppy if the machine was not hibernated when it last closed Windows - if it was hibernated it simply ignores my OS and unhibernates Windows regardless.

The only reason I'm thinking about putting my OS on the internal hard drive at all is that the BIOS doesn't appear to notice a second USB drive plugged in at the same time, so I have no means to copy from one USB device to another while using my OS - having a version of it on the hard drive would give me more backup options without having to run Windows. Even so, I probably won't do this, but it would be interesting to know if it's a viable option.

[Rudster816]

1. There is no truth to that. What the BIOS does is looks at the very first sector of the first drive in the boot order, checks if the last two bytes are 0xAA55, if it is, it loads it at 0x7C00 and jumps to it. On a IBM style partitioned disk, this will be the MBR, which will check all the partitions and look for one that's 'Active', and load the volumes first sector at 0x7C00 (after moving itself out of the way of course) and jump to it the same way the BIOS does. VBR's (Volume Boot Records) are agnostic to rather or not they are loaded by the BIOS or an MBR because of this, so they will work on a partitioned disk as a VBR, or as an MBR in an unpartioned disk.

2. They should line up, but CHS has been obsolete for so long Microsoft might have re purposed any CHS values (I'm not aware of what you're talking about specifically). I know their Win7\Vista NTFS VBR does not support loading from HDD's that do not support LBA, so I wouldn't be surprised if they completely got rid of anything related to CHS for anything created by Vista\7. I think you'd have a hard enough time find a machine without LBA that supported XP, let alone Vista or 7.

3. I don't see any reason Windows would ignore the partition table. If it ends up screwing stuff up and you're unable to create a second partition with Windows's built in stuff, I'm sure a third party utility would be able to do it. I don't foresee this being a problem though. What I would expect is Windows to list a partition in Disk Manager that's 1.44MB as 'Unknown', and the rest of the space on the flash drive as unallocated. You should be able to format this entire unallocated area with a FAT32\etc partition.

4. I'm pretty useless when it comes to DD, but I can help you test it. Download the hex editor HxD, you can use it to see the physical sectors on your HDD's\Flash drives, and even write to them if you wish. It will make guessing and checking easy if it comes down to that. You'll also be able to easily see exactly what Windows changes when it adds your FAT32 partition.

5. I know FAT partitions typically leave 38 'reserved' sectors for various reasons. You're free to not use any space, but you'll probably see it as unallocated in Windows' disk manager with the option to format it. I think the safest thing to do is see what Windows does and just do that.

6. I've never heard of such a large gap at the end of a HDD formatted by Windows. I know the XP installer sometimes left 8MB off on my disks. Windows Vista optionally partitions a 100MB "boot partition" if you have decided to encrypt your disk with BitLocker. Windows 7 will do this regardless of rather or not you're using BitLocker or not. Windows stores the ramdisk in the file C:\hiberfil.sys (or whatever drive letter applies). Might be be exactly hiberfil.sys, but its close to that. If your system is like my laptop, the BIOS seems fully aware that Windows hibernated the computer. When I boot up my laptop after it hibernates, I cannot open the BIOS menu afterwards. I'm not sure if it would boot from another device, but I doubt it. My guess is that this is some extra (aka useless) security feature.

So long as you have a Windows disk handy that will be able to repair the MBR, you can modify it to your hearts content. Just back it up manually first, because it will contain the partition data. Windows will typically have several backup MBRs on various parts of the disk. It should be trivial to boot up from a Windows DVD and repair it. The easiest way is to probably use the recovery console if thats available (I haven't had to use it since XP, so I don't know if it exists on Win7\Vista disks). I know there is a command like 'fixmbr'. If you don't have a Windows disk, back up your MBR on a flash drive, and have a Linux live CD handy. Even better, just edit the .iso you have (or the one you download) to add your mbr as a binary file on it, then burn it. Either way, make sure you can boot from it and restore the MBR before you smash your existing one.





My 2 cents: Support FAT32 from your OS. It's the de-facto standard filesystem for practically all removable media, so its not like you would be doing it just to make development easier.

[DavidCooper]

Thanks for answering all of that, Rudster816. Just a few things left to clear up:-

VBR's (Volume Boot Records) are agnostic to rather [/whether] or not they are loaded by the BIOS or an MBR because of this, so they will work on a partitioned disk as a VBR, or as an MBR in an unpartioned disk.

1. If an OS is installed in different places on disk in different machines, how does the VBR normally know which sector it's been stored in in order to know where it can find the OS it's supposed to load? I've never heard anything about the MBR passing the VBR's LBA to it in any way, or the VBR looking up the MBR to read the partition table (which isn't guranteed to have been relocated to 600h), but the only other option appears to be to code the LBA directly into the VBR to match its location on the disk. Until the other day, I had assumed that the VBR and OS would be in a form that would be no different wherever they were stored on the disk, and what you've said above seems to agree with that, so something doesn't quite fit.

[Edit: resolved now, I think - see the post below this.]

[My plan, by the way, is for my MBR to pass the LBA of the relevant boot sector to it in registers, so the code in the boot sectors (which I'm not calling VBRs because most of them won't be at the start of an official partition) will not need to change according to where it's stored.]

4. I've downloaded HxD to try it out (which I'll do once I've posted this) - it looks as if it's a lot more user-friendly than dd, so it may well be a better road to go down.

6. Partition table of my hard drive (only two entries):-

0 20 21 0,
12 92 54 7e, (CHS end-location of 1st partition)
0 8 0 0,
0 c0 5d 0; (LBA end-location of 1st partition)

80 92 55 7e, (CHS start-location of 2nd partition: one sector after end of 1st partition)
7 fe ff ff,
0 c8 5d 0, (LBA start-location of 2nd partition: 8*256 sectors after end of 1st partition)
0 c8 43 12. (as compared with size of drive b0 9e a1 12)

By using Int 13h AH=48h (Extended Read Drive Param.s) the drive size comes out as b0 9e a1 12 (which is the expected 160GB, taking a GB as 1000*1000*KB). I make that 3GB of unused space after the 2nd partition. I'll have to explore it to see if there's anything in it because it seems like an awful lot of wasted space. Maybe it's riddled with bad sectors.

As for the existing MBR, I can copy that off to a floppy without difficulty using my OS and put it back again, so if that 3GB space really is unused and undamaged, I'm now a lot more tempted to try putting my OS in it (and perhaps with a new following FAT32 partition as well).

[Edit: I misinterpreted the way the partition table works and it turns out that there is no extra space at the end - it isn't the LBA of the end of the partition that's listed in the table, but the size of the partition. That explains both anomalies! Not enough room to bother putting my OS on there then after all, but maybe it's still worth replacing the MBR to check for another drive with an operating system on it and to boot that instead if one is found, thereby overriding any unhibernate signal (probably in CMOS RAM) which may have prevented the BIOS from choosing to boot that operating system. If the mystery signal isn't affected by running that OS, the machine may still be able to unhibernate the next time.]

My 2 cents: Support FAT32 from your OS. It's the de-facto standard filesystem for practically all removable media, so its not like you would be doing it just to make development easier.

I will do. I want to be able to store BMP photos and other data in the FAT32 partition and access them from my OS, but I want to prepare the way for that by getting my OS in a bootable partition of my own format and with a FAT32 partition in place on the same drive which Windows can write files to. I'll then write code to read the FAT32 partition before I try writing code to write to it or to create new FAT32 partitions from scratch.

7. New question relating to the chrysocome.net/dd page:-

--size will tell dd to figure out the size of the device and make sure it does not read past that point. This is important for USB sticks which stop working if you read past the end of them. This is not on by default because getting the correct size of the device is not always possible. Some devices also keep returning bogus data past the end of the device without returning a suitable error code

Does anyone know if this is referring to a few specific USB sticks or if it's something that applies to USB sticks generally?

[DavidCooper]

1. Ah, found it at last in the wiki! (So many different pages on the boot process, and all my previous searches failed to find the right one.) http://wiki.osdev.org/System_Initialization_%28x86%29:-

The function of the MBR Bootstrap code is (usually) to search the Partition Table for the partition marked with an "active" flag (flag byte equals 0x80), then load and run the bootsector (that is, the OS-specific Bootloader) of that partition (the very first sector). This is what "generic" MBR Bootstraps do, such as the one that FDISK writes to a newly partitioned disk. The bootstrap code is expected to pass a pointer (in DS:SI) to a memory copy of the booted partition table entry, to the OS-specific bootloader -- as well as reproduce the above System "Environment" that the OS-specific bootloader expects to see.

So that's how the VBR gets the LBA from the MBR to know where to load in the OS from.

[Edit: Found it in the MBR page too now - "Note: it is intended for the value of DL, and the DS:SI pointer to be passed all the way into the kernel, for the kernel's use. This also means that the relocated MBR should not be overwritten during the boot process -- because the DS:SI pointer is aimed at a partition table entry inside that MBR, and needs to remain valid."]

[DavidCooper]

I think I'm close to getting the full picture now when it comes to how operating systems are booted from hard drives, but there's one thing that's come up a number of times in the past which brings up another question, and I think I may be answering it as I write this, so I'll keep going as it may help others to get the full picture too. If you don't partition the disk but simply stick floppy disk code right at the start of it, it appears to act like a floppy disk, and this has come up a number of times in threads with people having trouble booting from USB devices. However, there was talk of having to change some BIOS setting to accommodate this so that the BIOS can emulate a floppy disk, and that would be necessary if the code expects to be using drive 0 and expects there to be only 18 sectors per track.

One person informed me that if you store your OS at the start of the device it will behave like a floppy disk, whereas if you store it at the start of a partition, it behaves like a hard disk, but clearly it isn't that simple. If you put it at the start of a partition it will fail to load the next sector(s) as it won't know where to find them unless it has been written to look up its partition details at DS:SI and to use the data found there to work out where it is located on the disk. Alternatively, if you put it at the start of the drive, it will fail to work properly unless the BIOS is switched into a mode where it emulates a floppy drive. (Whether it then also allows you to go beyond track 79 all the way up to 1024 is another question, but I don't suppose it matters as it isn't really a route that anyone's likely to want to go down.) There must be a third option though, and that is to stick your code at the start of the device without having any partitions on it and using CHS or LBA to access the entire disk (if CHS has sufficient range), and the BIOS would be in non-floppy-emulation mode. I got the impression that some BIOSes would automatically treat a drive with no partitions on it as a floppy disk, so that may block this third option from working, but that doesn't seem too likely now: I suspect that information was wrong and that you always have to set something manually in the BIOS to enable or disable floppy emulation.

[egos]

1. This legend exists since new too intellectual/stupid BIOSes were manufactured. I will risk to assume how it would work. When BIOS option "USB-HDD" is not enabled but BIOS found MBR instead VBR, it would hide all sectors of the drive except active partition and correct (virtually) its VBR field HiddenSectors. I use only option "USB-HDD" enabled and while I did not observe that my MBR boot loader was left.

2. Forget about CHS BIOS geometry (naturally we don't mean a floppy) stored on the drive. Always translate LBA to CHS using actual SPT and Heads parameters returned by BIOS (after function 8 was called). Naturally if it is required, i.e. when EDD reading function is not supported for the drive at all.

3. No. Windows works bad with more than one partition on the flash drive. It is strange. But you can organize and use free space beyond the partition for your purposes.

4. You can use any suitable tools but they should use locking/dismounting technique under Windows Vista/Seven.

5. Anywhere. In general a flash drive should have following structure: mbr [zzz] partition [zzz]

6. I'm using dual booting everytime few years. For my OS I use exactly the small gap at the end of hard disk (if there is no FAT partition anywhere on the disk). You had asked good question about hibernation. Still now I have no problems with it. I promise to take some experiments with this matter tomorrow.

The main question for me is do you have the drivers for USB controllers/USB mass storage devices?

The easiest way is to probably use the recovery console if thats available (I haven't had to use it since XP, so I don't know if it exists on Win7\Vista disks). I know there is a command like 'fixmbr'.

Search for "bootrec /fixmbr" and "bootsect /mbr".

If an OS is installed in different places on disk in different machines, how does the VBR normally know which sector it's been stored in in order to know where it can find the OS it's supposed to load?

Search for "BPB HiddenSectors field". Pointer to descriptor (PT record) of boot partition (DS:SI) is a little reliable. Moreover it's not passed by BIOS if you use "super floppy" structure (without MBR) on the disk. So stage 1 boot loaders usually do not use this parameter. In my Boot Spec. is described special signature to guarantee correctness of this parameter (byte at 7DFFh = 88h) for that reason.

[Yoda]

DavidCooper,
"Superfloppy" format that you heard about does nothing with number of sectors or tracks. It just means that the drive is non-partitioned and whole drive as treated as one big partition. That is, for partitioned drives first sector has a partion structure of the disk where specified the starts and lengths of the partitions on the disk layout. In "superfloppy" format there is no partition structure in the first sector, instead it contains the first sector of an appropriate file system.
For BIOS options USB-HDD and USB-FDD means the following.
1. In USB-HDD mode BIOS usually passes 80h drive number in DL, while in USB-FDD mode passes 0.
2. In USB-FDD mode BIOS patches BPB area in the first sector for FAT file systems to store the boot drive number in it also.
For well organized MBR and VBR boot code it doesn't matter which format is used. It should boot in any case.

Note! I've seen an old buggy BIOSes that incorrectly patch BPB area in superfloppy FAT32 format. They assume that there is FAT16 file system and store 0 (floppy boot drive) in first byte of Sectors-Per-FAT field of FAT32 header, spoiling it.

If you aim to boot from FAT32 flash drive and make it bootable under Windows without spoiling data, I suggest to try my boot tools (see the link in my signature).

[stevenson]

he bootstrap code is expected to pass a pointer (in DS:SI) to a memory copy of the booted partition table entry, to the OS-specific bootloader

Not true, at least for vast majority of computers. In best case scenario filesystem is expected to tell where it starts (FAT has "hiddenSectors" field). In worst case you need to examine partition tables yourself, and optionally read beginning of partition from disk and verify that its yours.

[DavidCooper]

Thanks egos for all that info.

1. This legend exists since new too intellectual/stupid BIOSes were manufactured. I will risk to assume how it would work. When BIOS option "USB-HDD" is not enabled but BIOS found MBR instead VBR, it would hide all sectors of the drive except active partition and correct (virtually) its VBR field HiddenSectors. I use only option "USB-HDD" enabled and while I did not observe that my MBR boot loader was left.

So it applies to a fourth possible case where there is an MBR but the BIOS is in the wrong mode for it. That means we have four cases which a OS should ideally be able to handle (ignoring optical drives):-

(A) If it's in USB-FDD mode and the OS is written right at the start of the disk without an MBR, the drive number will be 0 and the OS will assume that there are 18 sectors per track. It will only be possible for the OS to work out that it isn't actually on a floppy disk by exploring tracks far beyond track 83 (decimal).

(B) If it's in USB-HDD mode and the OS is in a partition with an MBR pointing to it from the start of the disk, the OS will see that it's drive 80h (or above) and work out where it's been stored on the disk by looking at the partition table entry pointed at by DS:SI (which should be set correctly by the MBR).

(C) If it's in USB-HDD mode and the OS is written right at the start of the disk, the drive number will tell the OS that it's on a hard drive, but DS:SI would only be set correctly for it if the BIOS generated a partition table entry for the OS to look up (with that table telling it that its partition is the whole disk). You say that BIOSes don't do that, so that makes it problematic: it sounds as if you've solved this by getting your MBR to modify the last byte of your VBR after loading it in, so if it isn't modified it tells the VBR that it's been stored in the first sector of the disk, though that isn't going to work if the wrong MBR is used. An alternative way to do it would be for the VBR to load the first sector of the disk in again and to check to see if it is the VBR, and that check would need to be made in case (B) too.

(D) If it's in USB-FDD mode and there's an MBR with the OS in a partition, the drive number will be 0 and the BIOS will act as if there are 18 sectors per track while also hiding every part of the disk that isn't in that partition, the result being that the OS will be completely unable to tell that it is not running in case (A). What happens though if the BIOS doesn't perform this trick of hiding the rest of the disk from the OS? Maybe it always does, but if not, the MBR's code would have to check to see if it's been loaded from drive 0 and then it would have to work out where the OS's VBR is by recalculating it's location on the basis that there are 18 sectors per track, and once the VBR starts running it will have to go through all that again, loading in the first sector on the drive again to see if it's an MBR so that it too can work out where it (the VBR) is located on the disk, and then it would have to do all its loading and saving on the basis of 18 sectors per track as well... Complicated! I hope it never has to do that.

I'm going to make sure my OS works in cases (A) and (B), and it should only be put on floppy disks or into a partition on a flash drive or hard drive, so case (C) won't normally apply. Case (D) should hopefully always behave like case (A), though the user will have to be told that their BIOS is probably in the wrong mode when the ability to access the 2nd partition doesn't appear.

3. No. Windows works bad with more than one partition on the flash drive. It is strange. But you can organize and use free space beyond the partition for your purposes.

I might be better off working out how to format FAT32 partitions for myself and get my OS to generate one of the right size to fill the rest of the drive with it. I hope Windows doesn't also have difficulty reading from and writing to a 2nd partition once it's set up.

6. I'm using dual booting everytime few years. For my OS I use exactly the small gap at the end of hard disk (if there is no FAT partition anywhere on the disk). You had asked good question about hibernation. Still now I have no problems with it. I promise to take some experiments with this matter tomorrow.

It turns out that I have 1.88MB of space left at the end, so I could put a floppy disk sized partition into that with my OS in it and make the machine multiboot, though there isn't a lot of point in doing so until it can read the NTFS partition, and that's a long way down my to-do list.

The main question for me is do you have the drivers for USB controllers/USB mass storage devices?

No - I'm going to be using the BIOS for loading and saving for a good while yet.

[DavidCooper]

Thanks too to Yoda - I used some of your information for my post above.

2. In USB-FDD mode BIOS patches BPB area in the first sector for FAT file systems to store the boot drive number in it also.

Note! I've seen an old buggy BIOSes that incorrectly patch BPB area in superfloppy FAT32 format. They assume that there is FAT16 file system and store 0 (floppy boot drive) in first byte of Sectors-Per-FAT field of FAT32 header, spoiling it.

So I should probably just assume that it could try to do it with something that isn't even a FAT system. I assume the above applies to the case where the VBR is in the first sector of the disk.

If you aim to boot from FAT32 flash drive and make it bootable under Windows without spoiling data, I suggest to try my boot tools (see the link in my signature).

I've read about your boot tools and I'd certainly rather use them than GRUB, but I'm keen to continue following my own path at the moment.

[DavidCooper]

Hi Stevenson,

The bootstrap code is expected to pass a pointer (in DS:SI) to a memory copy of the booted partition table entry, to the OS-specific bootloader

Not true, at least for vast majority of computers. In best case scenario filesystem is expected to tell where it starts (FAT has "hiddenSectors" field). In worst case you need to examine partition tables yourself, and optionally read beginning of partition from disk and verify that its yours.

Does that mean that DS:SI can't be relied on at all, or is the problem just with case (C) as labelled in my post a couple of posts up from here? It looks as if you would have to load the first sector in again to check it just to be absolutely sure, but it's only going to matter if your OS is stored without your own MBR. Then again though, with a customised MBR, if someone decides to replace it to make the machine multiboot for another operating system, your signal disappears and the problem returns, so yes: if you're going to cover all bases you really should load the first sector back in to do that irritating check.

[Yoda]

(A) If it's in USB-FDD mode and the OS is written right at the start of the disk without an MBR, the drive number will be 0

Right.

and the OS will assume that there are 18 sectors per track.

Wrong. The actual values for sectors per track and number of heads will be that filled in BPB area in boot sector. But LBA access works well in USB-FDD mode in modern BIOSes, so you don't need to bother about CHS values.

the OS will see that it's drive 80h (or above) and work out where it's been stored on the disk by looking at the partition table entry pointed at by DS:SI (which should be set correctly by the MBR).

Actually, there is no obligation for MBR to pass any values to volume boot code. It depends on a particular file system and MBR/VBR boot code. As for FATs, there is a special field BPB_HiddSec in BPB area of boot sector which indicate the number of hidden sectors preceding current partition. FAT boot code may rely on this value to detect actual start of it's partition.

If it's in USB-FDD mode and there's an MBR with the OS in a partition, the drive number will be 0 and the BIOS will act as if there are 18 sectors per track

No. Forget about 18 SPT.

the result being that the OS will be completely unable to tell that it is not running in case (A)

For correct boot code the result will be OK, since BIOS passes drive number in DL and the boot code will use it for disk access instead of that is written in BPB.

What happens though if the BIOS doesn't perform this trick of hiding the rest of the disk from the OS?

BIOS doesn't hide anything in any mode from the OS.

the MBR's code would have to check to see if it's been loaded from drive 0 and then it would have to work out where the OS's VBR is by recalculating it's location on the basis that there are 18 sectors per track

Again, MBR should not recalculate anything and to check anything. It just uses input value in DL register as the boot disk number. After that it should load in LBA access the first sector of active partition to 7C00h, load DL register with the number of disk and pass control to VBR. That's all MBR is obliged to do. VBR code then uses value in DL the same way for further loading. For FAT32 no other input information is needed.

So I should probably just assume that it could try to do it with something that isn't even a FAT system.

For non-FAT/NTFS file systems there is only one problem (for partitioned disks, not superfloppy) - obtaining number of hidden sectors, i.e. sectors, preceding the partition. AFAIK, still there is no common reliable standard for passing parameters from MBR to VBR. So the reliable way would be to fill in the number of hidden sectors somewhere in you boot record.

Does that mean that DS:SI can't be relied on

Yes, DS:SI can't be relied on. Reading first sector and analyzing partition structure is also unreliable way because:
1. There may be multiple similar partitions. How will you determine which of them you are booted from? Even if you check the "active partition" flag, don't forget about boot managers that may boot from other partitions.
2. Your file system may reside in extended partition, so analyzing the EBR chain may add additional complication to analasys.
3. Your file system may reside on GUID partition, so you may encounter the need to analyze GUID partition table.
... and so on.

[DavidCooper]

Thanks again Yoda for clearing up lots of things.

and the OS will assume that there are 18 sectors per track.

Wrong. The actual values for sectors per track and number of heads will be that filled in BPB area in boot sector. But LBA access works well in USB-FDD mode in modern BIOSes, so you don't need to bother about CHS values.

My starting point for this is the idea that you might initially have a primitive operating system designed to run on a floppy disk and that this primitive operating system always expects there to be 18 sectors per track. If you copy that OS onto the start of a flash drive and run it in USB-FDD mode, I was assuming it would still work, but when the OS tries to access the first sector on the other side of the first track it will actually get the 64th sector on the drive instead. Of course, there are other floppies out there with 9 or 36 sectors per track, so it makes sense for USB-FDD mode not to assume 18 sectors per track, and that means that a primitive operating system which works fine on a floppy disk while assuming 18 sectors per track is automatically going to fail to work in USB-FDD mode (unless it only uses the first 18 tracks on the disk). This is something that people who are just starting out ought to know so that they can do things the right way from the very start.

So far as I can see, there are only two ways this can work then. (1) The OS must have a BPB, the BIOS has to write the actual number of sectors per track into the BPB on loading, and the OS then uses that value to calculate how to ask the BIOS for the sectors it wants. Or: (2) The number of sectors per track must be put into the BPB (or stored in some other way in the OS) when the OS is originally written into the partition, thereby modifying it so that it can load itself correctly in future from that location on that device. Which of these two is correct?

Actually, there is no obligation for MBR to pass any values to volume boot code. It depends on a particular file system and MBR/VBR boot code. As for FATs, there is a special field BPB_HiddSec in BPB area of boot sector which indicate the number of hidden sectors preceding current partition. FAT boot code may rely on this value to detect actual start of it's partition.

So I'm guessing that that value would have to be coded into it in advance, when the OS is originally written into that partition (unless the BIOS writes it in, but that would be hard for it to do as it would either have to guess what the MBR's going to do or fill in appropriate values in the BPB of every VBR in every bootable partition).

If it's in USB-FDD mode and there's an MBR with the OS in a partition, the drive number will be 0 and the BIOS will act as if there are 18 sectors per track

No. Forget about 18 SPT.

Okay - so again the primitive OS which just assumes it's on a floppy disk with 18 sectors per track will fail to load correctly in this situation.

What happens though if the BIOS doesn't perform this trick of hiding the rest of the disk from the OS?

BIOS doesn't hide anything in any mode from the OS.

So there is no case where the BIOS will ever present a single partition to the operating system as if that partition is the entire drive. That myth can be forgotten.

the MBR's code would have to check to see if it's been loaded from drive 0 and then it would have to work out where the OS's VBR is by recalculating it's location on the basis that there are 18 sectors per track

Again, MBR should not recalculate anything and to check anything. It just uses input value in DL register as the boot disk number. After that it should load in LBA access the first sector of active partition to 7C00h, load DL register with the number of disk and pass control to VBR. That's all MBR is obliged to do. VBR code then uses value in DL the same way for further loading. For FAT32 no other input information is needed.

So that means that if there's an MBR and the machine's in USB-FDD mode, it will behave exactly like USB-HDD mode except for perhaps giving it the drive number 0. That makes things simple

So I should probably just assume that it could try to do it with something that isn't even a FAT system.

For non-FAT/NTFS file systems there is only one problem (for partitioned disks, not superfloppy) - obtaining number of hidden sectors, i.e. sectors, preceding the partition. AFAIK, still there is no common reliable standard for passing parameters from MBR to VBR. So the reliable way would be to fill in the number of hidden sectors somewhere in you boot record.

So to be completely safe, it should have to be written in some way into the OS code when the OS is stored in a partition right at the outset.

Does that mean that DS:SI can't be relied on

Yes, DS:SI can't be relied on. Reading first sector and analyzing partition structure is also unreliable way because:
1. There may be multiple similar partitions. How will you determine which of them you are booted from? Even if you check the "active partition" flag, don't forget about boot managers that may boot from other partitions.

I was visualising the VRB having a unique code in it which it could look for to see if it has loaded itself in, thereby detecting the case where the OS has been written at the start of the drive without the disk being partitioned. If the location is hard coded into the OS when it's originally written to the drive, that will not be necessary, but if you're relying on DS:SI and someone puts the OS at the start of the drive by mistake, looking up the first sector to see if it's the VBR would be a worthwhile check.

Okay, where does that leave us? Here are the four cases again with the latest corrections, and wherever there are any errors in them, the clearest way to deal with them might be to quote them with the errors in bold, and then print them again not in a quote with the faults corrected:-

(A) If it's in USB-FDD mode and the OS is written right at the start of the disk without an MBR, the OS will check the BPB to work out how to load itself. The value for sectors per track is written into the BPB by the BIOS / The value for sectors per track is written into the BPB right at the outset when the OS is originally written to the disk. [One of these options needs to be deleted, although cases where there is no BPB need to be explained if the former statement is true.]

(B) If it's in USB-HDD mode and the OS is in a partition with an MBR pointing to it from the start of the disk, the OS can perhaps work out where it's been stored on the disk by looking at the partition table entry pointed at by DS:SI, but that only works if the MBR sets it up correctly and indicates that it has done so, so it's safer to hard code the location in some way into the OS (e.g. in a BPB).

(C) If it's in USB-HDD mode and the OS is written right at the start of the disk, DS:SI will be no use. If the location is hard coded into the OS (e.g. in the BPB) the OS will load correctly.

(D) If it's in USB-FDD mode and there's an MBR with the OS in a partition, DS:SI can be used if the MBR signals that it is right, or the location can be hard coded into the OS. The OS will load exactly as in case (B).

There seems to be so little difference now between the USB-FDD and USB-HDD modes that I can't see why they bother providing two modes. What am I missing? Is it only the drive number that changes?

Anyway, my own plan is to put my OS in a partition and never at the start of a disk (unless it's on a floppy). My own MBR will be multiboot, offering to boot any partition that's labelled as active, but also offering to boot subpartitions within my own special partition(s) which will be indicated by a special coding outside the table, the start location being passed to the OS using DS:SI to point at a table entry for it stored in a gap sector. The MBR will modify a VBR byte after loading it, so if the VBR doesn't see the modified value in itself, it will assume it has been placed at the start of a disk. In the case where the modified byte is detected, DS:SI is practically guaranteed to be correct as there will be no need for anyone to tamper with the MBR. In the other cases where it is at the start of the drive, it can simply assume it is on a floppy disk because no one sane would want to put it at the start of a flash drive. The special partition my OS will go into will be a multiple of 1.44MB in size, and each floppy-disk sized chunk of it will be able to hold a floppy disk image which can be selected for booting by the MBR code. Any OS that fits on a floppy disk but which is also capable of booting from a hard drive can probably be modified to work in this setup, appearing in the multiboot menu for direct booting. A 64MB partition can for example hold 45 of these floppy disk images, and all 45 of those could appear in the multiboot menu, along with more OSes from other partitions. This will enable dozens or even hundreds of operating systems to be stored on a single flash drive, and with all of them being available for direct booting. It will also be possible to boot any one of them from within my own OS. If these OSes use DS:SI to work out their location, they'll boot fine no matter where they are put, but they'll need to use the same method of detecting that they're not at the start of a disk (the MBR changing a particular VBR byte to a certain value), or they'll have to be modified to assume that they will never be placed at the start of a disk. If they need a value posted into their BPB by the BIOS (or hard coded into them to state their position), I'll have to write some extra MBR code to do that same job (or their location can be manually programmed into them when they're installed). To get an OS into the special partition, its disk image can be saved to the FAT32 partition, then when my OS is booted up it can hunt for the disk image in the FAT32 partition and copy it into a subpartition in the special partition from which it can be booted. The original reason for designing a system for booting up many OSes in this way was to allow me to have many previous versions of my own OS still available for booting after I create new versions, but I can see now that it could be fun to have many other OSes on the same drive as well, and potentially useful in education too as it would keep things tidy and speed things up.

[Yoda]

(1) The OS must have a BPB, the BIOS has to write the actual number of sectors per track into the BPB on loading, and the OS then uses that value to calculate how to ask the BIOS for the sectors it wants.

Yet not. Will BIOS use/patch that values or won't greately depends on particular BIOS implementation. The ONLY correct way, if you really need to stick to CHS access, is first to read drive parameters: INT 13h function AH=8 (note that this call destroys DL value with number of hard disk drives, so for subsequent calls you need to reload it by drive number) then use the values passed to you by BIOS. Real BIOS may patch values in BPB (if recognizes it), may use values filled, may ignore them and use it's own translation scheme, but you have no reliable method to know, which method the BIOS have chosen. The only thing you can do is to know the results of chosen translation by INT 13h function 8.

So I'm guessing that that value would have to be coded into it in advance, when the OS is originally written into that partition (unless the BIOS writes it in, but that would be hard for it to do as it would either have to guess what the MBR's going to do or fill in appropriate values in the BPB of every VBR in every bootable partition).

BIOS and MBR never patch the Hidden Sectors field in BPB. BIOS because it always loads the very first sector and never VBR. MBR doesn't patch because there are multitude of file systems. For FATs/NTFS number of hidden sectors is placed at bytes 1Ch:1Fh, but this may not be true for other file systems. You need to fill this field on formatting file system or you may check/fix this field at any time later directly on disk.

No. Forget about 18 SPT.

Okay - so again the primitive OS which just assumes it's on a floppy disk with 18 sectors per track will fail to load correctly in this situation.

It will work if it will get drive parameters first.

So that means that if there's an MBR and the machine's in USB-FDD mode, it will behave exactly like USB-HDD mode except for perhaps giving it the drive number 0.

Yes.

For non-FAT/NTFS file systems there is only one problem (for partitioned disks, not superfloppy) - obtaining number of hidden sectors, i.e. sectors, preceding the partition... So the reliable way would be to fill in the number of hidden sectors somewhere in you boot record.

So to be completely safe, it should have to be written in some way into the OS code when the OS is stored in a partition right at the outset.

Not to the code, but to the field specially assigned for that purposes in file system (FATs/NTFS) or to the space in file system that known in advance to be absolutely free (other file systems).

I was visualising the VRB having a unique code in it which it could look for to see if it has loaded itself in, thereby detecting the case where the OS has been written at the start of the drive without the disk being partitioned. If the location is hard coded into the OS when it's originally written to the drive, that will not be necessary, but if you're relying on DS:SI and someone puts the OS at the start of the drive by mistake, looking up the first sector to see if it's the VBR would be a worthwhile check.

May that [unique for the drive] code be just 32-bit value of hidden sectors? For unpartitioned disk it will just store 0. Remember that 1) this is the simplest way and don't force you to analyze the disk structures and 2) you have only 448 bytes of code for FAT12/16 and hardly predicted size and location of boot code for FAT32. It is really not a simple task to fit code to that space.

(A) If it's in USB-FDD mode and the OS is written right at the start of the disk without an MBR, the OS will check the BPB to work out how to load itself. The value for sectors per track is written into the BPB by the BIOS / The value for sectors per track is written into the BPB right at the outset when the OS is originally written to the disk.

OS should not check BPB. OS should first try LBA access, then [if fail] check INT 13h/fun 8 for CHS access.

(B) If it's in USB-HDD mode and the OS is in a partition with an MBR pointing to it from the start of the disk, the OS can perhaps work out where it's been stored on the disk by looking at the partition table entry pointed at by DS:SI, but that only works if the MBR sets it up correctly and indicates that it has done so, so it's safer to hard code the location in some way into the OS (e.g. in a BPB).

Yes, partition start hardcoded in BPB is the only reliable way.

(C) If it's in USB-HDD mode and the OS is written right at the start of the disk, DS:SI will be no use. If the location is hard coded into the OS (e.g. in the BPB) the OS will load correctly.

That's why you better forget about DS:SI at all.

(D) If it's in USB-FDD mode and there's an MBR with the OS in a partition, DS:SI can be used if the MBR signals that it is right, or the location can be hard coded into the OS. The OS will load exactly as in case (B).

The only difference for the OS will be that the drive number passed in DL will be 0 instead of 80h. And VBR code must use this value instead of that written in BPB.

There seems to be so little difference now between the USB-FDD and USB-HDD modes that I can't see why they bother providing two modes. What am I missing? Is it only the drive number that changes?

In general yes, the most noticeable difference will be the drive number.
You may ask, why were these modes introduced at all? That's because there is an old boot code still exists that uses the hardcoded drive number from BPB area instead of using that passed in DL. This boot code will work only in one of these two modes. The correct boot code will work in any mode.

The MBR will modify a VBR byte after loading it

MBR should not modify anything in loaded VBR since there may be the boot sector from an unknown file system.

Look. Here are some fundamental points for booting. You'll succeed if you follow it.
1. Use boot drive number passed by BIOS in DL. Only very old and buggy BIOSes pass this value incorrectly, but you better forget about that buggy machines.
2. Pass this number from MBR to VBR in the same DL register.
3. Don't rely on schemes that are not widely adopted, such as DS:SI.
4. Design VBR code so that it will work the same way in partitioned or non-partitioned drive.
5. First check LBA access and use it if present.
6. If not present, get device geometry by BIOS call INT 13h function 8 and use it for future CHS access.
7. Hardcode number of hidden sectors in BPB area of VBR and use it in VBR code.
8. Don't patch anything in VBR by MBR unless you are absolutely shure that that's your's VBR.
9. Don't use strange partitioning schemes (multiple partitions inside another partition) since OSes that you load (if they are not yours) won't see them and won't be able to load files from them. You better use a variety of extended partitions, but still not every OS will be able to continue booting from them.

[DavidCooper]

Thanks again Yoda - I think you've probably cleared everything up now, so I can now start work on writing the code to get this done.

The ONLY correct way, if you really need to stick to CHS access, is first to read drive parameters.

Good point - I've always overlooked that option.

Real BIOS may patch values in BPB (if recognizes it), may use values filled, may ignore them and use it's own translation scheme, but you have no reliable method to know, which method the BIOS have chosen. The only thing you can do is to know the results of chosen translation by INT 13h function 8.

I can see now why things are so confusing, given that there are numerous ways of doing things that can't be entirely relied on, and it also explains why there are so many conflicting accounts of how things work and how things are supposed to be done.

No. Forget about 18 SPT.

Okay - so again the primitive OS which just assumes it's on a floppy disk with 18 sectors per track will fail to load correctly in this situation.

It will work if it will get drive parameters first.

Only if it's then programmed to use what it finds there properly - at the moment my OS simply assumes 18 sectors per track and doesn't have the wit to switch away from that, so a lot of code needs to be modified to allow it to load from any other disk geometry. I'm not going to worry about using CHS for anything other than standard floppies at the moment, so I'll focus on adding LBA versions of all the loading and saving code that I use.

For non-FAT/NTFS file systems there is only one problem (for partitioned disks, not superfloppy) - obtaining number of hidden sectors, i.e. sectors, preceding the partition... So the reliable way would be to fill in the number of hidden sectors somewhere in you boot record.

So to be completely safe, it should have to be written in some way into the OS code when the OS is stored in a partition right at the outset.

Not to the code, but to the field specially assigned for that purposes in file system (FATs/NTFS) or to the space in file system that known in advance to be absolutely free (other file systems).

Well, for most OSes it will be a field in the file system that needs to be modified, but it mine the file system and OS aren't properly separated from each other, so I think of the whole thing being the OS.

I was visualising the VBR having a unique code in it which it could look for to see if it has loaded itself in, thereby detecting the case where the OS has been written at the start of the drive without the disk being partitioned. If the location is hard coded into the OS when it's originally written to the drive, that will not be necessary, but if you're relying on DS:SI and someone puts the OS at the start of the drive by mistake, looking up the first sector to see if it's the VBR would be a worthwhile check.

May that [unique for the drive] code be just 32-bit value of hidden sectors?

I thought the whole point of using DS:SI was that you didn't need to have any difference between the OS (including its file system table) in one partition and the same OS in another partition, so they wouldn't need to store a hidden sector value or have any unique identifier to tell each other apart. If an identical copy of the OS is placed at the start of the disk instead of in a partition, finding that the first sector on the disk is itself (meaning that the VBR has loaded in the VBR) would confirm that it's at the start of the disk and not in a partition - it can't be a different copy because there won't be any other copy of the OS (or of any other OS) on that disk. The alternative way of doing things is to hard code the start address in some way into the OS (or its file system table) to make sure it works in the location where it's been stored. I always imagined that the ideal was to be able to copy an OS into a partition and not have to modify a single byte of it (including the file system) to make it function properly, but that doesn't appear to be the normal way to do things after all. If you move an OS from one partition to another and don't update fields in its file system, it will crash, so it isn't really a better way of doing things - just different. What I've been looking for is a way of making the unmodified OS (including the data in the file system table) load without a crash regardless of where it's stored on the disk, but it appears that this isn't a requirement of a proper OS after all - they simply crash if they aren't modified to run in the location where they've been placed. The only way it could work properly would be if DS:SI was set correctly by all MBRs, and ideally with a standard way of indicating that the MBR has done this (DEC SP to put the stack out of alignment would be a good way - the VBR would then look at that before setting up its own stack and if bit-0 isn't 1 it could assume that it's been located at the start of the drive). That's how things really ought to have been done, so that's how I'm going to do it.

OS should not check BPB. OS should first try LBA access, then [if fail] check INT 13h/fun 8 for CHS access.

That's a useful tip - I can use that to distinguish the case of the OS on a real floppy from the case of the OS on the start of a flash drive.

You may ask, why were these modes introduced at all? That's because there is an old boot code still exists that uses the hardcoded drive number from BPB area instead of using that passed in DL. This boot code will work only in one of these two modes. The correct boot code will work in any mode.

Thanks for clearing that up, and for blowing away all the other misinformation and myths which have blocked my road to understanding all this for a very long time.

Look. Here are some fundamental points for booting. You'll succeed if you follow it.
1. Use boot drive number passed by BIOS in DL. Only very old and buggy BIOSes pass this value incorrectly, but you better forget about that buggy machines.
2. Pass this number from MBR to VBR in the same DL register.
3. Don't rely on schemes that are not widely adopted, such as DS:SI.
4. Design VBR code so that it will work the same way in partitioned or non-partitioned drive.
5. First check LBA access and use it if present.
6. If not present, get device geometry by BIOS call INT 13h function 8 and use it for future CHS access.
7. Hardcode number of hidden sectors in BPB area of VBR and use it in VBR code.
8. Don't patch anything in VBR by MBR unless you are absolutely shure that that's your's VBR.
9. Don't use strange partitioning schemes (multiple partitions inside another partition) since OSes that you load (if they are not yours) won't see them and won't be able to load files from them. You better use a variety of extended partitions, but still not every OS will be able to continue booting from them.

That's very good advice which I will follow for the most part, and most people should probably follow the entirety of it. The places where I will explore alternative paths are:-

3: I can make DS:SI work fine, and indeed it's vital to get the kind of functionality I need. If the misaligned stack signal doesn't show up, the extra functionality will disappear and my OS will know that it's either working with the wrong MBR or that it's been placed at the start of a disk, so it'll have to find another way to work out where it's been stored. Whether I can solve that completely depends on how much functionality I can fit in the VBR.

7: I think I may be able to avoid doing that, but again I'll have to see how much I can fit in the VBR once I've added LBA code to it.

9: I need this special partition system because I have to be able to boot up many versions of my OS from the same drive. No other operating system is going to be able to read my file system at the moment anyway, and the ability to boot other operating systems from subpartitions is just an extra little bonus which could be useful, even if they can't be found easily from the outside.

Okay - I think I know what I'm doing now, and I hope there's enough here to help other people find their way through the same minefield. Thanks again to Yoda and the others. I'll return to this thread before long (I hope) to let you know if I've managed to get it all to work the way I think it can. My aim is to get my OS to boot from any position without having it's location coded into it - the only case that I'm not sure I can cover in this way is where someone else's MBR is used (if there isn't enough room in the VBR for code to load in the MBR again and to explore it adequately - I suspect there's enough space if there aren't more than four partitions to check), but that could be sorted by allowing the location to be hard coded into the OS if it's placed in such an unsuitable environment and it would be up to the person putting it there to make sure it is done properly - that isn't really any different from what has to be done with other OSes anyway.