Migrating OS from floppy to USB flash and USB hard drives

[Yoda]

Glad to help you.
May the Force be with you!

[egos]

Hi, jedis

First of all i would like to say that different BIOSes have different behaviour to the same USB options.

I found the PCs that hide sectors even if USB-HDD option was enabled. In one my PC there is a special option "Forced FDD" that does this. Here is the note for it: Forced FDD option can be used to force a HDD formatted drive to boot as HDD. I checked, it actually makes field HiddenSectors zeroed (virtually)! So this thing is popular enough. As to the CHS BIOS geometry, the situation is same. Some (oldest) BIOSes use hard-coded CHS geometry, and do not anymore. So we can have hard-coded CHS geometry 18x2 for the floppy mode (and as floppy-specific boot loaders usually do not use function 8 they can be wrong). Some BIOSes use hard-coded CHS geometry and then patch by its values (virtually) BPB fields SPT, Heads. Some (modern) BIOSes use the values of BPB fields SPT, Heads as CHS BIOS geometry for the drive. And so on.

You can find here some right notes.

Edited: to force a HDD formatted drive to boot as FDD.

[DavidCooper]

Hi folks,

Thanks for that link, egos. So there can be complications, with many of them affecting low-capacity flash drives. I doubt I can avoid all the possible problems at the first go, so the best way forward's to get some code written, do tests, then modify as necessary.

I've ditched the DS:SI idea - my MBR will still set DS:SI up to help any loader that might depend on it, but it'll also pass the LBA of the location in ECX to speed things up and misalign the stack with DEC SP to indicate that both these things have been done. I've reworked my VBR so that it starts by looking for this signal, and if it finds it it jumps straight to code that loads the OS using LBA and the ECX value as the start location. If the signal is missing, it then uses INT 13h AH=8 and checks the returned sector value (after cancelling the top two bits) to see if it's 18. If it's 18, it assumes it's a floppy disk and boots up using my old code for loading from one of those. If there are more sectors than that per track, it assumes LBA is available (currently without checking, but I'll sort that later), then it loads in the first sector of the disk to see if it finds itself there. If it finds itself, it sets EAX to 0 and jumps to the code to load the OS using LBA. If it finds an MBR instead, it must be the wrong MBR, so the only option is to trust the offset LBA value stored in the OS and not write the contents of EAX on top of it - that means jumping past the point just after where that is done.

What this means is that the OS will boot correctly even if the wrong location value is written into the code, just so long as my own MBR is used or if the OS has been put right at the start of the drive. If the first copy of my OS in the special partition has the right location value stored in it, it will also boot correctly if someone replaces my MBR with someone else's. This provides the maximum amount of safety, or more accurately, it will do once I've dealt with CHS properly for ancient systems which lack LBA - I don't know how likely that is to occur on a machine with USB capability, but it will certainly apply on many old machines if the OS is installed on the internal hard drive. The good news is that I've still got 90 bytes of room left to play with, plus at least another 100 which can be moved out of the boot sector if necessary - the current version of my CHS load code actually loads two whole modules of the OS and the disk directory (to three different destinations), so that can be hacked down a bit if more space is needed.

Next step's to write LBA load/save code for the OS to use for all subsequent loads and saves, and then I can try putting it on a flash drive where it should with any luck work both at the start of the drive and at the start of a standard partition. After that, I'll write the MBR and the mini-OS that will live in gap sectors and emerge to help do the more complex multi-boot stuff.

[egos]

Universal stage 1 boot loaders (which use LBA/LBA-to-CHS translation with actual CHS parameters taken from BIOS) work fine with whole disks/primary partitions using HiddenSectors field. My FAT(1x) boot loader can work even with floppies but I don't use it for that purpose because it does not control diskette parameter table and more large (1 kb). I use DS:SI only in EBR boot loader with my MBR boot loader (when extended boot signature 88h was found). If you will write own universal stage 1 boot loader and will use it with MBR boot loader based on the same technology this will be enough.

[DavidCooper]

I see it gets a lot more complicated and that GPT's going to be necessary soon, so whatever I do now with MBRs won't last long. All that really matters is putting something together that boots up, and realistically, anyone who uses my OS or MBR will be using both of them, so it should all work fine.

I definitely can't get the BIOS (on the machine I'm using for programming) to handle more than one USB drive plugged into it - only one shows up, and if I try taking that one out and plugging another into the same place, it won't respond to it, so there's no shortcut available to do the transfer from floppy to flash drive: I'll have to go via Rawwrite and HxD under Windows, though now that I've managed to work out how HxD works it should at least be easy (and very safe, so there's definitely no need for me to use dd - no more worries about how ordinary people are going to be able to download the image and get it onto a flash drive).

[DavidCooper]

I see there are also problems with my original idea of using partitions on a USB drive - Windows doesn't want anything to do with it once it's no longer a FAT32 drive - it won't even let the hex editor access it unless I agree to format the drive first, thereby losing/damaging the data that I'd need to be able to copy off the drive to make a disk image that can be written to another USB disk or for sent across the Web. This means that I'll have to explore one of the following two options instead if I want to be able to go on working with a partitioned USB drive in Windows:-

(1): Leave the start of a FAT32 bootsector in the MBR to trick Windows into allowing a hex editor to access it without insisting on formatting it first (I've tried this and it works). It can then have all sorts of partitions on it which Windows won't notice, but the only way to read from it or write to it within Windows would be through a hex editor: even if one of the partitions was FAT32, Windows wouldn't recognise it there - all it would find is what might appear to be a massively corrupted directory at the start of the disk.

(2): Keep it as a proper FAT32 drive so that Windows can use it like any other FAT32 USB drive, but shorten the amount of space it's allowed to write to, thereby creating a hidden area at the end which can contain additional partitions. The first sector can still be an MBR, without Windows realising that it specifies partitions, and the first partition would start from sector 0 (which is obviously not normal - it would prevent the first sector of that partition from being a VBR for that partition, though a VBR for it could be located elsewhere if the FAT32 partition needed to contain an OS as the MBR would know how to find it). Everything I planned to do should still be possible if Windows can be fooled by this, but the FAT32 partition has to come first and has to appear not to be a partition. I hope it works, because I want to be able to do such things as write .bmp photo files to the FAT32 partition within Windows and then access these files after booting OSes in other partitions which aren't FAT32.

[Yoda]

Windows doesn't want anything to do with it once it's no longer a FAT32 drive - it won't even let the hex editor access it unless I agree to format the drive first

Download and use DMDE - one of the most powerful free low level disk utility. It will edit the unformatted drive, including copying drive image to or from the file.

[egos]

Maybe the better way to have configuration with multiple versions is to put each version in different folder of FAT partition and to use something like GRUB to make choice between versions. For example:

Code: Select all

color light-gray/blue black/light-gray
timeout 10

title Link 0.52 (current version)
kernel /system/link.os system
module /system/link.fs

title Link 0.51 (previous version)
kernel /system51/link.os system51
module /system51/link.fs

My kernel/taskman takes system configuration files from system folder (which is a kernel parameter) so I can use "Multiple versions on single volume" configuration.

[DavidCooper]

Download and use DMDE - one of the most powerful free low level disk utility. It will edit the unformatted drive, including copying drive image to or from the file.

Thanks for pointing me towards that - it'll make the first option viable if the second option doesn't work out. For the moment I'm going ahead with the idea of keeping the disk in a form that Windows can treat as FAT32 while having partitions on it which Windows doesn't notice.

Maybe the better way to have configuration with multiple versions is to put each version in different folder of FAT partition and to use something like GRUB to make choice between versions. For example:

Code: Select all

color light-gray/blue black/light-gray
timeout 10

title Link 0.52 (current version)
kernel /system/link.os system
module /system/link.fs

title Link 0.51 (previous version)
kernel /system51/link.os system51
module /system51/link.fs

My kernel/taskman takes system configuration files from system folder (which is a kernel parameter) so I can use "Multiple versions on single volume" configuration.

The trouble with FAT32 (although I'm making some assumptions about how it works as I haven't read up on it fully) is that things can move around and files aren't necessarily stored in continuous blocks. I need my own partition(s) so that I can stick data somewhere and know it's fixed in place and can be expanded into the space following it. I only want a FAT32 partition as an addition which Windows can access so that it's easy to export and import data - I'll probably restrict it to a gigabyte and use the rest of the space my own way, while the code I use to read from or write to the FAT32 partition will be the absolute minimum necessary to do that job, avoiding long filenames and initially only dealing with files stored as contiguous blocks, never deleting a file unless it's the last one in the partition (typically they'll all be deleted at the same time anyway as it's only going to be used for data transfer and never for longterm storage).

[Edit: have read up on FAT32 properly now and can see that it doesn't do anything fancy. Changing one entry in the BPB is all it takes to reduce its range to 1GB, thereby leaving the rest of the disk free for hidden partitions which Windows won't see or interfere with, and it's clear that this should be the only value that counts when it comes to detecting the size of the FAT32 partition. One strange thing though is that the value that was there before (BPB_TotSec32 = EC 5B 78 00 = 7887852 sectors) is greater than the number of sectors that exist on the drive (7872440), and it was Windows that put the value there when it formatted it.]

I still haven't managed to get my OS to boot from a USB flash drive, and the reason for that is that my new code for loading and saving uses LBA (while reserving CHS for floppy disks), but the BIOS on the machine I use for programming seems to be unable to use LBA with USB drives (despite being able to use it without difficulty for the internal 160GB hard drive). That means I'm going to have to write a third version of all my loading and saving code, so I might as well do it properly this time and write a version which can handle any CHS geometry it encounters. It also means my machine can only access a little over 8GB of USB drives via the BIOS. The simplest way round that will be to find a machine with a more capable BIOS, but 8GB is plenty for now and should see me through to the time when I've written code capable of handling USB stuff directly without having to go through the BIOS. I just hope this BIOS is actually capable of writing to a USB drive reliably in addition to reading from one, but time will tell.

[DavidCooper]

Cancel a lot of the above: I've just discovered where I've been going wrong - I've been opening the drive the wrong way in HxD (logical drive instead of physical drive) and writing to the wrong places in it as a result. There is an MBR in it after all, and what I took for sector 0 was actually just the start of the only partition, but because the partition was labelled with 80h for bootable, whatever I put in its boot sector did get run, even though it was 64 sectors away from where I thought it was. Maybe it'll respond to LBA after all...

[Edit: Just done Int 13h with AH=41h etc. to see if LBA access is available, and the carry flag was clear, so it definitely should be - that means it really should be able to boot now, and yet it's still crashing.]

[egos]

The main rule for sizes is: volume size <= partition size <= available space (starting after MBR)!

Be careful when you're changing PT/BPB fields! Some changing requires reformatting volumes (maybe with zeroing BPB fields before).

As I said above, if you will write boot loader(s) based on "universal algorithm" you will have no problem with CHS geometry (almost ).

[Combuster]

I'd put the inequality signs the other way around

[egos]

[Edit: Just done Int 13h with AH=41h etc. to see if LBA access is available, and the carry flag was clear, so it definitely should be - that means it really should be able to boot now, and yet it's still crashing.]

Show the code.

Try to use other USB-XXX option. This helped me in 100%.

[Yoda]

Just done Int 13h with AH=41h etc. to see if LBA access is available, and the carry flag was clear, so it definitely should be - that means it really should be able to boot now, and yet it's still crashing.

It is insufficient to check just carry flag for LBA access availability test.
Yet there may be a crash for other reasons, not concerning LBA availability.

[DavidCooper]

The mismatch in disk size was imaginary - just a stupid error of mine caused by being new to hex editors. The BIOS reports 7936683 sectors by returning 254 127 243 for heads, sectors, cylinders (Int 13h AH=8); HxD shows 7897088 sectors now that I'm using it properly; Windows formats the drive to use 7887852 of them.

I linked earlier to something that said you can damage a flash drive by trying to read beyond the end of it, so I'm now wondering how HxD makes its decision not to display all the ones the BIOS says are available, though I haven't tried Int 13h AH=48h yet (it takes so long setting it up to do each experiment when doing it all in a bootsector) so maybe it'll give a different figure from Int 13h AH=8h.

[Edit: Just done Int 13h with AH=41h etc. to see if LBA access is available, and the carry flag was clear, so it definitely should be - that means it really should be able to boot now, and yet it's still crashing.]

Show the code.

I'm not looking for help debugging it just yet, and it's complex - it goes through tables of data loading them into registers and a DAP, then it calls Int 13h after loading AH with 2 or 42h depending on whether it's using CHS or LBA. The CHS version definitely works when loading from a floppy, and the values that end up in the DAP look correct. When attempting to boot from a flash drive, the code is definitely running through as it displays the CHS track number on the screen as it goes along (which doesn't have any impact on where it's actually reading from if it's using LBA), but it gets to a point where it stops loading and the beeper goes off repeatedly and I have to switch the thing off, so it isn't so much a crash as a freeze, though a very noisy one. I'll get it sorted eventually - it just takes so a long time setting up each experiment and my hands are falling to pieces due to RSI.

Try to use other USB-XXX option. This helped me in 100%.

I'm not getting any option to change the USB mode - when I boot from USB floppy it calls it a USB floppy in the boot menu, and when I boot from a flash drive it just offers "USB:" (I think, from memory). The only choice is between booting from that and the internal hard drive with no way to change the mode of anything.

It is insufficient to check just carry flag for LBA access availability test.
Yet there may be a crash for other reasons, not concerning LBA availability.

Don't worry - I'll get there. My priority in posting here at the moment is just to correct errors in my earlier posts which could mislead other people who might be thinking about trying to boot from flash drives.