Migrating OS from floppy to USB flash and USB hard drives
Posted: Sun Apr 08, 2012 11:04 pm
Up until now I've done everything with floppy disks, but the small capacities are becoming highly limiting when it comes to data storage (even though there's still more than enough room for the OS itself) and I need to move on to something better. Because I'll need to go on using the BIOS for loading and saving for a while yet, I have to be sure that what I want to do will actually turn out to be possible. I don't want to use the hard drive in the machine as I wan't to keep Windows on there. I'm looking at MicroSD cards with a simple USB reader as the ideal way forward as this would hopefully allow 16GB cards to be switched like floppy disks without confusing the BIOS, and this matters because the tools I use to write my OS are all part of my OS: I've always worked by booting my OS, modifying it, changing disk, and saving the new version, so if the new version doesn't boot (this occasionally happens), I can simply boot up the old version and use it to load and modify the new version until it does boot properly. I also have a long series of previous versions sitting around on other disks which I can go back to if necessary to fix anything that's become corrupted and gone unnoticed for a long time. I want to go on working the same way, though a viable alternative may simply be to use ordinary USB flash drives (or USB hard drives) and have multiple versions of my OS stored on each (in a single partition) with a multiboot option in my own MBR which allows different versions to be booted, though that would have the little disadvantage of needing to be backed up regularly within Windows using dd in order to get a copy into safe storage on a different device. I've decided to go for the latter option first and to try out MicroSD cards later on where I can simply retain the option of booting different versions of my OS on each card too (which I might as well do as it'll only take up about 15MB for ten versions - each one living in 1.44MB of space and being possible to copy to a floppy disk at any time such that it'll always still be able to boot from one of those on an ancient machine).
One of the barriers to me doing this is getting my code from floppy disk to a USB flash drive in the first place, written in the right place on it, and done safely (without damaging anything on the machine I work on). Rawwrite makes the first step of that easy - I can make bits and pieces on floppy disks and get them into Windows as binary files without difficulty. The tool I need for the second half of the process appears to be dd, but the instructions that I've managed to find for using it are not sufficient for me to be confident about what I'm doing (and it isn't clear to me that the commands are going to be exactly the same for different versions of dd either). I can't afford to wipe out any drives by making mistakes. Even more importantly, I want other people to be able to get my OS onto a flash drive too after downloading the image, and that means that people who may know even less than I do are going to have to be able to do it safely too.
Anyway, there are a few questions that need to be resolved, so if anyone thinks they can help out, please do:-
1. I've read a lot of confusing stuff about how the BIOS boots up a machine, and one of those confusions relates to partitions. I've heard more than once that the BIOS will look for the bootable partition, load its boot sector into 7c00 and then treat that partition as if it's a drive in its own right, so that makes it sound as if you would access the second sector of that partition by treating it as the second sector of a drive, but that appears to conflict with how MBRs are supposed to work so I think it must be wrong, but I need to know for certain - it could only be right if the MBR code was completely sidelined by a BIOS, because if the MBR is allowed to run, its code must be able to access the entire drive, and when it loads in the boot sector from the right partition, there's no way the BIOS could recognise the point when that boot sector takes over from the MBR to start loading further sectors from within its partition - the BIOS isn't suddenly going to switch the way it works to restrict loads and saves to within the partition and with the first sector of that partition suddenly becoming LBA 0 (or sector 1 of track 0, head 0). So, what I need to know is, is there any truth in the idea that a BIOS might make a partition behave as if it's a drive or can I just assume that the BIOS always gives access to the entire drive?
2. How important is it for the CHS values to match up with the LBA values in the MBR's partition table? I've had a look at the hard drive in the machine I usually run my OS on, and what I've found is this: two entries, the first partition (MSDOS5.0) starting at the 1MB point and stopping 1MB short of the second partition (NTFS) if you go by the LBA values, but there is no such gap between the two partitions if you go by the CHS values. Is there some reason why the two systems shouldn't always match up (obviously not counting the point where CHS hits the FE FF FF limit and can't go any higher anyway)?
3. When copying the image of my OS to a flash drive, the image is only going to be about 16MB at the most (although it could be a lot smaller in the downloadable-from-Web version as it only really needs to hold a single copy of the 1.44MB OS - the other nine copies can be generated from that later on so that the multiboot options won't cause a crash if the wrong number's typed in when booting [while the copies will diverge later still as the user modifies them, always keeping a history of older versions in case they make a mistake at some point]), but the important point here is that the image isn't actually going to contain any part of the second partition specified in the MBR - the image will only hold everything from the MBR to the end of the first partition (or just to the end of the first copy of the OS within that first partition). Copying that to the flash drive should be easy enough using dd (provided that the command is formulated correctly), but I don't know how easy it would then be to use the rest of the disk. Assuming that the second partition is defined correctly in the MBR, is it simply a matter of asking Windows to format the second partition as FAT32, or is it more awkward than that? Also, could Windows mess up my MBR in the process, or would it limit itself to changing the partition type byte in the partition table? (That second question isn't so important as the MBR can simply be written to the disk again afterwards if it gets corrupted.)
[The size of the drive that my OS could be copied to could obviously come from a limitless range of sizes, so if someone follows simple instructions to copy the disk image onto a drive, the end location stated in the MBR's partition table for the second partition is invariably going to be wrong, but it can be adjusted afterwards from within my OS to fit the actual size of the drive, so that won't be a problem, just so long as the 2nd partition can be formatted later on as FAT32 within Windows.]
4. When building my OS for the first time on a flash drive, I'm going to need to copy the floppy image into the first partition location (with my OS modified to boot using LBA if DL!=0 and from the place where the first partition starts), and then I'm going to have to copy the MBR in from another floppy image containing just that one sector, so I need two versions of dd commands for these things, and they have to work with a Windows version of dd. I have seen a command for doing this kind of thing somewhere, but it had a great long serial number as part of it, presumably to identify the flash drive, and I don't know how you're supposed to get hold of that number if it's required. For the MBR part of it, dd if=mbr.bin of=???? bs=512 count=1 should be on the right lines, but I don't know what to put in place of the ????. Can anyone suggest what the ???? should be. For the OS part, I'd need something similar with a seek=???? added to it, but I think I can probably work that out myself. Once I've built my OS on the flash drive and got it working, I'll be able to copy multiple versions of it through the rest of the first partition from within my OS and I can work on getting the MBR to boot them all (pressing a number key to select any one of ten versions - each one taking up 1.44MB of space), so the other things I'll need to be able to do with dd should be simple enough as it's just a matter of copying the first 16MB of the drive into a file using dd, and then it can be copied back onto any drive using dd again with "if" and "of" switched over.
5. Where should the first partition start? Are there any limitations on where it should be, and how much space is it worth leaving in front of it? I could certainly use an extra sector or two for the multiboot code and could probably store MBR copies in the gap too, but I don't know if I want to leave a megabyte of empty space.
6. On another issue (not USB drives, but internal hard disk), I notice that there's a 3GB gap at the end of the hard drive in my machine, so I'm wondering if that's used by Windows XP for hibernating or some other vital purpose. Does anyone know if that's what this space is for? Given that the machine only has 1GB of RAM in it at the moment, I'm just wondering if it might be possible to put my OS into the end of that space without getting in the way of Windows, and whether it would be safe to mess with the MBR in order to make the machine multiboot for this? It looks as if it should be easy enough to chainload the original MBR from a new location in the first gap to boot Windows in the normal way, but I'd like to know if anyone's out there has done that. One complication may be how the system boots up if it's been hibernated, because the BIOS only boots my OS from USB floppy if the machine was not hibernated when it last closed Windows - if it was hibernated it simply ignores my OS and unhibernates Windows regardless.
The only reason I'm thinking about putting my OS on the internal hard drive at all is that the BIOS doesn't appear to notice a second USB drive plugged in at the same time, so I have no means to copy from one USB device to another while using my OS - having a version of it on the hard drive would give me more backup options without having to run Windows. Even so, I probably won't do this, but it would be interesting to know if it's a viable option.
One of the barriers to me doing this is getting my code from floppy disk to a USB flash drive in the first place, written in the right place on it, and done safely (without damaging anything on the machine I work on). Rawwrite makes the first step of that easy - I can make bits and pieces on floppy disks and get them into Windows as binary files without difficulty. The tool I need for the second half of the process appears to be dd, but the instructions that I've managed to find for using it are not sufficient for me to be confident about what I'm doing (and it isn't clear to me that the commands are going to be exactly the same for different versions of dd either). I can't afford to wipe out any drives by making mistakes. Even more importantly, I want other people to be able to get my OS onto a flash drive too after downloading the image, and that means that people who may know even less than I do are going to have to be able to do it safely too.
Anyway, there are a few questions that need to be resolved, so if anyone thinks they can help out, please do:-
1. I've read a lot of confusing stuff about how the BIOS boots up a machine, and one of those confusions relates to partitions. I've heard more than once that the BIOS will look for the bootable partition, load its boot sector into 7c00 and then treat that partition as if it's a drive in its own right, so that makes it sound as if you would access the second sector of that partition by treating it as the second sector of a drive, but that appears to conflict with how MBRs are supposed to work so I think it must be wrong, but I need to know for certain - it could only be right if the MBR code was completely sidelined by a BIOS, because if the MBR is allowed to run, its code must be able to access the entire drive, and when it loads in the boot sector from the right partition, there's no way the BIOS could recognise the point when that boot sector takes over from the MBR to start loading further sectors from within its partition - the BIOS isn't suddenly going to switch the way it works to restrict loads and saves to within the partition and with the first sector of that partition suddenly becoming LBA 0 (or sector 1 of track 0, head 0). So, what I need to know is, is there any truth in the idea that a BIOS might make a partition behave as if it's a drive or can I just assume that the BIOS always gives access to the entire drive?
2. How important is it for the CHS values to match up with the LBA values in the MBR's partition table? I've had a look at the hard drive in the machine I usually run my OS on, and what I've found is this: two entries, the first partition (MSDOS5.0) starting at the 1MB point and stopping 1MB short of the second partition (NTFS) if you go by the LBA values, but there is no such gap between the two partitions if you go by the CHS values. Is there some reason why the two systems shouldn't always match up (obviously not counting the point where CHS hits the FE FF FF limit and can't go any higher anyway)?
3. When copying the image of my OS to a flash drive, the image is only going to be about 16MB at the most (although it could be a lot smaller in the downloadable-from-Web version as it only really needs to hold a single copy of the 1.44MB OS - the other nine copies can be generated from that later on so that the multiboot options won't cause a crash if the wrong number's typed in when booting [while the copies will diverge later still as the user modifies them, always keeping a history of older versions in case they make a mistake at some point]), but the important point here is that the image isn't actually going to contain any part of the second partition specified in the MBR - the image will only hold everything from the MBR to the end of the first partition (or just to the end of the first copy of the OS within that first partition). Copying that to the flash drive should be easy enough using dd (provided that the command is formulated correctly), but I don't know how easy it would then be to use the rest of the disk. Assuming that the second partition is defined correctly in the MBR, is it simply a matter of asking Windows to format the second partition as FAT32, or is it more awkward than that? Also, could Windows mess up my MBR in the process, or would it limit itself to changing the partition type byte in the partition table? (That second question isn't so important as the MBR can simply be written to the disk again afterwards if it gets corrupted.)
[The size of the drive that my OS could be copied to could obviously come from a limitless range of sizes, so if someone follows simple instructions to copy the disk image onto a drive, the end location stated in the MBR's partition table for the second partition is invariably going to be wrong, but it can be adjusted afterwards from within my OS to fit the actual size of the drive, so that won't be a problem, just so long as the 2nd partition can be formatted later on as FAT32 within Windows.]
4. When building my OS for the first time on a flash drive, I'm going to need to copy the floppy image into the first partition location (with my OS modified to boot using LBA if DL!=0 and from the place where the first partition starts), and then I'm going to have to copy the MBR in from another floppy image containing just that one sector, so I need two versions of dd commands for these things, and they have to work with a Windows version of dd. I have seen a command for doing this kind of thing somewhere, but it had a great long serial number as part of it, presumably to identify the flash drive, and I don't know how you're supposed to get hold of that number if it's required. For the MBR part of it, dd if=mbr.bin of=???? bs=512 count=1 should be on the right lines, but I don't know what to put in place of the ????. Can anyone suggest what the ???? should be. For the OS part, I'd need something similar with a seek=???? added to it, but I think I can probably work that out myself. Once I've built my OS on the flash drive and got it working, I'll be able to copy multiple versions of it through the rest of the first partition from within my OS and I can work on getting the MBR to boot them all (pressing a number key to select any one of ten versions - each one taking up 1.44MB of space), so the other things I'll need to be able to do with dd should be simple enough as it's just a matter of copying the first 16MB of the drive into a file using dd, and then it can be copied back onto any drive using dd again with "if" and "of" switched over.
5. Where should the first partition start? Are there any limitations on where it should be, and how much space is it worth leaving in front of it? I could certainly use an extra sector or two for the multiboot code and could probably store MBR copies in the gap too, but I don't know if I want to leave a megabyte of empty space.
6. On another issue (not USB drives, but internal hard disk), I notice that there's a 3GB gap at the end of the hard drive in my machine, so I'm wondering if that's used by Windows XP for hibernating or some other vital purpose. Does anyone know if that's what this space is for? Given that the machine only has 1GB of RAM in it at the moment, I'm just wondering if it might be possible to put my OS into the end of that space without getting in the way of Windows, and whether it would be safe to mess with the MBR in order to make the machine multiboot for this? It looks as if it should be easy enough to chainload the original MBR from a new location in the first gap to boot Windows in the normal way, but I'd like to know if anyone's out there has done that. One complication may be how the system boots up if it's been hibernated, because the BIOS only boots my OS from USB floppy if the machine was not hibernated when it last closed Windows - if it was hibernated it simply ignores my OS and unhibernates Windows regardless.
The only reason I'm thinking about putting my OS on the internal hard drive at all is that the BIOS doesn't appear to notice a second USB drive plugged in at the same time, so I have no means to copy from one USB device to another while using my OS - having a version of it on the hard drive would give me more backup options without having to run Windows. Even so, I probably won't do this, but it would be interesting to know if it's a viable option.