OSDev.org

Posted: **Tue May 30, 2017 9:39 am**

I am building an os and I will insert it into a usb storage device.
My question is how can I read-write sectors to the usb storage device?
At http://wiki.osdev.org/Main_Page I can find the FDC for floppies and the ATA/SATA for hard disks and maybe CD/DVD.
What of all this I need for the usb storage device?

Posted: **Tue May 30, 2017 9:50 am**

Have you searched the wiki and the forum first? Tip: we had a thread about this very question roughly 3 weeks ago.

Please link me to any of your research findings.

Posted: **Wed May 31, 2017 12:54 pm**

Universal Serial Bus
USB Mass Storage Class Devices

You will need to read through both of these pages in order to read/write data to a USB storage device in 32-bit Protected Mode.

If you are still in 16-bit Real Mode, you can use the same BIOS methods that you would normally use to read from/write to a hard drive.

Good luck! (USB is probably in the top 3 hardest standards to implement, unfortunately.)

Posted: **Wed May 31, 2017 2:01 pm**

What you are probably asking, and is probably more likely the case, is that your machine, like most all machines now, no longer have floppy drives so you either have to write your work to a CDROM, the installed hard drive, or a USB flash drive to be able to boot it.

Writing to CDROM can be quite expensive since you may do it often, and can be a bit of work in itself. Writing to the hard-drive requires another operating system to be installed and running just to read from the external media where your OS is waiting to be installed, or requires this already loaded and running OS to be able to write to a partition of an existing hard-drive and be able to tell the system to boot that partition instead of the current one. Not a wise choice for the beginner.

Today, the easiest way is to write your OS (or test code) to a USB flash drive and boot the flash drive. Luckily, BIOS manufactures created a way to allow one to boot a USB flash drive/floppy drive/other media, still giving the "illusion" that it is a regular hard drive or floppy drive, allowing access via BIOS interrupt services.

Is this what you are wondering?

If so, there is a (somewhat) simple process that you can do to accomplish this.

1) You will need to format the USB media as either a floppy drive or a hard drive type image. To format it as a floppy drive will use the first 1.44meg and the remaining will be inaccessible via the BIOS services. You can format it as a hard drive image and use all of the capability as long as the BIOS supports the size of the media.

2) You need to write to the USB media as a disk image, overwriting everything that is on the media each time. However, if your Host OS allows you to mount the USB media and will give you access to it (*and* supports the file system you have installed on it), you can write to it with ease.

3) You will need to make your Test system boot from the USB by the settings within the CMOS startup. Tell it that you have a USB flash media as a hard drive image or a floppy image.

4) The test machine should now boot the USB media as the floppy or the hard drive.

5) Now you can access the USB media via BIOS Disk Read/Write services and as far as your code and the BIOS is concerned, it is an installed floppy/hard drive.

A few notes that you will need to know:
1) Some BIOSes will determine if the media is a floppy image or a hard drive image by the (lack of) MBR that is in LBA 0. There is no other way to tell the BIOS what size of media it is, except for this MBR at LBA 0. If no MBR exists, the BIOS will read the first sector expecting a BPB...
2) As stated in an earlier post, this only works in real mode, or via an x86 virtual mode. As soon as you move to protected mode, the BIOS services are no longer accessible.
3) As soon as you reset the USB controller (via your OS's USB driver), the BIOS will have no idea the state of the USB and can and may read and/or write to the wrong place on the media.

Does this help?
Ben
http://www.fysnet.net/the_universal_serial_bus.htm

Posted: **Thu Jun 01, 2017 2:22 am**

BenLunt wrote:To format it as a floppy drive will use the first 1.44meg and the remaining will be inaccessible via the BIOS services.

Are you sure about that? I've had no trouble using "superfloppy" formats up to 8GB.

Posted: **Thu Jun 01, 2017 3:21 am**

Hi,

Octocontrabass wrote:
BenLunt wrote:To format it as a floppy drive will use the first 1.44meg and the remaining will be inaccessible via the BIOS services.
Are you sure about that? I've had no trouble using "superfloppy" formats up to 8GB.

The problem is that there never was any official standard describing how "boot from USB" is supposed to work, so each BIOS does whatever it feels like, and you can't guarantee that anything will actually work on all computers (even if you test it on 50 computers and know it worked for those 50 computers).

For example; I wouldn't want to assume that at least one BIOS won't look for a BPB in the first sector, see "heads > 2", decide that it can't possibly be a valid floppy disk, and then decide to use hard disk emulation because the BPB doesn't make sense.

In the absence of an actual standard that describes how "boot from USB" is supposed to work, an OS developer has to fall back on "risk minimisation" because it's impossible to test on every computer (including computers that don't exist yet). This would include (e.g.) only using "valid for a real floppy" formats in the BPB if you depend on "emulated floppy"; or (even better) writing boot code that does not depend on "emulated floppy" (e.g. make it look "valid for a real floppy" in case the BIOS doesn't support/use hard disk emulation, but also use "drive number" in DL to determine what the BIOS decided to treat the device as in case the BIOS decided to treat it as a hard disk).

Cheers,

Brendan

Posted: **Thu Jun 01, 2017 11:26 am**

Octocontrabass wrote:
BenLunt wrote:To format it as a floppy drive will use the first 1.44meg and the remaining will be inaccessible via the BIOS services.
Are you sure about that? I've had no trouble using "superfloppy" formats up to 8GB.

As Brendan stated, each BIOS can and will be different. If you wish to make sure that your emulated floppy/hard drive image will be bootable on most hardware/BIOSes, you must stick with the 1.44Meg floppy image, using a known and standard 1.44Meg floppy BPB.

If you are only using your emulated floppy (USB thumb drive) on your machine, simply as a transferable media device, and don't plan on using it for anything else, and it supports an 8Gig SuperFloppy format, by all means, use it. However, don't think that you can then take that same USB thumb drive to another computer and expect it to boot with no problem.

To get back to the OP, if you want to (nearly) guarantee that your emulated floppy (USB thumb drive) will boot on (most) every computer, format it to a 1.44Meg floppy, with 2 heads, 18 sectors per track, with 80 cylinders, for a total of 2880 sectors.

If you need more space than the 1.44Meg allows, you can format it to a hard drive image, using a valid FAT32 BPB. Then you can use the total capacity of the drive. This is less of a guarantee to work, but most BIOSes will support a similar format.

On a side note, if you are using this emulated floppy (USB Thumb Drive) as an installation image, you can still use the remaining capacity of the thumb drive. For example, the boot up and installation part can easily fit within the 1.44Meg allowed, with the installation creating a 1.44Meg RAM drive, moving itself over to that RAM drive. Then within this 1.44Meg space, have a minimal USB stack, resetting the USB device, now having access to the whole capacity of the drive, and ignoring the first 1.44Meg worth of sectors. Just a thought.

Ben

Posted: **Thu Jun 01, 2017 5:19 pm**

BenLunt wrote: If you are only using your emulated floppy (USB thumb drive) on your machine, simply as a transferable media device, and don't plan on using it for anything else, and it supports an 8Gig SuperFloppy format, by all means, use it. However, don't think that you can then take that same USB thumb drive to another computer and expect it to boot with no problem.

Out of curiosity, is there some actual issues with SuperFloppy?

I've not done any research since I don't see much point in using SuperFloppy, assuming HDD emulation is available. However, when it comes to PC as a platform and the BIOS's that exist, nothing is guaranteed, so you can never "think that it will work" with anything. So is there a specific reason that SuperFloppy here makes an exception where it's particularly badly supported?

Put in another way, if you take a 1999 machine then probably it won't work, but if you take 2015 machine is there any reason to believe that the support is sketchy or significantly below 98% (or 99,9%)? I'm curious if this is a legacy problem or still a current one. I would assume/expect that almost all "modern" (let's say 5 years and less) PC supports USB booting without reading the mobo specs/manuals, and that is supports HDD emulation.

Both you and Brendan made a point that support might not be there, but technically support might not be there for almost anything (except Windows)..

Posted: **Fri Jun 02, 2017 12:11 am**

Hi,

LtG wrote:Put in another way, if you take a 1999 machine then probably it won't work, but if you take 2015 machine is there any reason to believe that the support is sketchy or significantly below 98% (or 99,9%)? I'm curious if this is a legacy problem or still a current one. I would assume/expect that almost all "modern" (let's say 5 years and less) PC supports USB booting without reading the mobo specs/manuals, and that is supports HDD emulation.

For old machines you can expect "buggy/quirky because USB was new back then", and for new machines you can probably expect "buggy for floppy emulation because nobody cares about floppy any more". For future machines, I'd predict "buggy for BIOS because manufacturer only bothered testing "GPT formatted USB flash booted by UEFI" properly".

Essentially, the only thing that really changes over time is the reason it's potentially buggy.

Note that for newer machines there's a third option - "CD emulation" (e.g. BIOS and/or UEFI supporting El Torito for USB flash). You can expect that to follow the same "buggy because it's new -> buggy because nobody cares about CD anymore" transition.

It's always about risk minimisation. For a general "rule of thumb", for the critical pieces of an OS (boot code, kernel, drivers, etc), over half of the code and data should be purely to check for things that could've gone wrong, handle things that have gone wrong, and avoid things that might go wrong; which means that (for the critical pieces of an OS) over half the development time should be spent worrying about things that "shouldn't" happen.

Cheers,

Brendan

Posted: **Fri Jun 02, 2017 12:15 pm**

Brendan wrote:Hi,

LtG wrote:Put in another way, if you take a 1999 machine then probably it won't work, but if you take 2015 machine is there any reason to believe that the support is sketchy or significantly below 98% (or 99,9%)? I'm curious if this is a legacy problem or still a current one. I would assume/expect that almost all "modern" (let's say 5 years and less) PC supports USB booting without reading the mobo specs/manuals, and that is supports HDD emulation.
For old machines you can expect "buggy/quirky because USB was new back then", and for new machines you can probably expect "buggy for floppy emulation because nobody cares about floppy any more". For future machines, I'd predict "buggy for BIOS because manufacturer only bothered testing "GPT formatted USB flash booted by UEFI" properly".

Essentially, the only thing that really changes over time is the reason it's potentially buggy.

Note that for newer machines there's a third option - "CD emulation" (e.g. BIOS and/or UEFI supporting El Torito for USB flash). You can expect that to follow the same "buggy because it's new -> buggy because nobody cares about CD anymore" transition.

It's always about risk minimisation. For a general "rule of thumb", for the critical pieces of an OS (boot code, kernel, drivers, etc), over half of the code and data should be purely to check for things that could've gone wrong, handle things that have gone wrong, and avoid things that might go wrong; which means that (for the critical pieces of an OS) over half the development time should be spent worrying about things that "shouldn't" happen.

True, but I was really asking if either of you know of any real issues with these subjects. Whether these are actually problem areas, not whether someone could make a mobo with faulty BIOS.

Overall, I agree, defensive coding is a goo idea, but are these things known to cause issues? I would expect that as the standards are pretty static that all newer systems support them fully, especially if we're talking about (which we weren't) USB HDD emulation. Obviously anyone could create a mobo with a BIOS that checks if the MBR and partitions belong to MS Windows, and do magic, and if not simply fail, but that would be pretty stupid.

So are you transferring earlier BIOS/firmware issues to present day tech and assuming that there _might_ be issues with this specific _thing_ (superfloppy), or if you are aware that there really is, and that it's still current? Not that I'm still planning to use it, because assuming superfloppy is supported, why wouldn't HDD emulation be supported? And if HDD is supported, why would anyone want to use anything else?

But, out of curiosity, I'm interested if superfloppy is known to be bad, or at least fail "often", with whatever version of "often" you want to use..

Posted: **Fri Jun 02, 2017 2:18 pm**

LtG wrote:But, out of curiosity, I'm interested if superfloppy is known to be bad, or at least fail "often", with whatever version of "often" you want to use..

I personally do not have any experience with SuperFloppy and therefore cannot comment either way. Sorry. I have used 1.44Meg and Harddrive emulation on multiple machines with no problems, though both setups where "stock" BPB images, nothing out of the ordinary.

I do remember trying to use a CD (El Toro) boot on a USB stick one time, and if I remember correctly, the BIOS wasn't new enough to support it.

In my experience, I have only used a USB stick as a bootable media transfer device. A device to boot to my OS or test code, then copy information to media already attached to the machine.

I still use (real) floppy disks and drives. They are extremely cheap and are very easy to program and control. The only down side is that most new machines don't even have a FDC anymore, so it must be a USB Floppy disk drive, which by the way, I have and the BIOS emulates it as a floppy disk just fine.

Ben

Posted: **Fri Jun 02, 2017 2:38 pm**

Hi,

LtG wrote:
Brendan wrote:It's always about risk minimisation. For a general "rule of thumb", for the critical pieces of an OS (boot code, kernel, drivers, etc), over half of the code and data should be purely to check for things that could've gone wrong, handle things that have gone wrong, and avoid things that might go wrong; which means that (for the critical pieces of an OS) over half the development time should be spent worrying about things that "shouldn't" happen.
True, but I was really asking if either of you know of any real issues with these subjects. Whether these are actually problem areas, not whether someone could make a mobo with faulty BIOS.

Overall, I agree, defensive coding is a goo idea, but are these things known to cause issues? I would expect that as the standards are pretty static that all newer systems support them fully, especially if we're talking about (which we weren't) USB HDD emulation. Obviously anyone could create a mobo with a BIOS that checks if the MBR and partitions belong to MS Windows, and do magic, and if not simply fail, but that would be pretty stupid.

So are you transferring earlier BIOS/firmware issues to present day tech and assuming that there _might_ be issues with this specific _thing_ (superfloppy), or if you are aware that there really is, and that it's still current? Not that I'm still planning to use it, because assuming superfloppy is supported, why wouldn't HDD emulation be supported? And if HDD is supported, why would anyone want to use anything else?

I've never used "supper floppy" and will never user "super floppy". Using "super floppy" sounds about as intelligent as licking a tall metal tower during a thunderstorm (if I licked 10 metal towers, does that mean licking an 11th metal tower would be safe? How many metals towers do I need to lick before I can incorrectly assume that licking metal towers is always safe? Why would any intelligent person consider it the first place?).

Today I added some code to check if "memory mapped PCI configuration space" areas are actually marked as "reserved" in the BIOS's memory map, and discovered the VirtualBox can't even get "int 0x15, eax=0xE820" right.

Cheers,

Brendan

Posted: **Fri Jun 02, 2017 2:43 pm**

BenLunt wrote:I do remember trying to use a CD (El Toro) boot on a USB stick one time, and if I remember correctly, the BIOS wasn't new enough to support it.

Are you sure you're not thinking of something like isohybrid?

Posted: **Sat Jun 03, 2017 3:55 am**

Brendan wrote:Today I added some code to check if "memory mapped PCI configuration space" areas are actually marked as "reserved" in the BIOS's memory map, and discovered the VirtualBox can't even get "int 0x15, eax=0xE820" right.

First off, points to whomever put a lmgtfy link in the wiki article... but someone should probably change that.

Second, are you saying that VirtualBox is giving you a "MCFG" table that points to a memory address that is in an "available" memory map area?

I just dumped my ACPI tables in VirtualBox, and (after switching to ICH9 chipset), my MCFG table has one entry pointing to address 0xDC000000. While this is not explicitly listed in my memory map, I'm also not giving my VM 4GB of RAM.

Posted: **Sat Jun 03, 2017 4:37 am**

Hi,

SpyderTL wrote:
Brendan wrote:Today I added some code to check if "memory mapped PCI configuration space" areas are actually marked as "reserved" in the BIOS's memory map, and discovered the VirtualBox can't even get "int 0x15, eax=0xE820" right.
First off, points to whomever put a lmgtfy link in the wiki article... but someone should probably change that.

Second, are you saying that VirtualBox is giving you a "MCFG" table that points to a memory address that is in an "available" memory map area?

I'm saying that VirtualBox gives me a "MCFG" that points to a memory area that was not mentioned by the memory map at all when it should have been mentioned in the memory map as "reserved".

Note that the ACPI spec says:

The BIOS does not return a range description for the memory mapping of PCI devices, ISA Option ROMs, and ISA Plug and Play cards because the OS has mechanisms available to detect them.
The BIOS returns chip set-defined address holes that are not being used by devices as reserved. Address ranges defined for baseboard memory-mapped I/O devices, such as APICs, are returned as reserved.

The memory mapped PCI configuration space areas are not from a PCI device (e.g. they don't come from something sitting on a PCI bus), they're a baseboard memory-mapped I/O device (e.g. they do come from something sitting on some kind of CPU host bus or link, the same as APICs, HPET, etc).

The reason this is important is that (if an OS reconfigures PCI device BARs) "not mentioned by firmware's memory map" means "free for a PCI device to use".

SpyderTL wrote:I just dumped my ACPI tables in VirtualBox, and (after switching to ICH9 chipset), my MCFG table has one entry pointing to address 0xDC000000. While this is not explicitly listed in my memory map, I'm also not giving my VM 4GB of RAM.

That's what I get too. It's wrong.

For the real computers I've tested, for "int 0x15, eax=0xE820" they all say that any "memory mapped PCI configuration space areas" (and areas used for HPET, APICs, etc) are reserved.

Also note that Linux has a similar check to mine and will report:

Code: Select all

PCI: BIOS Bug: MCFG area at dc000000 is not E820-reserved
PCI: Not using MMCONFIG

Of course Linux is still a buggy mess - when it detects this problem it assumes that the memory map is correct (and that the "MCFG" must be wrong) even though it's far more likely that the memory map is wrong (and that the "MCFG" may be correct); so there's a chance that (if Linux reconfigures PCI device BARs) it'll cause problems that could've/should've been avoided.

Cheers,

Brendan

OSDev.org

Read from usb storage device

Read from usb storage device

Re: Read from usb storage device

Re: Read from usb storage device

Re: Read from usb storage device

Re: Read from usb storage device

Re: Read from usb storage device

Re: Read from usb storage device

Re: Read from usb storage device

Re: Read from usb storage device

Re: Read from usb storage device

Re: Read from usb storage device

Re: Read from usb storage device

Re: Read from usb storage device

Re: Read from usb storage device

Re: Read from usb storage device