[TUTORIAL]: How to read (and supposedly write) floppies.

[Brynet-Inc]

Sorry, I didn't mean to, I was wondering if anybody else (presumably writing in ASM) had used the code.

Is there a C compiler that compiles into NASM?

I'm pretty sure nwcc can output code in NASM and AT&T/GAS syntax...

http://nwcc.sourceforge.net/

GCC outputs AT&T syntax.. which is in return compiled by the GNU Assembler.

[mystran]

Please do not start the C vs. ASM here as well.

Sorry, I didn't mean to, I was wondering if anybody else (presumably writing in ASM) had used the code.

The tutorial is under 24 hours old.

Is there a C compiler that compiles into NASM?

Yes. You.

The point is not to copy the code as such, but to read the tutorial and understand what needs to be done in order to deal with a floppy drive. With said information one is then supposed to be able to write a proper driver.

[Kevin McGuire]

Now if you want me to say it is great I can. Do not feel like I am saying it is not good enough or feel like this is a fact and not a opinion. This is just what I use as a second pass over any document I write. I get my idea out, then I revise it and just about always try to double each paragraph in size as a general rule of thumb.

I know you can do another pass and rewrite each paragraph to be twice a big. Trust me by doing so it will cause you to force you're self to express more information in the writing thus creating a more useful document, and if done too incorrectly you could just bloat the document and start talking about what you did last week.

Try to use words as much as possible then code, and finally use images. If so it will let you squeeze the most of that potential a article has.

You have to image the first draft's paragraphs being written in a high level language, and then you act as a computer and perform a second pass and translate it to a lower level until you reach what you feel gives a balance between explaining the universe and telling someone what a floppy disk is.

[~]

What about PIO-like access?

Am I right when I say that the data register to read/write without using DMA is 0x3F5?

Is it also right to say that it must be read byte-to-byte? Or maybe could it be read from dword-to-dword using INSD/OUTSD?

Should bit 3 of port 0x3F2 ("DMA & IO interface enabled") be cleared to 0 if one plans to not use DMA (I think not...)?

A diminute reference here:

http://www.ousob.com/ng/progref/ng1159e.php

[mystran]

I actually have not bothered to figure out how to do PIO. Reason for this is that several sources claim that not nearly all controllers support it, so DMA is supposedly the safest way. Plus with DMA you don't need to worry about serving the interrupts in time.

But how to do it would probably be found in the controller datasheet. One for Intel 82077AA" can be found at http://bos.asmhackers.net/docs/floppy/d ... asheet.pdf

The DOR bit you are referring to is "DMA and interrupts." So as far as I understand, if you want interrupts you'll enable it, and then set NDMA during SPECIFY if you don't want DMA as well.

Ofcourse if you don't want interrupts either, then you probably don't need the DMA bit in DOR either, but for all practical purposes you probably want interrupts anyway, so who cares..

As for 32-bit transfer with INSD/OUTSD, no. It won't work. This should be obvious when you look at the register addresses for the floppy controller, and then read from "The Intel Manual" how word/dword I/O works.

[~]

Thanks for the reply. I just thought that maybe the floppy driver could be simpler if it's done this way, one shouldn't care about initializing DMA, and anyway floppy is already too slow to make any difference.

Now, in the following document it says nothing about recalibration, maybe it's wrong and the cause of why my driver locks up:

Code: Select all

FDC Programming Considerations

        Three phases of command execution:

        1.  Command phase; commands are sent from the CPU to the FDC via
            port 3F5h;  bit 6 of the Status Register at 3F4h must be zero
        2.  Execution phase; FDC executes instruction & generates INT 6
        3.  Result phase; status and other information is available to CPU;
            INT 6 sets bit 7 of BIOS Data Area location 40:3E which can
            be polled for completion status


        Example of a read operation:

        1. turn disk motor on and set delay time for drive spin up
        2. perform seek operation;  wait for disk interrupt
        3. prepare DMA chip to move data to memory
        4. send read command and wait for transfer complete interrupt
        5. read status information
        6. turn disk motor off

Also, the tutorial would be easier to put in practice if it had a main.c sample program with a "main()" entry point to demonstrate its functionality. Just a thought.

I'd dare to test PIO support in all of the floppies I have, like 5 of them from different years, from 1997 and 2004, 2006, and a laptop one... If that works without problems I'd feel very confident that it will work nowadays...

[bubach]

If you want asm source, take a look at mine:
http://bos.asmhackers.net/docs/floppy/snippet_10

special thanks to dex, who let me use his floppy code for reference.

[mystran]

Also, the tutorial would be easier to put in practice if it had a main.c sample program with a "main()" entry point to demonstrate its functionality. Just a thought.

Yeah agreed. I'll see of adding one.

I'd dare to test PIO support in all of the floppies I have, like 5 of them from different years, from 1997 and 2004, 2006, and a laptop one... If that works without problems I'd feel very confident that it will work nowadays...

Could be. I'll look into it at some point.

[Brendan]

Hi,

IMHO the tutorial is very good - I've just got a few notes...

The important thing here is that DMA controls how much data we transfer. Bochs happens to be kind enough to not report an error if you run into the end of track before DMA says it's time to stop, but real hardware will (at least mine does). We'll get to the DMA details (which several sources actually get wrong) later.

AFAIK the EOT value (End Of Track) can also be used to control how much data is transfered. For example, you can read sector 3 with EOT set to 3 and you'd only get one sector.

It may be worth mentioning that the floppy hardware, etc doesn't really care much about sector numbers. Each sector on the floppy begins with some control data that includes a sector number, and when you ask to read sector N the floppy drive will read the first sector that has N as it's sector number in the sector's control data.

This means that sector numbers don't need to begin with 1 and don't even need to be sequential. If you want to format a disk where sectors are numbered from 33 to 50 then that'll work fine. If you want to format a disk where sectors are numbered non-sequentially (e.g. 2, 3, 5, 7, 11, 13, 17, ...) then that can work too. For non-sequential sector numbers you won't be able to read more than one sector per transfer, as the hardware does "next_sector_number = last_sector_number + 1".

Also each track on the disk can be formatted in a completely different way, and sectors don't necessarily need to be 512 bytes long. AFAIK this includes changing the data rate between tracks, and the number of sectors per track, the bytes per sector, sector numbering, etc.

Usually it doesn't make sense to use unusual disk formats because other OSs wouldn't be able to read them. However if you want to make sure another OS can't read your disks it can be a good idea - it wasn't that uncommon for non-sequential sector numbers to be used for copy protection in old DOS software (where you couldn't copy the disk with standard software because standard software expected sequential sectors).

There is another purpose for all of this - increased disk capacities. For floppies there's an "intersector gap" and control data that's used by the hardware to determine where one sector ends and another starts. Larger sectors means less gaps and less control data, and can mean less wasted space on the disk and higher capacities. Also you can format the disk with smaller "intersector gaps" - this isn't that unusual either (there's a 1680 KB floppy format that reduces the intersector gap to make space for 21 sectors per track on a standard 1440 KB floppy).

It may be possible (I haven't tried it or done the calculations, but it seems likely) to format a floppy with three 4 KB sectors per track, giving 1920 KB on a standard 1440 KB disk.

Because each track can be formatted differently, it would also be possible to use 18 (512 byte) sectors for the first cylinder so that DOS/Windows could read the first sector. This way you could set values in the BPB to tell DOS/Windows that the floppy only has one track, and format the remaining tracks any way you like. For example, you could use 18 (512 byte) sectors on the first cylinder and three 4 KB sectors per track for the other 79 cylinders and end up with a 1914 KB floppy that doesn't cause problems on DOS/Windows. Of course DOS/Windows won't be able to read more than 18 KB (which can be a good thing).

What is covered:

I would be tempted to include some information on detecting what media is in the drive and if it's write protected or not. For example, for a disk drive that supports 2880 KB disks you could have a 2800 KB disk, a 1440 KB disk, a 720 KB disk, a 360 KB disk, a "custom" format disk, or nothing.

Some information is returned in "status register 3" which is returned by the "sense drive status" command. This is mostly used to determine if the floppy is write protected or not (it also includes a "two side" flag but I doubt that it's very useful - a 1440 KB floppy could be formatted as a single sided 360 KB disk regardless of what the hardware is or says).

First you'd find the data rate using trial and error - e.g. try to read sector 0, head 0, track 0 using a 1000 Kbps data rate, then try 500 Kbps, 300 Kbps and 250 Kbps. The first data rate that works without error is the correct data rate. Next try to read read sector 0, head 1, track 0 to see if the disk is double sided or not.

Lastly, you can "probe" for the number of sectors per track. For example, if the data rate is 500 Kbps you could try to read sector 21 (to see if it's a 1680 KB disk) then sector 18 (to see if it's a 1440 KB disk).

For non-standard disk formats, detection is your problem. If you support other sector sizes you could try every (supported) sector size from largest to smallest to see what works without error. AFAIK if you try to read a small sector when a large sector is present it may actually work (i.e. you might successfully read the first 512 bytes of a 1024 byte sector).

DOS/Windows uses a data table in the first sector, called the BPB (or "BIOS Parameter Block", which AFAIK has nothing to do with the BIOS) to determine the media format. Your OS could use this too, but it doesn't work if the BPB isn't present (possible for non-microsoft format disks). This can causes problems in your low level disk driver that the higher level file system code shouldn't need to care about. The BPB also can't support some non-standard formats (e.g. if you format tracks differently), and I'd consider the BPB to be "Microsoft specific" in that it contains fields that make no sense for other file systems (e.g. number of FATs, sectors per FAT, sectors per "allocation unit"). If you use the BPB and put correct values in there, then DOS/Windows will try to use it and give an error message. If you don't use the BPB then DOS/Windows will assume the disk is unformatted (which IMHO is better for "non-FAT" disks). Of course if you're formatting the disk as FAT then you'd put a BPB in there (but that's a file system issue not a floppy driver issue).

There can be several floppy controllers, each with up to four drives. On most modern computers you'll find one controller, with one or two drives (=one cable), so whether or not you want to support more controllers is up to you. All of them will normally use the same DMA 8-bit channel (2), and the same IRQ (6). This obviously means that you can only access one drive at any given time.

For some historical information see this web page. For all (modern - 80386 and later) practical purposes it's no longer possible to have 4 devices on the same floppy controller.

For resources, there is nothing to say additional floppy controllers can't (or won't) use different DMA channels and IRQs. Most ISA "I/O expansion cards" have jumpers that are used to set the I/O ports, DMA and IRQ used by the floppy controller.

Because the disk rotates the same speed whether you are reading or not, it'll take about the same time to read one sector or the whole track. Well, technically reading a single sector takes on average 37% of the average time to read a full track, or about 20% of the time to read both tracks, assuming the second track starts where the second track ends. So if we end up reading the other sectors as well, we'll waste around 6.6 (single track) or 7.2 times (multi-track) by reading sector at a time.

More accurately, there's an average of half a disk revolution before any read or write, plus the time taken to read/write the data. There's also "intersector gaps", which mean that for 18 sectors per track the time taken to read/write a sector is less than (but not equal to) 1/18ths of a revolution - you don't need to wait for the gap at the end of the last sector. This means reading a single sector costs an average of roughly 55% of a revolution (for 18 sector disks) and reading an entire track costs an average of about 145% of a revolution. For reading an entire cylinder (both heads) you'd have to wait for the gap at the end of the last track and any unused space at the end of the track before you'd see the first sector of the track on side 2. This is a result of the way the "format track" command works (i.e. wait for the index mark, then start creating the first sector of the track). This makes reading a full cylinder (very roughly) equivelent to about 245% of a revolution.

The main benefit of caching entire cylinders is that you eliminate the need for seeks (once a cylinder is in the cache you never need to seek to it again, except for writes which are less common). The other benefit is that it makes it easy to support truly arbitrary transfers (e.g. read 123 bytes starting from the 12345th byte).

As far as I understand, it doesn't matter much if you set or clear the "skip deleted" bit. If you don't set it (I don't) the read will fail when it hits a deleted address mark (which you can write with a special command it seems) while if you set it, the sector with such a mark will get skipped. So what are these deleted address marks then? I have no idea, but normal floppies seem to work quite fine whatever the setting is, so I'd guess it's something that's useful for tapes. There would be a READ_TRACK command for reading the whole track, deleted or not, but you can't do that multi-tracked on both heads.

The "skip deleted" flag (and deleted sectors in general) is a method of avoiding bad sectors. For an example, instead of formatting a floppy with 18 sectors per track numbered 1 to 18, you could format it with 18 sectors per track where the first 17 sectors are used (number 1 to 17) and the last is unused (numbered 255 perhaps). When you detect a bad sector (e.g. when you're verifying the data after a write) you could do a "read track" followed by "format track", "write track" and then a "write deleted"; so that the track ends up with the bad sector marked as "deleted" and the last (previously unused) sector is used to make up the difference. In this case the floppy hardware will skip the bad sector and the disk driver can continue using sectors 1 to 17 like normal.

AFAIK the deleted flag isn't actually used by other OSs. This makes it a viable alternative for data hiding (e.g. have a hidden sector marked as "deleted" containing data that only your OS knows about).


Cheers,

Brendan

[Solar]

There is another purpose for all of this - increased disk capacities. For floppies there's an "intersector gap" and control data that's used by the hardware to determine where one sector ends and another starts. Larger sectors means less gaps and less control data, and can mean less wasted space on the disk and higher capacities. Also you can format the disk with smaller "intersector gaps" - this isn't that unusual either (there's a 1680 KB floppy format that reduces the intersector gap to make space for 21 sectors per track on a standard 1440 KB floppy).

It may be possible (I haven't tried it or done the calculations, but it seems likely) to format a floppy with three 4 KB sectors per track, giving 1920 KB on a standard 1440 KB disk.

Not all PC floppy disk drives might support this, but it's true. The floppy drives used on the Amiga read the whole track at once, no matter what you requested. The standard format packed 1760 KB / disk, and there were formats around getting quite close to the "optimum" 1920 KB.

[mystran]

AFAIK the EOT value (End Of Track) can also be used to control how much data is transfered. For example, you can read sector 3 with EOT set to 3 and you'd only get one sector.

Indeed. But using DMA is more reliable, as how the "end-of-track" is reported seems to wary, at least between my real floppy drive and Bochs. It seems that on real-hardware, you get status=abnormal-termination if anything other than TC from DMA causes the transfer to terminate. On Bochs on the other hand, you'll get status=success, and it does "something." Since using setting the DMA count to the correct value works properly in either case, and seems to be the method most commonly used (based on tracing DMA/FDD debug messages on Bochs running various operating systems), I'd personally avoid using other methods.

This means that sector numbers don't need to begin with 1 and don't even need to be sequential. If you want to format a disk where sectors are numbered from 33 to 50 then that'll work fine. If you want to format a disk where sectors are numbered non-sequentially (e.g. 2, 3, 5, 7, 11, 13, 17, ...) then that can work too. For non-sequential sector numbers you won't be able to read more than one sector per transfer, as the hardware does "next_sector_number = last_sector_number + 1".

Also each track on the disk can be formatted in a completely different way, and sectors don't necessarily need to be 512 bytes long. AFAIK this includes changing the data rate between tracks, and the number of sectors per track, the bytes per sector, sector numbering, etc.

Yeah, I know all that. I left it out from the first version of the tutorial 'cos I wanted to get that out fast, and for the example code it doesn't make sense to deal with all that stuff, because it makes the whole code a lot harder to follow. As for changing data rate between tracks, or the number of sectors per track, the latter has been discussed in LKML at some point, and the issue is that there is no known convention for this, and the only proper solution then seems to be for user to tell the driver all those details, because otherwise you'd need autodetecting that for each track separately, which is supposedly nasty if you don't want to unnecessarily slow down the common case.

Basicly, for all normal formats the number of sectors per track, and the number of bytes per sector stay constant for the whole disk. The hardware doesn't enforce this, but almost all existing software seems to rely on it.

Usually it doesn't make sense to use unusual disk formats because other OSs wouldn't be able to read them. However if you want to make sure another OS can't read your disks it can be a good idea - it wasn't that uncommon for non-sequential sector numbers to be used for copy protection in old DOS software (where you couldn't copy the disk with standard software because standard software expected sequential sectors).

That's an awful way to do copy protection. It does mean that whoever wants to copy the disks needs to figure out the format, but you can detect that kind of stuff by trying to read it in various ways. The non-sequential sector numbers is probably the only thing I might consider worth supporting, and only because you get that as a side-effect if you write a driver that falls back into reading sector at a time if it has trouble reading track at a time, because that could be used to combat bad sectors as well (retry only those sectors that fail).

There is another purpose for all of this - increased disk capacities.

Yes. In fact, it seems like GRUB for example seems to set the expected GAP to 0 when it's reading the disk. Again, I left this out of the first version of the tutorial, because it needlessly complicates the code, especially if you want to autodetect such disks.

It may be possible (I haven't tried it or done the calculations, but it seems likely) to format a floppy with three 4 KB sectors per track, giving 1920 KB on a standard 1440 KB disk

Linux floppy driver supports 1440, 1600, 1680, 1722, 1743, 1760, 1840 and 1920. This was discussed in another thread here. I've had success with no more than 1680 with my drive (IIRC), but some drives work with more.

[...] Of course DOS/Windows won't be able to read more than 18 KB (which can be a good thing).

I don't personally care for such trickery, but yes, it's possible.

I would be tempted to include some information on detecting what media is in the drive and if it's write protected or not. For example, for a disk drive that supports 2880 KB disks you could have a 2800 KB disk, a 1440 KB disk, a 720 KB disk, a 360 KB disk, a "custom" format disk, or nothing.

Planned to be included with media-change detection into some future version, once I've got reliable code in my own kernel to use as an example. I don't wanna go writing tutorials with code that hasn't been properly tested.

DOS/Windows uses a data table in the first sector, called the BPB (or "BIOS Parameter Block", [...]

I'd just leave it as a FAT issue, and ignore it in the floppy driver.

For resources, there is nothing to say additional floppy controllers can't (or won't) use different DMA channels and IRQs. Most ISA "I/O expansion cards" have jumpers that are used to set the I/O ports, DMA and IRQ used by the floppy controller.

Ok, I'll word it differently, and simply say that you'll normally see one controller with DMA2/IRQ6, and be done with it. The target audience of my tutorial is people that want to get their OS to read/write normal floppies on a normal PC, so that they can have some storage available on machines they want to test their OS with. I should probably state that.

More accurately, [...]

Yeah ok. I did some quick calculations to get some approximate numbers in order to demonstrate the point. I'll see if I can be bothered to fix that.

The main benefit of caching entire cylinders is that you eliminate the need for seeks (once a cylinder is in the cache you never need to seek to it again, except for writes which are less common). The other benefit is that it makes it easy to support truly arbitrary transfers (e.g. read 123 bytes starting from the 12345th byte).

You also get some amount of read-ahead for free. Sure, if the filesystem is fragemented, that won't necessarily help much, but it's better than nothing. Plus you can easily cache the whole disk in memory on modern systems if you want, so you could just do write-back with 1.44MB (or whatever the size) cache. If you do that, it makes sense to populate the cache in large blocks. The point about reducing seeks is good though, because that's indeed the slowest part with random per-sector access.

The "skip deleted" flag (and deleted sectors in general) is a method of avoiding bad sectors. For an example, instead of formatting a floppy with 18 sectors per track numbered 1 to 18, you could format it with 18 sectors per track where the first 17 sectors are used (number 1 to 17) and the last is unused (numbered 255 perhaps). When you detect a bad sector (e.g. when you're verifying the data after a write) you could do a "read track" followed by "format track", "write track" and then a "write deleted"; so that the track ends up with the bad sector marked as "deleted" and the last (previously unused) sector is used to make up the difference. In this case the floppy hardware will skip the bad sector and the disk driver can continue using sectors 1 to 17 like normal.

Yeah ok, I thought of that possibility as well. Doesn't seem like a good idea to me: floppies are cheap, and once a sector fails, it's likely there'll be more failing sectors in the future, so it's better to just try to save the data to a fresh disk if possible, and throw the problematic into the recycle bin.

AFAIK the deleted flag isn't actually used by other OSs. This makes it a viable alternative for data hiding (e.g. have a hidden sector marked as "deleted" containing data that only your OS knows about).

Ok, thanks for the info. As for data hiding, I think it's kinda pointless...

[mystran]

I'll see if I can do a proper revision of the tutorial tomorrow. Today I'm going to work on my own OS.

[Brendan]

Hi,

Usually it doesn't make sense to use unusual disk formats because other OSs wouldn't be able to read them. However if you want to make sure another OS can't read your disks it can be a good idea - it wasn't that uncommon for non-sequential sector numbers to be used for copy protection in old DOS software (where you couldn't copy the disk with standard software because standard software expected sequential sectors).

That's an awful way to do copy protection. It does mean that whoever wants to copy the disks needs to figure out the format, but you can detect that kind of stuff by trying to read it in various ways. The non-sequential sector numbers is probably the only thing I might consider worth supporting, and only because you get that as a side-effect if you write a driver that falls back into reading sector at a time if it has trouble reading track at a time, because that could be used to combat bad sectors as well (retry only those sectors that fail).

Awful, but possible. Detecting the actual format can be very difficult, especially if a combination of methods is used - you'd need to write custom trial and error code, which is well beyond the abilities of most casual users. AFAIK there is no other way to prevent a disk from being copied - without some non-standard mangling, something as simple as "cat /dev/fd0 > /dev/fd1" or a few minutes with RAWwrite works (even if the data itself is heavily encrypted).

It may be possible (I haven't tried it or done the calculations, but it seems likely) to format a floppy with three 4 KB sectors per track, giving 1920 KB on a standard 1440 KB disk

Linux floppy driver supports 1440, 1600, 1680, 1722, 1743, 1760, 1840 and 1920. This was discussed in another thread here. I've had success with no more than 1680 with my drive (IIRC), but some drives work with more.

I found some interesting information regarding this that is definately worth reading if you're interested in "extreme floppy techniques". It also seems the Linux floppy driver is far more advanced than the very conservative "/dev/fdprm" file on my computer indicates, and there's a reason for supporting unusual floppy formats that I didn't think of - disk formats from non-80x86 systems. For e.g. I thought it would've been impossible to read the CBM1541 format disks (from the days of Commodore 64) on any 80x86 system, but it appears to be fully supported by Linux.

AFAIK the deleted flag isn't actually used by other OSs. This makes it a viable alternative for data hiding (e.g. have a hidden sector marked as "deleted" containing data that only your OS knows about).

Ok, thanks for the info. As for data hiding, I think it's kinda pointless...

I probably wouldn't bother using the "deleted" flag for data hiding (or any other purpose). However I can't guarantee that every OS developer will never use the deleted flag for data hiding (or any other purpose)...


Cheers,

Brendan

[mystran]

Awful, but possible. Detecting the actual format can be very difficult, especially if a combination of methods is used - you'd need to write custom trial and error code, which is well beyond the abilities of most casual users. AFAIK there is no other way to prevent a disk from being copied - without some non-standard mangling, something as simple as "cat /dev/fd0 > /dev/fd1" or a few minutes with RAWwrite works (even if the data itself is heavily encrypted).

Actually, I can see a good reason NOT to support such things in a driver: copy-protections like that are evil, hence it need not be possible to use such tricks under my OS. OTOH, if the disk has copy-protection, I don't see a reason to help anyone copy it.

I found some interesting information regarding this that is definately worth reading if you're interested in "extreme floppy techniques". It also seems the Linux floppy driver is far more advanced than the very conservative "/dev/fdprm" file on my computer indicates, and there's a reason for supporting unusual floppy formats that I didn't think of - disk formats from non-80x86 systems. For e.g. I thought it would've been impossible to read the CBM1541 format disks (from the days of Commodore 64) on any 80x86 system, but it appears to be fully supported by Linux.

I'm going to insist though, that such things are beyond the scope of my tutorial.

[piranha]

How do you write an IRQ_wait routine?

Do you have an example?