OSDev.org

Hi guys,
since some of you already know that I recently finished my floppy driver, I'm dissapointed with one thing : DMA.

I can't believe that DMA could not take more than 16-bit long memory, so basically I could not make use of my allocator and stack for floppy disk purposes. (That probably limits other devices which work on DMA).

Is there any way to avoid this? I mean to read full 32-bit addresses?

Busmastering DMA uses 32-bit addresses.

Lukand wrote:(That probably limits other devices which work on DMA).

The limits only apply to ISA DMA, not other types of DMA.

Lukand wrote:Is there any way to avoid this? I mean to read full 32-bit addresses?

Use a newer device that doesn't rely on ISA DMA, like a USB floppy drive.

Better still, forget obsolete technology.

Is there any hack to make floppy use busmastering DMA?

Lukand wrote:Is there any hack to make floppy use busmastering DMA?

Octocontrabass wrote:Use a newer device that doesn't rely on ISA DMA, like a USB floppy drive.

Hi,

Lukand wrote:since some of you already know that I recently finished my floppy driver, I'm dissapointed with one thing : DMA.

I can't believe that DMA could not take more than 16-bit long memory, so basically I could not make use of my allocator and stack for floppy disk purposes. (That probably limits other devices which work on DMA).

Is there any way to avoid this? I mean to read full 32-bit addresses?

The ISA DMA controller supports 24-bit addressing (the first 16 MiB of the physical address space).

For more than that, if you have an IOMMU you should be able to map any memory into the 16 MiB area that the ISA DMA thinks it's using. However; computers old enough to have floppy are probably too old to support IOMMU, so it's probably not worth the hassle of supporting this. The other alternative is to use PIO, which very slow/wasteful (you literally waste a huge amount of CPU time waiting for each read to happen) but doesn't have any addressing restrictions.

Typically, you just allocate a single buffer (large enough for an entire floppy track - e.g. 21*512 bytes rounded up to page size), and then copy data between the buffer and your disk cache whenever necessary.

Also note that (because floppies are slow and relatively unreliable) it might be a good idea for the OS to create/maintain a "floppy disk image catalogue" in memory or on hard disk. In this case, when a floppy disk is inserted you'd read the first track, generate a hash (md5?), and use the hash to find the corresponding floppy disk image in the catalogue; and if there is no image in the catalogue (the floppy hasn't been seen before) you'd read the entire floppy and create a catalogue entry. This means that the user would be able to keep using many floppies even when they aren't present/inserted, and could create new images without using an actual floppy (like the way we create ISO images for CDs), and "burn" a floppy image to a floppy disk whenever they feel like it.


Cheers,

Brendan

Brendan wrote: Also note that (because floppies are slow and relatively unreliable) it might be a good idea for the OS to create/maintain a "floppy disk image catalogue" in memory or on hard disk. In this case, when a floppy disk is inserted you'd read the first track, generate a hash (md5?), and use the hash to find the corresponding floppy disk image in the catalogue;

Well that's just broken. Then what happens when a user puts the disk in another computer and modifies a file (i.e. nothing on the first track; can happen if the file is in a subdirectory and the size doesn't change or on a non-FAT filesystem)? This is almost guaranteed to happen at some point.

Also, physical floppies are more common than you may think (or have any right to be). USB floppy drives use the SCSI command set and therefore cannot read any non-standard formats; most can't even be used with 720KB disks, let alone things like Microsoft's DMF or IBM's XDF "extended" format disks that much early-90s software was distributed on. Anybody who's interested in "retro computing" will have at least one relatively modern system with a standard floppy drive. It's far cheaper and easier to use a standard floppy drive than a specialized "KryoFlux" type device (which is read-only anyway). Note that the sets of alternative-OS hobbyists and retro enthusiasts probably have significant crossover...

Finding a brand-new motherboard with a floppy controller isn't difficult at all, it took me mere seconds to locate one on a popular online retailer's website.

Hi,

mallard wrote:

Brendan wrote:Also note that (because floppies are slow and relatively unreliable) it might be a good idea for the OS to create/maintain a "floppy disk image catalogue" in memory or on hard disk. In this case, when a floppy disk is inserted you'd read the first track, generate a hash (md5?), and use the hash to find the corresponding floppy disk image in the catalogue;

Well that's just broken. Then what happens when a user puts the disk in another computer and modifies a file (i.e. nothing on the first track; can happen if the file is in a subdirectory and the size doesn't change or on a non-FAT filesystem)? This is almost guaranteed to happen at some point.

Who cares?

99.9% of users won't use floppy at all, 0.0999% of users won't come across the problem (either because they mostly only use the images from the catalogue or because they don't use another OS or because they weren't very unlucky), and for the remaining 0.00001% of users you can have a "refresh image in catalogue from disk" button.

mallard wrote:Also, physical floppies are more common than you may think (or have any right to be).

Nonsense. I wouldn't be too surprised if half the people who have logged into the osdev forums in the last 30 days have never actually seen a real floppy disk; and this is a drastically skewed sample (not your average users).

mallard wrote:Finding a brand-new motherboard with a floppy controller isn't difficult at all, it took me mere seconds to locate one on a popular online retailer's website.

Sure, and searching for a motherboard that supports floppy (and then finding an actual floppy drive, and finding that last remaining company on earth that can still provide floppy disks to go in the drive) is something that every normal user does when purchasing their next media centre, NAS, web server or smartphone.


Cheers,

Brendan

Brendan wrote:99.9% of users won't use floppy at all, 0.0999% of users won't come across the problem

I'm certain that more than 1% of people who use floppies actually do use floppies. If floppy drives are as rare as you claim, the whole "cached image" subsystem is an awful lot of work to do for a small niche of users when the hardware can be supported as an ordinary block device... As-capable-as-the-device-itself support is vastly better than completely broken support. Re-imaging an entire floppy disk every time it's changed is an incredibly bad idea, it'll vastly reduce the lifespan of already somewhat fragile media! Trying to improve on the hardware using software is insane.

Brendan wrote:because they don't use another OS

Nobody is ever, ever, in a zillion years going to be using your OS exclusively. Even if it manages to reach the popularity of Linux (by far the most popular OS with "hobbyist" roots), you're still going to find at least 99% of users aren't using it exclusively. You must consider interoperability. The "re-invent the universe" approach is fundamentally broken.

Brendan wrote:I wouldn't be too surprised if half the people who have logged into the osdev forums in the last 30 days have never actually seen a real floppy disk

So, by your own admission, around half of your potential initial userbase (software/osdev enthusiasts) would be interested in floppy support (maybe slightly less since, admittedly "seen a floppy disk" != "has a floppy disk drive").

Brendan wrote:Sure, and searching for a motherboard that supports floppy (and then finding an actual floppy drive, and finding that last remaining company on earth that can still provide floppy disks to go in the drive) is something that every normal user does when purchasing their next media centre, NAS, web server or smartphone.

And when testing out a new, hobbyist OS, the first thing somebody does is try and install it on their "next media centre, NAS, web server or smartphone", right? Rather than using the 10+ year old PC that they're not using for anything important and, due to having a BIOS that can only be updated from DOS, has a built-in floppy drive... I've said it before and I'm saying it again; target your OS at last-generation (at best) hardware (plus VMs), at least until you have an actual userbase. Nobody is going to install an untried OS with next to no applications on a brand-new system.

I've seen and used thousands of floppy disks of all sizes (8" to 3"). I am most definitely one of the 50% who have seen a floppy disk - I still have a collection of them.

But I haven't used a floppy disk for at least 5 years. Unless I were writing an OS for a retro computer (such as the Amiga 1000 discussed in another thread) I wouldn't dream of including floppy disk support. No more would I support parallel-port tape drives, zip drives, or ferrite-core memory.

Hi,

mallard wrote:

Brendan wrote:99.9% of users won't use floppy at all, 0.0999% of users won't come across the problem

I'm certain that more than 1% of people who use floppies actually do use floppies.

For the 0% of people that still use real floppies (rather than virtual floppies in emulators), they'd be for firmware updates for ancient computers and/or nostalgia.

mallard wrote:If floppy drives are as rare as you claim, the whole "cached image" subsystem is an awful lot of work to do for a small niche of users when the hardware can be supported as an ordinary block device...

No; you just aren't smart enough to tell the difference between "using real floppies" and "using virtual floppies" (e.g. inside emulators, etc). Also note that it's probably easier to only support whole disk copies than it is to support arbitrary block access and disk insertion/removal.

mallard wrote:As-capable-as-the-device-itself support is vastly better than completely broken support. Re-imaging an entire floppy disk every time it's changed is an incredibly bad idea, it'll vastly reduce the lifespan of already somewhat fragile media! Trying to improve on the hardware using software is insane.

Obviously you wouldn't re-image the floppy every time you write, you'd only update the image in the catalogue. The only time you'd ever re-image a floppy is when you need to move the image to a different computer (and the network is down, and you can't find a USB flash drive).

mallard wrote:

Brendan wrote:because they don't use another OS

Nobody is ever, ever, in a zillion years going to be using your OS exclusively. Even if it manages to reach the popularity of Linux (by far the most popular OS with "hobbyist" roots), you're still going to find at least 99% of users aren't using it exclusively. You must consider interoperability. The "re-invent the universe" approach is fundamentally broken.

I have no idea why you're suddenly talking about my OS (rather than the idea of a floppy disk catalogue). Perhaps you're just having difficulty containing the low self-confidence that you've obtained from years of having no skills, no imagination and no ambition?

mallard wrote:

Brendan wrote:I wouldn't be too surprised if half the people who have logged into the osdev forums in the last 30 days have never actually seen a real floppy disk

So, by your own admission, around half of your potential initial userbase (software/osdev enthusiasts) would be interested in floppy support (maybe slightly less since, admittedly "seen a floppy disk" != "has a floppy disk drive").

I'd say maybe 75% would be interested in virtual floppy disk support for emulators; but that has nothing at all to do with actually seeing a real floppy disk and holding it in your own 2 hands.

mallard wrote:

Brendan wrote:Sure, and searching for a motherboard that supports floppy (and then finding an actual floppy drive, and finding that last remaining company on earth that can still provide floppy disks to go in the drive) is something that every normal user does when purchasing their next media centre, NAS, web server or smartphone.

And when testing out a new, hobbyist OS, the first thing somebody does is try and install it on their "next media centre, NAS, web server or smartphone", right? Rather than using the 10+ year old PC that they're not using for anything important and, due to having a BIOS that can only be updated from DOS, has a built-in floppy drive... I've said it before and I'm saying it again; target your OS at last-generation (at best) hardware (plus VMs), at least until you have an actual userbase. Nobody is going to install an untried OS with next to no applications on a brand-new system.

Heh, no. You'd have to go back at least 15 years to find hardware that actually came with floppy drive.

Also note that I do test on my newest desktop machine regularly (and several other computers that are too new to have a floppy drive, even though I used to deliberately buy computers with floppy drives for OS development). I even bought a USB floppy drive about 8 years ago (specifically for creating floppies for OS development, because I upgraded my development machine and the new computer didn't support floppy) and this USB floppy drive has still never been plugged into any computer.

Finally; the first thing you do with an OS is test it on whatever it's intended to be used for. If you intend for your OS to be used on servers you test it on servers, if you intend for your OS to be used on smartphones you test it on smartphones, etc. If you intend for your OS to be used on hardware that is 10+ years old now (e.g. hardware that will be 20+ years old by the time your OS is usable), then you're officially writing an OS that nobody (except you) can ever care about.


Cheers,

Brendan

Brendan wrote: No; you just aren't smart enough to tell the difference between "using real floppies" and "using virtual floppies" (e.g. inside emulators, etc). Also note that it's probably easier to only support whole disk copies than it is to support arbitrary block access and disk insertion/removal.

Sure, I'm not "smart enough" to have a clue what you're on about here. I'm also not clairvoyant. Both "real" and "virtual" floppies are best supported as ordinary block devices. The driver talks to what appears to be a standard floppy disk controller. Your OS shouldn't care whether it's running in an emulator/VM or not. Going to a whole lot of effort to implement an "image catalog" for a niche use-case is utterly stupid. If someone wants that sort of functionality, it can be implemented in an ordinary user application (something like WinImage for example).

Brendan wrote:Obviously you wouldn't re-image the floppy every time you write, you'd only update the image in the catalogue. The only time you'd ever re-image a floppy is when you need to move the image to a different computer (and the network is down, and you can't find a USB flash drive).

In order to actually use a floppy, your insane mechanism requires the entire disk to be read whenever it's been changed on another computer or written when it's been changed on this one. If somebody is using floppies at all, they're probably using the floppy to move data to/from an old, non-networked computer or to do something like a BIOS update. If they're using disk images to move data in/out of an emulator/VM then that's a completely different use-case and irrelevant to this discussion. Forcing all access to a physical floppy to go via complete rewrites/rereads of the disk is a great way to destroy people's disks and frustrate them with a multi-minute write process after they've (for example) made a trivial change to a configuration file.

Brendan wrote: I have no idea why you're suddenly talking about my OS (rather than the idea of a floppy disk catalogue). Perhaps you're just having difficulty containing the low self-confidence that you've obtained from years of having no skills, no imagination and no ambition?

I have no idea why you're suddenly insulting me (rather than discussing the merits of a "floppy disk catalogue"). Perhaps you don't like having your delusions of grandeur challenged? You worded your response as though you expected that users would be using your OS exclusively and that interoperability with other systems was thus unimportant. I pointed out how ridiculous that idea is. No one-man OS project has ever achieved any significant measure of popularity. Linux is the closest we've seen and Linus only really wrote a kernel; the userspace already existed (the GNU project) and it didn't really get popular until there were many hundreds of contributors to the kernel anyway. Since you appear to have no intention of encouraging third-party contributions to your OS, I assume you've got a team of thousands and a budget in the tens of millions (of USD/EUR/GBP or generally similarly valued currency) at least then?

Brendan wrote:I'd say maybe 75% would be interested in virtual floppy disk support for emulators; but that has nothing at all to do with actually seeing a real floppy disk and holding it in your own 2 hands.

What do you mean by "for emulators"? Do you mean running the OS in an emulator/VM or running an emulator on your OS? For the latter, floppy disk (image) support in entirely in the hands of user applications. For the former, why do you care whether the it's real or not?

Brendan wrote:Heh, no. You'd have to go back at least 15 years to find hardware that actually came with floppy drive.

15 was greater than 10 (i.e. within the definition of "10+") last time I checked...

Brendan wrote:Also note that I do test on my newest desktop machine regularly (and several other computers that are too new to have a floppy drive, even though I used to deliberately buy computers with floppy drives for OS development).

You're the developer, of course you do. It's entirely something else to convince potential users to abandon mainstream software on hardware they've spent significant money on. Convincing them to try your OS on their old, "obsolete", basically worthless hardware is going to be far easier. Once significant numbers have tried your OS, liked it, written software for it, etc. then they'll possibly be willing to try it on new hardware.

Brendan wrote:I even bought a USB floppy drive about 8 years ago (specifically for creating floppies for OS development, because I upgraded my development machine and the new computer didn't support floppy) and this USB floppy drive has still never been plugged into any computer.

I bought a USB floppy drive about 10 years ago orignally because I needed to set up DOS bootdisks for old systems and didn't have such a drive on my laptop. I bought a standard internal drive about 3 years ago when I was building a "retro gaming PC" using spare parts I had lying around. A couple of my other systems (such as the Dell P4 "junker" I use for testing my OS on real hardware, an ancient 80386sx laptop I use as a serial terminal for debugging my OS on real hardware and even one of my rackmount servers) also have floppy drives. The USB drive gets regular use (especially since it's luckily one of the few models that does support 720KB disks) for exchanging data with my Atari ST systems and the "retro PC" is the go-to machine whenever I encounter a non-standard format disk (a few times a year). I'm definitely not an "average" user, but I'm not probably a fairly light user of the "retro computing" group.

Brendan wrote:Finally; the first thing you do with an OS is test it on whatever it's intended to be used for. If you intend for your OS to be used on servers you test it on servers, if you intend for your OS to be used on smartphones you test it on smartphones, etc.

I agree. My initial use-cases are all around "reviving" otherwise "obsolete" hardware, because I'm not deluded enough to think that I can convince them to run my OS on new hardware. Also because older hardware is generally better understood and more "standard" than brand-new equipment. Only when (if) I have a significant userbase (and other contributors) will I even think about chasing the cutting-edge. Therefore, most of my real-hardware testing is currently done on an old Dell P4 system with 256MB RAM. Other "target" systems that I have include an OEM-white-labelled variant of the "Pepper Pad" tablet (533Mhz CPU, 256MB RAM, ~2006 vintage) and an 86duno (300Mhz CPU, 128MB RAM, in production since 2013). More powerful and greater RAM configurations (up to 4GB) are tested in VMs.

Brendan wrote:you're officially writing an OS that nobody (except you) can ever care about

I think you're mistaken on the definition of "officially" there. You're not in a position to say anything "official" about anybody else's work. What you mean is "in my humble opinion".

I use floppies as small flash drives (Yes I do have reader in my computer and have 5~ of them due to Serbian technological advancement)

They sound like a loud sawtooth.

>0,01% use floppy disks? I didn't knew that weed is allowed to smoke in UK (just kidding, don't take it seriously)
It is same like saying that nobody ever saw https://www.google.rs or https://thepiratebay.org/

Hi,

mallard wrote:

Brendan wrote:No; you just aren't smart enough to tell the difference between "using real floppies" and "using virtual floppies" (e.g. inside emulators, etc). Also note that it's probably easier to only support whole disk copies than it is to support arbitrary block access and disk insertion/removal.

Sure, I'm not "smart enough" to have a clue what you're on about here. I'm also not clairvoyant. Both "real" and "virtual" floppies are best supported as ordinary block devices. The driver talks to what appears to be a standard floppy disk controller. Your OS shouldn't care whether it's running in an emulator/VM or not. Going to a whole lot of effort to implement an "image catalog" for a niche use-case is utterly stupid. If someone wants that sort of functionality, it can be implemented in an ordinary user application (something like WinImage for example).

Sure, why do a nice stream-lined system when you can have a slower, less reliable and less flexible system, plus an eclectic mixture of tools (where various emulators have their own floppy disk catalogues, then additional tools to create and use floppy images).

mallard wrote:

Brendan wrote:Obviously you wouldn't re-image the floppy every time you write, you'd only update the image in the catalogue. The only time you'd ever re-image a floppy is when you need to move the image to a different computer (and the network is down, and you can't find a USB flash drive).

In order to actually use a floppy, your insane mechanism requires the entire disk to be read whenever it's been changed on another computer or written when it's been changed on this one.

You're just rephrasing what I've said here.

mallard wrote:If somebody is using floppies at all, they're probably using the floppy to move data to/from an old, non-networked computer or to do something like a BIOS update. If they're using disk images to move data in/out of an emulator/VM then that's a completely different use-case and irrelevant to this discussion. Forcing all access to a physical floppy to go via complete rewrites/rereads of the disk is a great way to destroy people's disks and frustrate them with a multi-minute write process after they've (for example) made a trivial change to a configuration file.

Except; nobody makes trivial changes to a config file. They download a "firmware update" disk image and write the entire thing to a disk, or read an entire disk to create an image to use in a virtual machine, or wipe/format a disk and fill it with files and read all the files on another computer. For all of these cases you're reading/writing the whole disk anyway; and it's faster, less frustrating and causes less wear to do it in a nice sequential order that minimises seeks (including pre-preparing the image and arranging files like people having been doing for CD burners for years) than it is to do "random access".

mallard wrote:

Brendan wrote:I have no idea why you're suddenly talking about my OS (rather than the idea of a floppy disk catalogue). Perhaps you're just having difficulty containing the low self-confidence that you've obtained from years of having no skills, no imagination and no ambition?

I have no idea why you're suddenly insulting me (rather than discussing the merits of a "floppy disk catalogue").

If you have no idea, let me make it perfectly clear. You are a worthless butt-hole that has used a flimsy and off-topic excuse to criticise my project (without provocation, and not for the first time) for no reason other than the fact that my design goals differ from yours; so now I'm treating you the way you deserve to be treated.

mallard wrote:

Brendan wrote:Heh, no. You'd have to go back at least 15 years to find hardware that actually came with floppy drive.

15 was greater than 10 (i.e. within the definition of "10+") last time I checked...

Yes; but "10+" was chosen to be deliberately misleading. Specifically; it was chosen to make it sound like people (who aren't OS developers but might want to try an alternative OS) are likely to have an old computer that has a floppy drive, and that therefore there's a valid reason for an OS to support a floppy drive for these users. This is false. People (who aren't OS developers but might want to try an alternative OS) are not likely to still have a computer that is so ancient that it actually has a floppy drive.

mallard wrote:

Brendan wrote:Also note that I do test on my newest desktop machine regularly (and several other computers that are too new to have a floppy drive, even though I used to deliberately buy computers with floppy drives for OS development).

You're the developer, of course you do. It's entirely something else to convince potential users to abandon mainstream software on hardware they've spent significant money on. Convincing them to try your OS on their old, "obsolete", basically worthless hardware is going to be far easier. Once significant numbers have tried your OS, liked it, written software for it, etc. then they'll possibly be willing to try it on new hardware.

Your fantasy might have been plausible 10 years ago, when the "old computer" people kept in the back of the cupboard actually had a floppy disk. Now, that "old computer" is a 64-bit machine with USB 2.0 and no floppy.

However; "plausible 10 years ago" does not mean "actually realistic 10 years ago". For the majority of users, dragging the old computer out from storage and plugging it in is just too much hassle. It's more likely that they'll test on something that they still use regularly simply because it's already in front of them; and even more likely that they simply won't test it at all.

mallard wrote:

Brendan wrote:I even bought a USB floppy drive about 8 years ago (specifically for creating floppies for OS development, because I upgraded my development machine and the new computer didn't support floppy) and this USB floppy drive has still never been plugged into any computer.

I bought a USB floppy drive about 10 years ago orignally because I needed to set up DOS bootdisks for old systems and didn't have such a drive on my laptop. I bought a standard internal drive about 3 years ago when I was building a "retro gaming PC" using spare parts I had lying around. A couple of my other systems (such as the Dell P4 "junker" I use for testing my OS on real hardware, an ancient 80386sx laptop I use as a serial terminal for debugging my OS on real hardware and even one of my rackmount servers) also have floppy drives. The USB drive gets regular use (especially since it's luckily one of the few models that does support 720KB disks) for exchanging data with my Atari ST systems and the "retro PC" is the go-to machine whenever I encounter a non-standard format disk (a few times a year). I'm definitely not an "average" user, but I'm not probably a fairly light user of the "retro computing" group.

And you think you are a typical user and that everyone collects retro hardware; and that people who do collect retro hardware are desperate for a new OS because they don't like old things?

mallard wrote:

Brendan wrote:Finally; the first thing you do with an OS is test it on whatever it's intended to be used for. If you intend for your OS to be used on servers you test it on servers, if you intend for your OS to be used on smartphones you test it on smartphones, etc.

I agree. My initial use-cases are all around "reviving" otherwise "obsolete" hardware, because I'm not deluded enough to think that I can convince them to run my OS on new hardware. Also because older hardware is generally better understood and more "standard" than brand-new equipment. Only when (if) I have a significant userbase (and other contributors) will I even think about chasing the cutting-edge. Therefore, most of my real-hardware testing is currently done on an old Dell P4 system with 256MB RAM. Other "target" systems that I have include an OEM-white-labelled variant of the "Pepper Pad" tablet (533Mhz CPU, 256MB RAM, ~2006 vintage) and an 86duno (300Mhz CPU, 128MB RAM, in production since 2013). More powerful and greater RAM configurations (up to 4GB) are tested in VMs.

You'll think about solving the "very few people have ancient hardware" problem *after* you have a significant userbase that can't exist because a significant number of people don't have ancient hardware? Awesome.


Cheers,

Brendan