Page 1 of 2
Reading and Writing HDD
Posted: Mon Aug 30, 2021 9:39 am
by thedude3253
Hey all,
I am getting ready to want to be able to read and write to a disk. I'm currently reading up on SATA specifications but before I get too deep into it I wanted to poll the community. Can your OS read and write data to a disk? If so how? Do you jump back and forth between real and protected mode to use INT 13h? or did you write a driver for communicating with the hardware? What kinds of data storage do you have support for?
I'm in new territory here so I kind of wanted to get a feel for how other people tackled the same problem before I end up going down any pointless rabbit holes
Re: Reading and Writing HDD
Posted: Mon Aug 30, 2021 10:21 am
by iansjack
Write a driver. It's one of the easiest bits of hardware to control.
Re: Reading and Writing HDD
Posted: Mon Aug 30, 2021 10:48 am
by Klakap
https://wiki.osdev.org/ATA_PIO_Mode
Here is everything you need to know to start writing driver for HDD.
There are two widely used types of communication with HDD. It is IDE and AHCI. So do not waste time in reading SATA spec now. You need it only for FIS types when you will write driver for AHCI.
Re: Reading and Writing HDD
Posted: Mon Aug 30, 2021 11:24 am
by foliagecanine
I would definitely agree with iansjack: writing a driver is the way to go.
Early on, writing an
IDE driver may be easier just to get the ability to read disks especially in emulators like QEMU, but I'd actually recommend writing an
AHCI driver instead.
It isn't terribly hard (compared to something like USB controllers) and works with pretty much any computer since 2010.
Plus, AFAIK the spec is completely free from Intel.
If you're using QEMU you can add an AHCI controller and a disk like so:
Code: Select all
(qemu command) -device ahci,id=ahci -drive file=DISK_IMAGE.img,id=disk,if=none,format=raw -device ide-hd,drive=disk,bus=ahci.0
thedude3253 wrote:Do you jump back and forth between real and protected mode to use INT 13h?
The only thing I ever use real mode/BIOS interrupts for at all anymore is VESA/VBE resolution switches. Switching is slow, especially for disk reads/writes which ideally should be somewhat asynchronous from the OS's perspective (but synchronous from the program's perspective). The only reason I have to do this for VESA/VBE is because I don't feel like writing a driver for tons of graphics cards and I really want the user of my OS to be able to switch resolutions without rebooting their computer.
Good luck!
Re: Reading and Writing HDD
Posted: Mon Aug 30, 2021 11:40 am
by nexos
foliagecanine wrote:Plus, AFAIK the spec is completely free from Intel.
Yes, but some structures (i.e. the FIS) or documented in the SATA spec, which costs $75 for SATA 3.4 and 3.5, $25 dollars for older versions.
Re: Reading and Writing HDD
Posted: Mon Aug 30, 2021 12:06 pm
by thedude3253
nexos wrote:foliagecanine wrote:Plus, AFAIK the spec is completely free from Intel.
Yes, but some structures (i.e. the FIS) or documented in the SATA spec, which costs $75 for SATA 3.4 and 3.5, $25 dollars for older versions.
So I'm ignorant, what is the FIS? In addition, why would they lock specs behind a paywall? As a userspace dev I'm not very familiar with the economics of these kinds of things
Edit: I spent 5 seconds on the AHCI osdev wiki page and figured out what FIS is lol
Re: Reading and Writing HDD
Posted: Tue Aug 31, 2021 2:43 pm
by Ethin
TBH I doubt there's a justifiable reason for them locking the specs behind paywalls other than the fact that these are usually non-profit companies. I mean, at least they aren't as bad as PCI-SIG: you want a spec from them, you gotta pay $3,000.00-$5,000.00 -- and I don't even think that what you get is a digital copy.
Re: Reading and Writing HDD
Posted: Tue Aug 31, 2021 2:57 pm
by nexos
Luckily, there are comprehensive PCI books out there. Look at the company Mindshare on Amazon, and they have rather expensive books about PCI and PCIe. But 40 dollars is a lot better then 3000 dollars!
Re: Reading and Writing HDD
Posted: Thu Sep 02, 2021 5:05 am
by rdos
Before you write a disc driver you need to figure out how it should interface with the OS. You need to figure out how you handle caching of disc sectors.
My old disc drivers passed a buffer of sectors descriptors that contained sector # and the linear buffer. My new disc drivers pass a start sector, sector count, and an array of physical addresses to use as buffers.
Modern hardware (AHCI, USB) uses physical addresses in schedules as sources & destinations in DMA operations, and so there is no reason to pass linear addresses to those.
Using the BIOS int 13 interface never was a good idea, and it certainly isn't today when EFI is part of many computers.
Re: Reading and Writing HDD
Posted: Thu Sep 02, 2021 9:58 am
by Ethin
rdos wrote:Before you write a disc driver you need to figure out how it should interface with the OS. You need to figure out how you handle caching of disc sectors.
My old disc drivers passed a buffer of sectors descriptors that contained sector # and the linear buffer. My new disc drivers pass a start sector, sector count, and an array of physical addresses to use as buffers.
Modern hardware (AHCI, USB) uses physical addresses in schedules as sources & destinations in DMA operations, and so there is no reason to pass linear addresses to those.
Using the BIOS int 13 interface never was a good idea, and it certainly isn't today when EFI is part of many computers.
I mean, not really. Don't over-complicate the objective. If you just want to read and write data (and for writing you have the VWC if supported) you needn't worry about caching of anything.
Re: Reading and Writing HDD
Posted: Thu Sep 02, 2021 10:05 am
by nexos
Ethin wrote:you needn't worry about caching of anything.
For SSDs, you're right, block caching is unnecessary as an SSD is very fast. But for hard disks or floppies, caching is very important. Think of it as the TLB. A block buffer cache is the disk equivalent of the TLB. Without it, memory access would always have to access memory, and things would be quite slow. Same thing for disk caches. Instead of reading / writing to disk every time, instead you only write when necessary. Without a disk cache, the system would be unacceptably slow.
Re: Reading and Writing HDD
Posted: Thu Sep 02, 2021 10:59 am
by iansjack
And it's particularly important to cache meta-data.
Re: Reading and Writing HDD
Posted: Thu Sep 02, 2021 12:43 pm
by nullplan
nexos wrote:For SSDs, you're right, block caching is unnecessary as an SSD is very fast. But for hard disks or floppies, [...] Without a disk cache, the system would be unacceptably slow.
iansjack wrote:And it's particularly important to cache meta-data.
I agree with both of you in part and disagree in part. Yes, caching is important, however, the engineering principle to remember here is to first make it work, then make it work fast. "The design process is an iterative one." Let the OP first figure out how to actually transfer the data, then stick a cache in there. Whether that takes the form of a block cache or an FS-level metadata cache is secondary. In this case, I would focus my work after the raw data transfer into the parsing of partition tables, then file systems. And then, when that works, then I would look at caching.
Re: Reading and Writing HDD
Posted: Thu Sep 02, 2021 1:23 pm
by Ethin
nullplan wrote:nexos wrote:For SSDs, you're right, block caching is unnecessary as an SSD is very fast. But for hard disks or floppies, [...] Without a disk cache, the system would be unacceptably slow.
iansjack wrote:And it's particularly important to cache meta-data.
I agree with both of you in part and disagree in part. Yes, caching is important, however, the engineering principle to remember here is to first make it work, then make it work fast. "The design process is an iterative one." Let the OP first figure out how to actually transfer the data, then stick a cache in there. Whether that takes the form of a block cache or an FS-level metadata cache is secondary. In this case, I would focus my work after the raw data transfer into the parsing of partition tables, then file systems. And then, when that works, then I would look at caching.
This is exactly what I was trying to say. Don't implement a cache when your just trying to read/write to disk. Once you've got an FS and partition layer in place, then worry about caching.
Re: Reading and Writing HDD
Posted: Thu Sep 02, 2021 1:59 pm
by rdos
Caching should not be part of the disc driver. It should be in a separate layer. However, it will determine how the disc is accessed. Generally, you should try to read / write as many sectors as possible in the same request, and avoid copying of data. The usual disc interface typically requires copying or using scatter-gather, which not all devices support. Therefore, it's more efficient to send physical addresses and make sure they have proper alignment. Something you get for free with a decent cache function.
IOW, if you do a lot of disc drivers like int 0x13, passing a buffer without alignment requirements, then you miht have trouble adapting this to a interface with better performance. So, you end up having to rewrite the disc drivers once more because you didn't plan properly.