Reading and Writing HDD

Question about which tools to use, bugs, the best way to implement a function, etc should go here. Don't forget to see if your question is answered in the wiki first! When in doubt post here.
thedude3253
Posts: 7
Joined: Wed Aug 25, 2021 12:16 pm

Reading and Writing HDD

Post by thedude3253 »

Hey all,

I am getting ready to want to be able to read and write to a disk. I'm currently reading up on SATA specifications but before I get too deep into it I wanted to poll the community. Can your OS read and write data to a disk? If so how? Do you jump back and forth between real and protected mode to use INT 13h? or did you write a driver for communicating with the hardware? What kinds of data storage do you have support for?

I'm in new territory here so I kind of wanted to get a feel for how other people tackled the same problem before I end up going down any pointless rabbit holes :lol:
My toy OS BaseDOS 8)
User avatar
iansjack
Member
Member
Posts: 4703
Joined: Sat Mar 31, 2012 3:07 am
Location: Chichester, UK

Re: Reading and Writing HDD

Post by iansjack »

Write a driver. It's one of the easiest bits of hardware to control.
Klakap
Member
Member
Posts: 297
Joined: Sat Mar 10, 2018 10:16 am

Re: Reading and Writing HDD

Post by Klakap »

https://wiki.osdev.org/ATA_PIO_Mode

Here is everything you need to know to start writing driver for HDD.

There are two widely used types of communication with HDD. It is IDE and AHCI. So do not waste time in reading SATA spec now. You need it only for FIS types when you will write driver for AHCI.
Last edited by Klakap on Mon Aug 30, 2021 11:25 am, edited 1 time in total.
foliagecanine
Member
Member
Posts: 148
Joined: Sun Aug 23, 2020 4:35 pm

Re: Reading and Writing HDD

Post by foliagecanine »

I would definitely agree with iansjack: writing a driver is the way to go.
Early on, writing an IDE driver may be easier just to get the ability to read disks especially in emulators like QEMU, but I'd actually recommend writing an AHCI driver instead.
It isn't terribly hard (compared to something like USB controllers) and works with pretty much any computer since 2010.
Plus, AFAIK the spec is completely free from Intel.
If you're using QEMU you can add an AHCI controller and a disk like so:

Code: Select all

(qemu command) -device ahci,id=ahci -drive file=DISK_IMAGE.img,id=disk,if=none,format=raw -device ide-hd,drive=disk,bus=ahci.0
thedude3253 wrote:Do you jump back and forth between real and protected mode to use INT 13h?
The only thing I ever use real mode/BIOS interrupts for at all anymore is VESA/VBE resolution switches. Switching is slow, especially for disk reads/writes which ideally should be somewhat asynchronous from the OS's perspective (but synchronous from the program's perspective). The only reason I have to do this for VESA/VBE is because I don't feel like writing a driver for tons of graphics cards and I really want the user of my OS to be able to switch resolutions without rebooting their computer.

Good luck!
My OS: TritiumOS
https://github.com/foliagecanine/tritium-os
void warranty(laptop_t laptop) { if (laptop.broken) return laptop; }
I don't get it: Why's the warranty void?
nexos
Member
Member
Posts: 1081
Joined: Tue Feb 18, 2020 3:29 pm
Libera.chat IRC: nexos

Re: Reading and Writing HDD

Post by nexos »

foliagecanine wrote:Plus, AFAIK the spec is completely free from Intel.
Yes, but some structures (i.e. the FIS) or documented in the SATA spec, which costs $75 for SATA 3.4 and 3.5, $25 dollars for older versions.
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg
thedude3253
Posts: 7
Joined: Wed Aug 25, 2021 12:16 pm

Re: Reading and Writing HDD

Post by thedude3253 »

nexos wrote:
foliagecanine wrote:Plus, AFAIK the spec is completely free from Intel.
Yes, but some structures (i.e. the FIS) or documented in the SATA spec, which costs $75 for SATA 3.4 and 3.5, $25 dollars for older versions.
So I'm ignorant, what is the FIS? In addition, why would they lock specs behind a paywall? As a userspace dev I'm not very familiar with the economics of these kinds of things

Edit: I spent 5 seconds on the AHCI osdev wiki page and figured out what FIS is lol
My toy OS BaseDOS 8)
Ethin
Member
Member
Posts: 625
Joined: Sun Jun 23, 2019 5:36 pm
Location: North Dakota, United States

Re: Reading and Writing HDD

Post by Ethin »

TBH I doubt there's a justifiable reason for them locking the specs behind paywalls other than the fact that these are usually non-profit companies. I mean, at least they aren't as bad as PCI-SIG: you want a spec from them, you gotta pay $3,000.00-$5,000.00 -- and I don't even think that what you get is a digital copy.
nexos
Member
Member
Posts: 1081
Joined: Tue Feb 18, 2020 3:29 pm
Libera.chat IRC: nexos

Re: Reading and Writing HDD

Post by nexos »

Luckily, there are comprehensive PCI books out there. Look at the company Mindshare on Amazon, and they have rather expensive books about PCI and PCIe. But 40 dollars is a lot better then 3000 dollars!
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg
rdos
Member
Member
Posts: 3297
Joined: Wed Oct 01, 2008 1:55 pm

Re: Reading and Writing HDD

Post by rdos »

Before you write a disc driver you need to figure out how it should interface with the OS. You need to figure out how you handle caching of disc sectors.

My old disc drivers passed a buffer of sectors descriptors that contained sector # and the linear buffer. My new disc drivers pass a start sector, sector count, and an array of physical addresses to use as buffers.

Modern hardware (AHCI, USB) uses physical addresses in schedules as sources & destinations in DMA operations, and so there is no reason to pass linear addresses to those.

Using the BIOS int 13 interface never was a good idea, and it certainly isn't today when EFI is part of many computers.
Ethin
Member
Member
Posts: 625
Joined: Sun Jun 23, 2019 5:36 pm
Location: North Dakota, United States

Re: Reading and Writing HDD

Post by Ethin »

rdos wrote:Before you write a disc driver you need to figure out how it should interface with the OS. You need to figure out how you handle caching of disc sectors.

My old disc drivers passed a buffer of sectors descriptors that contained sector # and the linear buffer. My new disc drivers pass a start sector, sector count, and an array of physical addresses to use as buffers.

Modern hardware (AHCI, USB) uses physical addresses in schedules as sources & destinations in DMA operations, and so there is no reason to pass linear addresses to those.

Using the BIOS int 13 interface never was a good idea, and it certainly isn't today when EFI is part of many computers.
I mean, not really. Don't over-complicate the objective. If you just want to read and write data (and for writing you have the VWC if supported) you needn't worry about caching of anything.
nexos
Member
Member
Posts: 1081
Joined: Tue Feb 18, 2020 3:29 pm
Libera.chat IRC: nexos

Re: Reading and Writing HDD

Post by nexos »

Ethin wrote:you needn't worry about caching of anything.
For SSDs, you're right, block caching is unnecessary as an SSD is very fast. But for hard disks or floppies, caching is very important. Think of it as the TLB. A block buffer cache is the disk equivalent of the TLB. Without it, memory access would always have to access memory, and things would be quite slow. Same thing for disk caches. Instead of reading / writing to disk every time, instead you only write when necessary. Without a disk cache, the system would be unacceptably slow.
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg
User avatar
iansjack
Member
Member
Posts: 4703
Joined: Sat Mar 31, 2012 3:07 am
Location: Chichester, UK

Re: Reading and Writing HDD

Post by iansjack »

And it's particularly important to cache meta-data.
nullplan
Member
Member
Posts: 1790
Joined: Wed Aug 30, 2017 8:24 am

Re: Reading and Writing HDD

Post by nullplan »

nexos wrote:For SSDs, you're right, block caching is unnecessary as an SSD is very fast. But for hard disks or floppies, [...] Without a disk cache, the system would be unacceptably slow.
iansjack wrote:And it's particularly important to cache meta-data.
I agree with both of you in part and disagree in part. Yes, caching is important, however, the engineering principle to remember here is to first make it work, then make it work fast. "The design process is an iterative one." Let the OP first figure out how to actually transfer the data, then stick a cache in there. Whether that takes the form of a block cache or an FS-level metadata cache is secondary. In this case, I would focus my work after the raw data transfer into the parsing of partition tables, then file systems. And then, when that works, then I would look at caching.
Carpe diem!
Ethin
Member
Member
Posts: 625
Joined: Sun Jun 23, 2019 5:36 pm
Location: North Dakota, United States

Re: Reading and Writing HDD

Post by Ethin »

nullplan wrote:
nexos wrote:For SSDs, you're right, block caching is unnecessary as an SSD is very fast. But for hard disks or floppies, [...] Without a disk cache, the system would be unacceptably slow.
iansjack wrote:And it's particularly important to cache meta-data.
I agree with both of you in part and disagree in part. Yes, caching is important, however, the engineering principle to remember here is to first make it work, then make it work fast. "The design process is an iterative one." Let the OP first figure out how to actually transfer the data, then stick a cache in there. Whether that takes the form of a block cache or an FS-level metadata cache is secondary. In this case, I would focus my work after the raw data transfer into the parsing of partition tables, then file systems. And then, when that works, then I would look at caching.
This is exactly what I was trying to say. Don't implement a cache when your just trying to read/write to disk. Once you've got an FS and partition layer in place, then worry about caching.
rdos
Member
Member
Posts: 3297
Joined: Wed Oct 01, 2008 1:55 pm

Re: Reading and Writing HDD

Post by rdos »

Caching should not be part of the disc driver. It should be in a separate layer. However, it will determine how the disc is accessed. Generally, you should try to read / write as many sectors as possible in the same request, and avoid copying of data. The usual disc interface typically requires copying or using scatter-gather, which not all devices support. Therefore, it's more efficient to send physical addresses and make sure they have proper alignment. Something you get for free with a decent cache function.

IOW, if you do a lot of disc drivers like int 0x13, passing a buffer without alignment requirements, then you miht have trouble adapting this to a interface with better performance. So, you end up having to rewrite the disc drivers once more because you didn't plan properly.
Post Reply