Chipset and driver (firmware)

[ComputerMail]

Hi,

i am creating an operating system and i happened at the writing of the hard disk driver, it is SATA and the processor x86-64, i am in protected mode. i saw this page http://wiki.osdev.org/AHCI and was surprised that writing hard disk driver does not involve i control how physicaly the hard disk works (head position, rotation speed, etc...), so my question is theorycal, what does in the computer control the hard disk physical working ? Is it the chipsets ? If yes, is chipset a hardware or software thing ?

[Love4Boobies]

The hard disk comes with some firmware on it that does the LBA-to-CHS conversion. This makes interfacing with the drive easier but also makes scheduling less efficient, as it cannot reliably take the disk's geometry into the equation. Even when you are able to send out commands in the CHS format, you cannot trust that they reflect the geometry as modern disks automatically remap bad sectors to spare ones, if available.

[Brendan]

Hi,

i am creating an operating system and i arrived at the writing of the hard disk driver, it is SATA and the processor x86-64, i am in protected mode. i saw this page http://wiki.osdev.org/AHCI and was surprised that writing hard disk driver does not involve i control how physicaly the hard disk works (head position, rotation speed, etc...), so my question is theorycal, what does in the computer control the hard disk physical working ? Is it the chipsets ? If yes, is chipset a hardware or software thing ?

The OS's driver tells (the PCI bus to tell) the SATA controller to send a command to a device attached to the controller, the SATA controller has some control logic to figure out how to send a command to the requested device, and the device has some logic (typically a small CPU and firmware and RAM - almost like a tiny little embedded computer) that decodes the command, determines if the command is sane, and performs the requested command; and takes care of things like:

• Device initialisation/power up, and reset
• Media initialisation and removal (for CD drives, etc)
• Emulating a different sector size if necessary (e.g. hard disk with 4 KiB sectors emulating a hard disk with 512 byte sectors)
• Wear levelling (for SSD)
• Bad sector replacement (for hard disk, where modern hard disks have "spare sectors" that are used to replace sectors that fail and extend the drive's "mean time between failures")
• Determining the most optimal way of performing the command/s (e.g. the order commands should be done when NCQ is being used)
• Power management (e.g. turning the drive's motor off to save some power if it hasn't been used for a while)
• Encoding/decoding the data if necessary (e.g. converting your raw bytes into some sort of error detecting or correcting code)
• Tracking various statistics, mainly to predict failures before they happen (Self-Monitoring, Analysis and Reporting Technology)
• Handling the motor speed, movement of heads, head selection, etc (including all the timing requirements).
• Handling any "retry N times if there was an error" fault recovery (potentially including replacing a sector with a "spare sector" if the drive had too much trouble reading but succeeded eventually). Note that this may include writing, where a drive can reading the data back (using a "read head") as it's being written (by a "write head") so that the drive can verify that the data was written correctly.
• Sending status/replies back to the controller; and handling or monitoring any data transfers between controller and drive's buffer
• Pre-fetching data into the device's buffer in anticipation of a future read

Cheers,

Brendan

[ComputerMail]

Thank you a lot for answers, by seeing all what the firmware does i assume no operating system developer pays attention to write his own one, is it right ? Also, same for any component's firmware ?

[dozniak]

Thank you a lot for answers, by seeing all what the firmware does i assume no operating system developer pays attention to write his own one, is it right ? Also, same for any component's firmware ?

You can't. It's specific for each device and is written by Hard Disk manufacturer.

[BrightLight]

Thank you a lot for answers, by seeing all what the firmware does i assume no operating system developer pays attention to write his own one, is it right ? Also, same for any component's firmware ?

You can't write your own firmware for a specific piece of hardware. This is all drive-specific, and things like AHCI and IDE are just interfaces to hide the differences between the actual drive differences. Brendan already explained how it works; you tell AHCI to send a command to the drive, and AHCI sends the command to the drive, and the drive does its drive-specific functions transparently to you and the AHCI, and when it finishes it returns its status to the AHCI and you can find out whether or not the command succeeded from there. IDE also follows similar theory to AHCI.

[Love4Boobies]

Well, you can do it but, in many cases, you'll need to do a bit of reverse engineering in order to figure out how to do it since the documentation is lacking. But people have done it before, they've even written malware that rests in such firmware, which needless to say, is very difficult to detect. Either way, custom firmware is not something an OS needs to concern itself with---ultimately it needs to trust the hardware and that responsibility lies outside the realm of the OS IMO.

[dozniak]

Well, you can do it but, in many cases, you'll need to do a bit of reverse engineering in order to figure out how to do it since the documentation is lacking. But people have done it before, they've even written malware that rests in such firmware, which needless to say, is very difficult to detect. Either way, custom firmware is not something an OS needs to concern itself with---ultimately it needs to trust the hardware and that responsibility lies outside the realm of the OS IMO.

UNLESS you're writing a trusted kernel for trusted computing system, but that's usually rare in hobby OSdev.

[Love4Boobies]

If it's a trusted computer, why should the OS care about the firmware? Presumably it should trust it implicitly. If it doesn't, pick different hardware.

There is really not much overlap between firmware development and OS development like there is between OS development and application development.

[Schol-R-LEA]

UNLESS you're writing a trusted kernel for trusted computing system, but that's usually rare in hobby OSdev.

I have a problem with the concept of Trusted Computing in general - or rather, with the narrative laid on top of it by marketing departments, which gives a highly distorted view of the idea - but that is going off topic enough that anything further said about it should be in a separate thread. Suffice it to say that actual Trusted Computing is incompatible with consumer-grade hardware and software.

Getting back to the topic, the real issue is that the CPU does not, as a rule, have access to the drive's firmware and internal hardware in the first place - the ATA model realistically requires that drive be managed by an embedded microcontroller with its own memory and firmware (which on higher-end devices are usually a flash memory, to allow for firmware updates, but could be an actual masked ROM in cheaper models), which the CPU has no direct interaction with - and a programmer would need details of the hardware which are not generally made available in order to affect the on-drive control electronics directly. This isolation is at least partially deliberate, as the primary goal of the controller design going back to the XT days was to relieve the CPU of these tasks - even with the original MFM drives, the CPU didn't need to do most of that work, and successive controller models have offloaded more and more of the work onto the controller board and the drive itself over time.

In most cases, the only interaction that the computer's CPU has with the firmware is to pass a firmware update to it, and a reputable hardware manufacturer should be using some kind of encryption on that and bundle it with with an ECC signature and a hash signature for tamper-checking (while those aren't foolproof, they do make tampering impractical for most casual attacks). I don't know if any do, but... anyway.

This isolation is a Good Thing, because these operations take a lot of computation, so off-loading it only makes sense - just as with CPU-driven video, a CPU-driven disk would eat an inordinate amount of cycles when in use. Also, it modularizes the system better, reducing the chances that a bug in the driver will trash the disk records or the disk itself - in the bad old days of minicomputers with loadable platter packs, and even in the early 8-bit home computer days with the Shugart, Apple ProFile, and Apple Sider drives, it was possible for a wonky driver to cause physical damage to the platters and/or read heads. Finally, it abstracts away the design of the drive itself, making it possible to use a single generic driver for not only many models of hard disk without regard to the drive's actual geometry or control electronics, but also with completely different storage technologies such as solid-state flash and NVMe.