How to read from SATA HDD (AHCI) on low level?

[natiiix]

I'm a little surprised that you have mastered the complexities of setting up paging in the x86 architecture but can't manage a relatively simple concept like accessing the PCI device registers.

Whatever, you are asking for finished code and that's not really what this site is designed for, as I understand it. It's more about guiding people to write such code (amongst other things). Perhaps you would have more joy on somewhere like StackExchange?

I haven't really mastered the paging, I just set up 2 pages that I use for everything. It's not like I need more than 8MB of memory anyways.

Regarding SO/SE.. People there know next to nothing about OS development. I've actually asked there before even registering here, but I've only got answer from one person and he said that he doesn't really know how AHCI works anyways. So much for SO/SE.

Anyways, I've got the probe_port() to work, but the read() function still kills the processor. Is there something that needs to be done before reading from the drive? Perhaps the port_rebase is a necessity?
Edit: port_rebase() doesn't work because stop_cmd() is for some reason taking forever.

[Octocontrabass]

I was referring to the impossibility of having the physical address already mapped when the virtual addresses are 0xC0000000 shifted are thus it would be significantly higher than the maximum size_t value.

And I was referring to the fact that paging is a lot more flexible than "add a constant value to the physical address". You could, for example, map a 4kB page from physical address 0xFD5FF000 to virtual address 0xC0800000, and use that to access the HBA.

Please refrain from linking whole Wikipedia pages, it's meaningless and definitely not helpful.

If reading isn't really your style, you can get a pretty good idea from this picture instead.

If your complaint is about the contents of the page I linked, why are you writing an OS?

[MollenOS]

I don't really need anything like a proper AHCI driver. I want just a very basic one.

Even a basic one requires a large part of what I said. But I'm giving up, you don't want help, you want someone to give you some source code you can use. I'm sorry to tell you it doesn't work like that. You have to put in the effort and write it yourself. OS Development is doing things from scratch, this is not C# development where you can find libraries for everything. Here there are no libraries you can just import and they'll work.

Good luck with your AHCI driver.

[natiiix]

And I was referring to the fact that paging is a lot more flexible than "add a constant value to the physical address". You could, for example, map a 4kB page from physical address 0xFD5FF000 to virtual address 0xC0800000, and use that to access the HBA.

If reading isn't really your style, you can get a pretty good idea from this picture instead.

If your complaint is about the contents of the page I linked, why are you writing an OS?

You could have saved plenty of time if you just wrote that it's possible to swap parts of virtual memory which I obviously know, it's just completely meaningless to me when I don't have access to a hard drive.
My complaint was about your approach to answer questions. You're making a point by linking a whole Wikipedia page as if that were some magical answer rather than bothering yourself to write a single sentence.

Even a basic one requires a large part of what I said. But I'm giving up, you don't want help, you want someone to give you some source code you can use. I'm sorry to tell you it doesn't work like that. You have to put in the effort and write it yourself. OS Development is doing things from scratch, this is not C# development where you can find libraries for everything. Here there are no libraries you can just import and they'll work.

Good luck with your AHCI driver.

C# is a language, not an environment. And I can't really see the problem with trying to find an already existing code instead of coming up with it on my own. Have you ever heard of not reinventing the wheel?

Once again please stop replying to what I've said long time ago, I've probably already figured it out by now, but there is still this crucial problem I haven't been able to surpass.

Anyways, I've got the probe_port() to work, but the read() function still kills the processor. Is there something that needs to be done before reading from the drive? Perhaps the port_rebase is a necessity?
Edit: port_rebase() doesn't work because stop_cmd() is for some reason taking forever.

[MollenOS]

Have you ever heard of not reinventing the wheel?

This is - absolutely - the entire point of OS Development, we are all reinventing the wheel for our own enjoyment.

[natiiix]

This is - absolutely - the entire point of OS Development, we are all reinventing the wheel for our own enjoyment.

That's absolutely not true. You're necessarily rewriting meaningless code if you do that, because most devices communicate in a standardized way. Therefore the right approach is to use already existing libraries to operate them.


Anyways...
Could someone please tell me what am I doing wrong? There is nothing being written to the buf buffer. I use the port index from probe_port() as HBA_MEM*->ports[index], is that the correct way to do it?
note: phystovirt() converts physical memory address to a virtual one and virttophys() does the exact opposite, the rest of the functions used are from Wiki.

Code: Select all

BOOL read(HBA_PORT *port, DWORD startl, DWORD starth, DWORD count, WORD *buf)
{
	port->is = (DWORD)-1;		// Clear pending interrupt bits
	int spin = 0; // Spin lock timeout counter
	int slot = find_cmdslot(port);
	term::writeline((size_t)slot, 16);
	if (slot == -1)
	{
		return FALSE;
	}
 
	HBA_CMD_HEADER *cmdheader = (HBA_CMD_HEADER*)phystovirt(port->clb);
	term::writeline((size_t)cmdheader, 16);
	cmdheader += slot;
	cmdheader->cfl = sizeof(FIS_REG_H2D)/sizeof(DWORD);	// Command FIS size
	cmdheader->w = 0;		// Read from device
	cmdheader->prdtl = (WORD)((count-1)>>4) + 1;	// PRDT entries count
 
	HBA_CMD_TBL *cmdtbl = (HBA_CMD_TBL*)phystovirt(cmdheader->ctba);
	memset(cmdtbl, 0, sizeof(HBA_CMD_TBL) + (cmdheader->prdtl-1)*sizeof(HBA_PRDT_ENTRY));
 
	// 8K bytes (16 sectors) per PRDT
	for (int i = 0; i < cmdheader->prdtl-1; i++)	
	{
		cmdtbl->prdt_entry[i].dba = (DWORD)virttophys(buf);
		cmdtbl->prdt_entry[i].dbc = 8*1024;	// 8K bytes
		cmdtbl->prdt_entry[i].i = 1;
		buf += 4*1024;	// 4K words
		count -= 16;	// 16 sectors
	}

	// Last entry
	cmdtbl->prdt_entry[cmdheader->prdtl-1].dba = (DWORD)virttophys(buf);
	cmdtbl->prdt_entry[cmdheader->prdtl-1].dbc = count<<9;	// 512 bytes per sector
	cmdtbl->prdt_entry[cmdheader->prdtl-1].i = 1;
 
	// Setup command
	FIS_REG_H2D *cmdfis = (FIS_REG_H2D*)phystovirt(&cmdtbl->cfis);
 
	cmdfis->fis_type = FIS_TYPE_REG_H2D;
	cmdfis->c = 1;	// Command
	cmdfis->command = ATA_CMD_READ_DMA_EX;
 
	cmdfis->lba0 = (BYTE)startl;
	cmdfis->lba1 = (BYTE)(startl>>8);
	cmdfis->lba2 = (BYTE)(startl>>16);
	cmdfis->device = 1<<6;	// LBA mode
 
	cmdfis->lba3 = (BYTE)(startl>>24);
	cmdfis->lba4 = (BYTE)starth;
	cmdfis->lba5 = (BYTE)(starth>>8);
 
    cmdfis->countl = (BYTE)count;
	cmdfis->counth = (BYTE)(count>>8);
 
	// The below loop waits until the port is no longer busy before issuing a new command
	while ((port->tfd & (ATA_DEV_BUSY | ATA_DEV_DRQ)) && spin < 1000000)
	{
		spin++;
	}
	if (spin == 1000000)
	{
		trace_ahci("Port is hung");
		return FALSE;
	}
 
	port->ci = 1<<slot;	// Issue command
 
	// Wait for completion
	while (1)
	{
		// In some longer duration reads, it may be helpful to spin on the DPS bit 
		// in the PxIS port field as well (1 << 5)
		if ((port->ci & (1<<slot)) == 0) 
			break;
		if (port->is & HBA_PxIS_TFES)	// Task file error
		{
			trace_ahci("Read disk error");
			return FALSE;
		}
	}
 
	// Check again
	if (port->is & HBA_PxIS_TFES)
	{
		trace_ahci("Read disk error");
		return FALSE;
	}
 
	return TRUE;
}

[MollenOS]

This is how i build the PRDT table in the AHCI command, I think the code you have copied is wrong

Code: Select all

// Build PRDT entries
	BufferPointer = Transaction->Address;
	while (BytesLeft > 0) {
		AHCIPrdtEntry_t *Prdt = &CommandTable->PrdtEntry[PrdtIndex];
		size_t TransferLength = MIN(AHCI_PRDT_MAX_LENGTH, BytesLeft);

		// Set buffer information and transfer sizes
		Prdt->DataBaseAddress = LODWORD(BufferPointer);
		Prdt->DataBaseAddressUpper = (sizeof(void*) > 4) ? HIDWORD(BufferPointer) : 0;
		Prdt->Descriptor = (TransferLength - 1); // N - 1

		// Adjust counters
		BufferPointer += TransferLength;
		BytesLeft -= TransferLength;
		PrdtIndex++;

		// If this is the last PRDT packet, set IOC
		if (BytesLeft == 0) {
			Prdt->Descriptor |= AHCI_PRDT_IOC;
		}
	}

	// Update command table to the new command
	Transaction->Device->Port->CommandList->Headers[Transaction->Slot].TableLength = (uint16_t)PrdtIndex;
	Transaction->Device->Port->CommandList->Headers[Transaction->Slot].Flags = (uint16_t)(CommandLength / 4);

[natiiix]

This is how i build the PRDT table in the AHCI command, I think the code you have copied is wrong

The code I've copied is straight from the Wiki. The only difference is that I've added translations from physical to virtual addresses, because it seemed to me that the original code didn't count with paging.

Could you be more specific as to what you believe might be the problem? Without proper context I can't really see much of a difference between the code I've posted and yours.

[Schol-R-LEA]

I can't really see the problem with trying to find an already existing code instead of coming up with it on my own.

The problem is that no such code or library exists - and it cannot exist, unless your kernel is a direct modification of an existing one which already provides those services, in which case this conversation would never have come up in the first place.

Things like this are inherently OS-dependent, meaning that no two operating systems can share the same code for them to any significant degree. AN OS-PORTABLE DRIVER IS AN IMPOSSIBILITY. It is something that must be written for the specific kernel it is used by.

And in case you are wondering, this is true even for things like ndiswrapper (a Windows driver compatibility layer which is, or at least was, widely used in Linux for driving some network cards which were based on proprietary controllers), as they rely on there being a kernel-specific bus or bridge driver doing the actual talking with the components.

At best, you can use some other operating systems' driver code as a guide to how to write one, but a) those drivers are, for the most part, written for efficiency rather than clarity (as you already noted regarding Linux), and b) are likely to be misleading, as the way they interact with the kernel will perforce be different from how yours will, and it is likely that you will end up copying code which fits that OS but not yours without realizing it.

μή εἶναι βασιλικήν ἀτραπόν ἐπί γεωμετρίαν ("there is no Royal road to geometry", which is what Euclid supposedly told Ptolemy Soter when the future successor of Alexander complained about his lessons) applies just as much to OS development as it does to geometry. I will admit that we have not been very helpful in explaining things to you, but part of the problem is that you have not really made it clear what you are having trouble understanding, and have insisted that you don't need to understand things which you do, in fact, need to understand to complete the task you have set for yourself.

[natiiix]

The problem is that no such code exists - and it cannot exist, unless your kernel is a direct modification of an existing one which already provides those services, in which case this conversation would never have come up in the first place.

Things like this are inherently OS-dependent, meaning that no two operating systems can share the same code for them to any significant degree. AN OS-PORTABLE DRIVER IS AN IMPOSSIBILITY. It is something that must be written for the specific kernel it is used by.

There can easily be a code which operates a device and doesn't rely on the OS at all. I've linked this code earlier as an example and it clearly proves that drivers don't inherently need to be OS dependent, even though the ATA PIO code specifically is very simple, the code can be implemented into any OS with ease.

Anyways, I've said this many times before, don't reply to what I've said long time ago as I've already got the core of the AHCI driver to work and thus I'm not necessarily looking for any library anymore.
I'd appreciate a lot more if someone could look through the read() function's code posted above and give me a hint of why it refuses to read anything reasonable and gives me only zeros on the output.

[Schol-R-LEA]

I was replying to something you said yesterday, in a thread that is four days old. I am not sure how that counts as 'a long time ago', especially since you never actually said that you had the driver working (you said that you didn't need the information, but sort of danced around the reason why).

Regarding Dragoniz3r's example, it is far from a complete driver; it only supports a specific sub-set of HDD interfaces, and even then, I can promise you it won't work for a lot of systems. Even given that, getting it to work with a specific OS will mean re-writing it and wrapping it up to fit into the driver model of the OS (or redesign the OS to force-fit the driver into it, as I suspect you did). It is, at best, a skeleton from which to write a driver, not a real driver on its own.

As I said earlier (I was adding it as you were replying, so you may not have seen it), you can use some other operating systems' driver code as a guide to how to write one, but a) those drivers are, for the most part, written for efficiency rather than clarity (as you already noted regarding Linux), and b) are likely to be misleading, as the way they interact with the kernel will perforce be different from how yours will, and it is likely that you will end up copying code which fits that OS but not yours without realizing it.

[natiiix]

I was replying to something you said yesterday, in a thread that is four days old. I am not sure how that counts as 'a long time ago', especially since you never actually said that you had the driver working (you said that you didn't need the information, but sort of danced around the reason why).

Regarding Dragoniz3r's example, it is far from a complete driver; it only supports a specific sub-set of HDD interfaces, and even then, I can promise you it won't work for a lot of systems. Even given that, getting it to work with a specific OS will mean re-writing it and wrapping it up to fit into the driver model of the OS (or redesign the OS to force-fit the driver into it, as I suspect you did). It is, at best, a skeleton from which to write a driver, not a real driver on its own.

As I said earlier (I was adding it as you were replying, so you may not have seen it), you can use some other operating systems' driver code as a guide to how to write one, but a) those drivers are, for the most part, written for efficiency rather than clarity (as you already noted regarding Linux), and b) are likely to be misleading, as the way they interact with the kernel will perforce be different from how yours will, and it is likely that you will end up copying code which fits that OS but not yours without realizing it.

I didn't have to make any changes to the code, nor to my OS, while implementing the code linked before.
Yesterday was a pretty long time ago and I've already stated this morning that the only thing not working is the reading algorithm which is pretty much just copied from the Wiki, yet it refuses to work. Frankly I couldn't find anything resembling a read() in anyone else's OS and therefore I couldn't fix it myself. That's why I've asked here what the problem might be, but I doubt anyone is going to take the time to read through the code and tell me what went wrong. ¯\_(ツ)_/¯
I believe the problem is related to memory addressing, but I can't really tell myself why it doesn't work.

[Schol-R-LEA]

Perhaps a bit of a do-over is called for. It might help if you explained what your goals are.

For example, why do you want to write an OS ? you've already mentioned wanting a diversion from your regular work, and that writing an OS had been a long-term plan of yours, but I think it might clear the air a bit if we had a better idea of just what attracts you to OS dev. It doesn't have to be anything special; something such as 'I felt like it', or 'I am curious about how they work' or even 'because it is there' would suffice, but there are a lot of reasons people pick this hobby so it relates to how we can help you best.

Similarly, "what parts of the OS primarily interest you?" would give us a better idea of what you want to focus on. I will warn you that most of the posters here are focused mainly on the kernel and drivers, so you may have trouble getting help on things like UI - we do have plenty of people who know them well, also, but there are some here with a bit of a 'UX isn't OSdev' attitude which can get in the way for members who are interested in what the users actually see and how usable it is.

(This is especially prevalent among those who are interested in Unix-like systems, where having both the shell and the GUI be entirely userland programs is seen as a positive thing, and those interested in MS-DOS workalikes, who tend to see the command interpreter as 'elegantly minimalist' and user interaction by the OS is generally considered as something to be avoided.)

[natiiix]

For example, why do you want to write an OS ? you've already mentioned wanting a diversion from your regular work, and that writing an OS had been a long-term plan of yours, but I think it might clear the air a bit if we had a better idea of just what attracts you to OS dev. It doesn't have to be anything special; something such as 'I felt like it', or 'I am curious about how they work' or even 'because it is there' would suffice, but there are a lot of reasons people pick this hobby so it relates to how we can help you best.

I've said that too. I want to develop certain user application which are very difficult to make on Windows or Linux as they involve things you can't do particularly easily in either one of those systems due to their high-level-ness.
So basically I want to create an OS with bash-like text interface and some basic programs such as a file explorer and text editor, both made the way they used to work in the DOS days. (pseudo-GUI in text mode)
I don't intend to implement anything like networking, mouse (even though mouse support existed in late DOS days) or fancy graphical mode.
And yes, I indeed am doing this mainly "for the sake of it", there is no superior motive.

Similarly, "what parts of the OS primarily interest you?" would give us a better idea of what you want to focus on. I will warn you that most of the posters here are focused mainly on the kernel and drivers, so you may have trouble getting help on things like UI - we do have plenty of people who know them well, also, but there are some here with a bit of a 'UX isn't OSdev' attitude which can get in the way.

That's strongly related to the previous question. I'm mainly interested in creating the very basic user interface in text mode while trying to abstract myself from the hardware as much as possible (as ironical as it might seem while talking about OS development).
I don't really need help with the interface itself, it's just the drivers that bother me, because I've never been all that interested in hardware.

[Schol-R-LEA]

For example, why do you want to write an OS ?

I've said that too. I want to develop certain user application which are very difficult to make on Windows or Linux as they involve things you can't do particularly easily in either one of those systems due to their high-level-ness.

Actually, this is the first time you have mentioned having any specific applications in mind (I just checked; I didn't recall you saying this, so I went over every post you've made previously to find it, and I don't see it). That does give me some idea of things, though I am now curious as to what sort of applications you mean, and why they weren't possible or satisfactory under the two systems you mentioned (I'm not doubting the assertion, I just want to understand why).

So basically I want to create an OS with bash-like text interface and some basic programs such as a file explorer and text editor, both made the way they used to work in the DOS days. (pseudo-GUI in text mode)
I don't intend to implement anything like networking, mouse (even though mouse support existed in late DOS days) or fancy graphical mode.

OK, knowing that helps.

And yes, I indeed am doing this mainly "for the sake of it", there is no superior motive.

Nothing wrong with that. If anything it probably makes it easier to help you than it would if you had some specific, overarching ambition that exceeded the knowledge of the other members.

Similarly, "what parts of the OS primarily interest you?" would give us a better idea of what you want to focus on. I will warn you that most of the posters here are focused mainly on the kernel and drivers, so you may have trouble getting help on things like UI - we do have plenty of people who know them well, also, but there are some here with a bit of a 'UX isn't OSdev' attitude which can get in the way.

That's strongly related to the previous question. I'm mainly interested in creating the very basic user interface in text mode while trying to abstract myself from the hardware as much as possible (as ironical as it might seem while talking about OS development).
I don't really need help with the interface itself, it's just the drivers that bother me, because I've never been all that interested in hardware.

OK, then, that clarifies a lot. However, I think you'll find it difficult to abstract the hardware away before you implement either an abstraction layer, or the low-level drivers to fit into said layer; and if your goal is to have abstraction without a defined abstraction layer or interface, uhm... got nothin', sorry.

I'd almost be tempted to recommend an exokernel approach, except that the necessary virtualization for an exokernel requires a lot of low-level work - and the whole idea of exokernels is to avoid abstracting the hardware, and instead have each 'application' basically be a slew of independent libraries, which is sort of at cross-purposes for what you seem to want.

OTOH, if you want abstraction but don't want a lot of layering overhead... well, you could try something like Synthesis, I suppose, though you really can't do that kind of design in C++ (the original was written in M68K assembly, and a high-level language implemention would almost certainly have to be in a homoiconic language like Lisp or Smalltalk), and again, it requires a very detailed understanding of the hardware and of advanced code generation techniques. I am one of the few people here right now crazy enough to consider that, and trust me, it is crazy.