SCSI dilemma

[jacks]

Hi all,

I need to communicate with SCSI(any physical interface - e.g. parallel, fibre channel, sas) tape drives and hard drives. After searching over SCSI specs, I'm confused as from where to begin.

For communicating with SCSI tape drives and SCSI hard drives, do I need specs other than SCSI block commands and SCSI stream commands specs. along with SCSI primary commands spec.
As far as I understood, besides these 3 specs no other spec should be needed.

Another thing I observed is, how do I communicate with a pci scsi controller(ie scsi host bus adapter). I didn't find any spec related to this. Is there any such spec. for SCSI HBA, just like we have ATA Host bus Adapter/PCI IDE controller specs. ?

[Kevin]

I think you'll also want at least the SAM (SCSI Architecture Model).

[Combuster]

The various SCSI specifications only document the signals and data of the actual SCSI bus, as well as having been reused for the packet commands on the (S)ATA bus.

The host adapter talks SCSI on one end, and some proprietary interface on the other. You'll need the manual for your specific SCSI card to know how to have the host adapter translate your commands into SCSI signals.

Be warned, there's also zero tutorials on this available, and close to no prior art among the regulars. That means this is one of those places where you can actually prove to be a capable OS developer rather than a copy-paster.

[jacks]

The various SCSI specifications only document the signals and data of the actual SCSI bus, as well as having been reused for the packet commands on the (S)ATA bus.

As far as I could get,, it is something like this:

SCSI block/stream commands } ___ PCI SCSI HBA ____ SCSI bus
SCSI Primary commands }

Now software/driver need not worry about actual physical interface of scsi device- e.g. fibre channel or SAS or parallel scsi.
From driver's perspective, any block device(Fibre channel hard disk, sas hard disk, parallel scsi hard disk, etc.) can be communicated via SCSI block commands and any stream device(parallel scsi tape drive, sas tape drive, etc.) can be communicated via SCSI stream commands.

The driver just have to send appropriate scsi commands to the SCSI HBA and that's all.

The host adapter talks SCSI on one end, and some proprietary interface on the other. You'll need the manual for your specific SCSI card to know how to have the host adapter translate your commands into SCSI signals.

The point is how can I send scsi command packet/block to the PCI SCSI HBA? I mean, is there something like task file register interface, or do I have to put command block in a specific memory location and SCSI HBA will fetch it from there.

But how do I know which I/O ports or memory mapped registers to use to communicate with PCI SCSI host, just like we do for a PCI IDE controller or via AHCI.

[Combuster]

this is one of those places where you can actually prove to be a capable OS developer

That includes answering your own questions.

What you need to go through is a mentality change. Your current attitude will never get you any further than the people you're asking questions from, you'll have to do your own research for that.

[jacks]

this is one of those places where you can actually prove to be a capable OS developer

That includes answering your own questions.

What you need to go through is a mentality change. Your current attitude will never get you any further than the people you're asking questions from, you'll have to do your own research for that.

Can you elaborate? It would be helpful.

As far as my own research is concerned, I am still searching about the issues.

The main problem is, I do not get, how can I communicate programmatically to the SCSI HBA. Is there such spec about SCSI HBA ? I was not able to find it. Is the communication interface b/w pci SCSI HBA and software is propiaratory, so that only manufacturer can provide their drivers to enable such communication ?

You said:

The host adapter talks SCSI on one end, and some proprietary interface on the other. You'll need the manual for your specific SCSI card to know how to have the host adapter translate your commands into SCSI signals.

If that is true then unlike pci ide controller, w/o knowing pci vendor id and device id, we can't communicate with any SCSI HBA, Is that true?

W/o knowing how to communicate with a SCSI HBA, it doesn't matter whether one is a scsi expert or not(from programming perspective).

That's why I asked the question here.

[Kevin]

The main problem is, I do not get, how can I communicate programmatically to the SCSI HBA. Is there such spec about SCSI HBA ? I was not able to find it. Is the communication interface b/w pci SCSI HBA and software is propiaratory, so that only manufacturer can provide their drivers to enable such communication ?

This is not standardised. You need to find documentation for the very specific HBA you want to talk to.

If that is true then unlike pci ide controller, w/o knowing pci vendor id and device id, we can't communicate with any SCSI HBA, Is that true?

Yes, just like with most other hardware. (Or well, strictly speaking no, there are certainly non-PCI HBAs for which you would use other means of identifying them...)

[Combuster]

If that is true then unlike pci ide controller

Even an IDE controller has a specific interface. The essential difference between that and SCSI it that the IDE controller is standardized and you can get away with having only one driver that works on the vast majority of machines out there. For that same reason it's bad practice but often functional to not care at all and just declare that it's there without even looking. Still, finding an IBM PC compatible IDE controller is not done by PCI device:vendor numbers as every chipset manufacturer and plug-in card has its own set of numbers and you'll never get to know all of them. Instead you should check the PCI class code which gives you the type and subtype of device, and for a few cases including the IDE controller, the specific interface standard. If you can't find the class code on the PCI bus, you don't have a PCI IDE controller, and you're left with guesswork on whether or not to go looking for an ISA version.

In other words, an IDE controller is fundamentally no different from a SCSI controller, or any other bit of hardware in your PC. Get the manual for the interface, write a driver, find the device, and have them cooperate.

[jacks]

If that is true then unlike pci ide controller

Even an IDE controller has a specific interface. The essential difference between that and SCSI it that the IDE controller is standardized and you can get away with having only one driver that works on the vast majority of machines out there. For that same reason it's bad practice but often functional to not care at all and just declare that it's there without even looking. Still, finding an IBM PC compatible IDE controller is not done by PCI device:vendor numbers as every chipset manufacturer and plug-in card has its own set of numbers and you'll never get to know all of them. Instead you should check the PCI class code which gives you the type and subtype of device, and for a few cases including the IDE controller, the specific interface standard. If you can't find the class code on the PCI bus, you don't have a PCI IDE controller, and you're left with guesswork on whether or not to go looking for an ISA version.

I know that a generic pci device driver probes the pci devices by class and subclass code. That was not my issue. My issue was - even if we searched the SCSI HBA, then how would we interpret the contents of its BARs?

In case of PCI IDE controller this interpretation is standarized. What I was asking if there is such an standarized interpretation for PCI SCSI HBAs, but now it seems that there is no such standard interpretation of BAR contents of a PCI SCSI HBA.

Only datasheet can help, if publicly available, but in most cases it is available only under NDA.

In other words, an IDE controller is fundamentally no different from a SCSI controller, or any other bit of hardware in your PC. Get the manual for the interface, write a driver, find the device, and have them cooperate.

One doesn't need to get a manual in case of IDE controller to get it work. It is only required for precise control and robustness, which comes after one make it work.

In case of SCSI control one can have different interpretations b/w manufacturers, so w/o datasheet or reverse engineering one can't even communicate with a SCSI HBA at basic level. Am I right?


P.S. I really have no idea about what attitude you were talking about. It would be nice if you can elaborate things.

[rdos]

I would chose one of two possible routes:

1. Look in the Linux source code for a driver for your specific SCSI adapter, or try to install Linux and see if it works.
2. Buy another machine with IDE/SATA instead

If Linux doesn't support your SCSI drive, chances that you will be able to get the specification are basically zero. If Linux has a driver, and it wasn't supplied by the manufacturer in binary form, you can either try to get the specification, or figure out how it works by studying Linux source.

[Combuster]

In case of PCI IDE controller this interpretation is standarized. What I was asking if there is such an standarized interpretation for PCI SCSI HBAs, but now it seems that there is no such standard interpretation of BAR contents of a PCI SCSI HBA.

If that was your real question, it took your four posts of running around before actually asking it. The answer is no, there is no standard host-side interface for SCSI. That fact was already mentioned (between the lines for lack of an direct question) several posts up.

<rant>
You could have found out yourself by just being observant. about 50% of the drivers windows loads before install are for SCSI. You could have looked up your selection of SCSI cards and find out they all need a separate driver. You could have deduced it all from the lack of information on a standard interface. You could have looked at the list of interfaces for PCI. You could even have seen the fragment "for your specific SCSI card" I posted earlier. Then maybe you would have found out for yourself that there is no standard and simply ask "am I correct about this"

That also brings me to the attitude: experimenting and testing your ideas is a core part of OS development business. If you have trouble deducing something as simple as "there's no standard interface for SCSI", which anybody whom has spent a fair share of time with such a system will be able to tell you, how will you fare with finding the answers to things nobody know about?

[jacks]

In case of PCI IDE controller this interpretation is standarized. What I was asking if there is such an standarized interpretation for PCI SCSI HBAs, but now it seems that there is no such standard interpretation of BAR contents of a PCI SCSI HBA.

If that was your real question, it took your four posts of running around before actually asking it. The answer is no, there is no standard host-side interface for SCSI. That fact was already mentioned (between the lines for lack of an direct question) several posts up.

Well, I tried to be clear by specifying ATA host adapter PCI IDE controller specs, in my first post. But I should have been more clear.
Thanks.

<rant>
You could have found out yourself by just being observant. about 50% of the drivers windows loads before install are for SCSI.

That should not mean that there is no standard for scsi host adapter. Windows loads drivers for some IDE controllers too(e.g. Intel, Sis, etc.)

You could have looked up your selection of SCSI cards and find out they all need a separate driver.

I don't know how could I find that they all need a separate driver, by just knowing the manufacturer of card/controller. I think you mean to say: by reading some publicly available datasheets of SCSI HBAs.
In case of IDE controllers drivers are required for IDE controller cards, even though PCI IDE controller spec exist.

You could have deduced it all from the lack of information on a standard interface.

Well, I'll not use word deduced, but it was a 50-50 situation for me before asking it here. You can say I just have some idea that such an standard may not exist. I was not sure though.

You could have looked at the list of interfaces for PCI.

I tried googling it before asking it here, but I didn't find anything that can give me a reason to conclude that there doesn't exists such a standard.

You could even have seen the fragment "for your specific SCSI card" I posted earlier.

Yes I did noted that, but I was not sure what you mean to say. On many earlier occasions, on other forums, whenever I asked such a standard related question regarding any I/O devices, many said that I will need a driver for that - w/o driver I will not be able to communicate with I/O device. They simply assumed that since most I/O devices come with their drivers provided by manufactures, w/o them it was not possible to communicate with the device. Later I figured out that standards for such device do exist, some are formal standards, other are de-facto.

So, I tried to confirm that only, by asking again. I hope you can understand.

Then maybe you would have found out for yourself that there is no standard and simply ask "am I correct about this"

Well, this was actually my intention when I asked it in my first post. That's why I gave the reference of ATA host/PCI IDE controller, to indicate what I am searching for.
May be I should have asked more clearly. That would have simplified things for all of us. I 'll keep your suggestions in mind next time. Thanks

That also brings me to the attitude: experimenting and testing your ideas is a core part of OS development business. If you have trouble deducing something as simple as "there's no standard interface for SCSI", which anybody whom has spent a fair share of time with such a system will be able to tell you, how will you fare with finding the answers to things nobody know about?

Well, I'm new to this driver kind of stuff, and I occasionaly find myself messed up in hardware specs. The definition of fair share of time is different for different persons. In other words, I have not spent fair enough time, neither I have much experience regarding such stuff, so may due to that it was not obvious for me to deduce this thing. I'm learning still, and it will take time for me to reach that stage where I can ask such question, that nobody here knows about.

Thanks for your positive criticism and throwing light on various aspects of how a OS developer should figure things out. I'll try to be better next time.

[jacks]

hello all,

I 'm here to share some info about SCSI HBA programming interface standarization. May be most of you are well aware of this, but for newbies like me it can be a hard time to connect various specs and concepts, so for them here it goes.

The SCSI HBA does not have any standard interface at hardware level, so that a generic SCSI HBA driver can be developed. At hardware level there are many variations on the interpretation of PCI BARs of a SCSI HBA.
To narrow down this chaos, some standards at software level were developed in course of time. These are :

Advanced SCSI Programming Interface(ASPI), developed by Adaptec
SCSI Pass Through Direct (SPTD), developed by Microsoft

Win32/DOS applications or drivers can send SCSI Command Descriptor Block (CDB) messages to SCSI devices by using these software interfaces. The only requirement is that the SCSI HBA driver must also implement same interface, ie ASPI, SPTD, etc.

Here is the scenario:

storage driver or application ----> ASPI or SPTD compliant SCSI HBA driver -----> SCSI HBA ----> SCSI device

In other words SCSI HBA driver acts as a black box for OS drivers, and upper layers that can send SCSI Command packets to the desired SCSI device.

The Common Access Method(CAM) spec. was also released for the same purpose. It is implemented by FreeBSD SCSI sub-system.

I hope this will be useful, at least for some of us.

Some Links:
http://support.microsoft.com/kb/251369
http://msdn.microsoft.com/library/ff565367.aspx
http://code.msdn.microsoft.com/windowsh ... h-a906ceef
http://www.adaptec.com/en-us/speed/soft ... a2_exe.htm
http://people.freebsd.org/~gibbs/ARTICLE-0001.html

______________________________________________________________________________________________

Just for confirmation:
Do SCSI hard drives have UDMA or DMA modes just like ATA hard drives have? I'm not talking about pci HBA bus master transfers, i am talking about the hard drive itself.

So far it seems that SCSI drives unknowingly use pci bus master transfers due scsi hba, i.e. from hard drive's perspective, it simply sends scsi response packets to scsi hba and that's all. It doesn't need to know how these packet reach to main memory.

So, am I correct that there is no concept of UDMA, PIO in SCSI hard drives?

Also, can anyone confirm that there is no concept of PIO data transfers in SCSI system, ie concepts like data register or data port. It seems that scsi data transfers are completely based on pci bus master dma, there is no concept of data port, etc in scsi, becoz whole data transfer is in the form of packets which are transfered b/w main memory and scsi hba. Am I right?

[Combuster]

hello all,

I 'm here to share some info about SCSI HBA programming interface standarization. May be most of you are well aware of this, but for newbies like me it can be a hard time to connect various specs and concepts, so for them here it goes.

It's rare to see someone with this level of manners

It seems that scsi data transfers are completely based on pci bus master dma, there is no concept of data port, etc in scsi, becoz whole data transfer is in the form of packets which are transfered b/w main memory and scsi hba. Am I right?

That's pretty much impossible with all the ISA host adapters out there that are completely incapable of bus mastering. Somewhat in line with the device's age you should expect to find it supports one or more of the list of PIO, MMIO, ISA DMA, and busmaster transfers. Of course from the host's perspective the DMA versions are the most efficient to use, but need not be the easiest for a given host controller.