FUSE filesystem?

[zap8600]

Hi. So then, what filesystem should I use? I thought about FAT12, but it is hard to find documentation. I thought about USTAR (as the main filesystem). What should I use?

[nullplan]

So then, what filesystem should I use? I thought about FAT12, but it is hard to find documentation.

Wat? FAT12 is one of the best-documented file systems on the planet. You can look at the Wikipedia article about it, our Wiki article on it, Ralf Brown's interrupt list, et cetera. Merely typing it into the search engine of your least distrust should reveal a treasure trove of documentation.

What should I use?

Well, what do you want from a file system? Personally, I'd suggest ext2 for starters, then upgrade to ext3 and ext4 later. Rome wasn't built in a day, either. But then, I am developing on Linux and have a couple of implementations available to cross-check my own. I am also creating a Unix OS, so ext2 fits nicely.

I would also strongly counsel against using any of the FAT file systems as starters unless you also immediately add VFAT support, because trying to breathe in that 8.3 corset is simply not very comfortable. If you feel adventurous and want to explore a Windows FS that is undocumented, you can try your hand at NTFS. There is an open-source implementation of it in the form of NTFS-3G, which works decently well. Oh, and that one is indeed a FUSE client.

[zap8600]

Hi. What meant about FAT12, sort of wasn't about it. I started about how hard it is to find information on making disk drivers for your own OS.

[iansjack]

Basically, all a disk driver does is to read and write sectors on the hard disk to/from a buffer in memory. Doing this with an IDE controller is relatively easy and is well documented. Or, as suggested earlier, use a RAM disk and the driver is trivial.

[BenLunt]

Hi, (Being I am quite fond of file systems, I had to chime in.)

There is a lot of information about FAT, and if you wish to use USB thumb drives across platforms (one being your own), you will need a FAT driver. The FAT specification can be found here with some more information found here.

You might also want to look into the exFAT filesystem. Some USB thumb drives will have this file system on them.

As for hard drive partitions, you may want to look at the LEAN file system, as well as the others mentioned already. Octocontrabass already mentioned the wiki for file systems.

If you develop on a Windows platform--though since you are asking about FUSE, you may be on a *nix platform--I have a Windows app that allows me to view and modify image files. You can easily add, delete, or replace a file within an existing partition. Quite a few file systems are supported, but some still need more work.

Now, for how to structure your driver system. The remaining is an example of how you can do it, and is of my own opinion. There may be/are better ways, and there are for sure worst ways.

Create a block of memory holding a number of 4096-byte structures. Each of these structures will represent a device and hold information about this device. The format of the first part of the 4k block is common among all devices. Things like device type, driver entry-point pointer, things like this. The remaining part is unique to the type of device. For example, a hard-drive device will need a place to store the I/O port values, etc. (for IDE type devices), as well as other information the hard-drive driver will need. When the driver is called, via the entry-point pointer, pass a pointer on the stack to this information.

As for file systems, you create another block of memory holding a number of 4096-byte (for example) structures. Again, the format of the first part is common among all file system drivers, while the second part is unique to the file system type. One of the items in the common area is a pointer to the hard-drive's device structure saved before. You will know which device block this file system will use because you just detected the hard-drive and have the device block ID on hand. With the pointer to the hard-drive's driver in the file system's common area, the file system driver can call the hard-drive's driver when it needs to read and write sectors/blocks.

With this (simple) type of device/file system structure, you can have an unlimited amount of device and file system types. RAM disk as the device, FAT as the file system. USB thumb drive and the device, LEAN as the file system. etc. The file system is not dependent of the device it resides on as long as all system calls are generic. I.e.: READ_BLOCK always sends the same format of parameters no matter the type of device it is sending to. The device always expects this generic system call format.

A generic device driver system call may expect five parameters:

Code: Select all

  bit32s name(struct DEVICE_ENTRY *, bit16u, bit64u, bit64u, void *)

The first is a pointer to the device block (explained earlier), with one 16-bit service number, two 64-bit parameters, and one pointer parameter. If you always make sure the 64-bit parameters are at least the size of a pointer, a specific device may cast one or more of the 64-bit parameters to another pointer.

A generic file system call may expect six parameters:

Code: Select all

bit32s  name(bit32u serv, bit64u qword0, bit32u dword1, bit64u qword2, void *ptr0, void *ptr1)

The first is a service number, three parameters (64-bit and 32-bit sizes), and two pointers.

For all generic devices and files systems, this example is quite enough information.

As long as all device drivers expect the same count and size of parameters, any file system can call it. As long as all file system drivers expect a specific count and size of parameters, a user interface app can call it, which in turn will call the underlining device.

It really isn't that complicated, and you can even nest devices deeper than DEVICE <--> FILESYSTEM. For example, you can have a DEVICE <--> CACHE <--> JOURNAL <--> FILESTEM structure, having at least four levels of drivers.

With this type of (simple) system, you can also create a block of memory for partitions. The information in each partition information block has a pointer to the file system driver. A very simple way is to have a partition named "C:\" (the C being the third letter in the English alphabet), using the third entry in the list of partitions. "D:\" would use the fourth, etc. A more relaxed way would to store the name of the partition in the partition structure. For example, name the partition "LEAN001" for the first found LEAN partition. Then when the user interface passes "LEAN001:" to the virtual file system driver, that driver can look through the partition list to find its entry, then know exactly which file system driver to call.

USER_INTERFACE <--> VIRTUAL_FILE_SYSTEM <--> FILE_SYSTEM <--> DEVICE

Hope this helps,
Ben
- http://www.fysnet.net/osdesign_book_series.htm

[zap8600]

Hi. Sorry I haven't posted in a while. I have been researching IDE drives. I actually might have a solution. The only problem is that I need port commands called outsl and insl. I don't get how to implement this. I'm also having trouble finding information about it. Could somebody help me?

[BenLunt]

Hi. Sorry I haven't posted in a while. I have been researching IDE drives. I actually might have a solution. The only problem is that I need port commands called outsl and insl. I don't get how to implement this. I'm also having trouble finding information about it. Could somebody help me?

When using IDE drivers, especially older hardware, you should only read and write 16-bit words, and not via a 'string' instruction. When reading in a sector, you should have a loop that does the following:

Code: Select all

i = 0
(A)
BUFFER[i] = IN 16-bit WORD
i = i + 1
LOOP (A) IF i < 256

You should not use INSW or especially INSD. Older hardware (ISA IDE's) will not be quick enough to handle the Stringed Port I/O instructions.

Another reason not to use 32-bit access:
1) on an ISA bus, there is no 32-bit Port Access.
2) on a PCI bus, and if the ATA(PI) version is 7 or less, you can check the IDENTIFY BLOCK to see if the drive handles 32-bit DWORD access (Bit 0 of Word 48).
3) (Last time I checked) QEMU doesn't allow 32-bit access anyway. (I haven't checked, maybe QEMU has since added an update to allow this)

For IDE drives, 16-bit reads and writes are plenty fast, so stay with single 16-bit port I/O access, looped 256 times.

Ben
- http://www.fysnet.net/osdesign_book_series.htm

[zap8600]

Hi. So if I use inw and outw (or something similar), will it still work? https://github.com/cfenollosa/os-tutorial is the tutorial my OS is based off of. If you go to lesson 22, enter the cpu folder, and look at the ports.c and ports.h files, you might be able to get a better understanding of what I'm talking about. You can also look at the type.h file in the same folder, if that also helps. I also have this PDF file which helped my create a base for a disk driver.
P.S. I'm also trying to get this data understandable by my OS, such as for the text editor I will build into my kernel. http://pages.cs.wisc.edu/~remzi/OSTEP/file-devices.pdf is the link to the PDF.

[Octocontrabass]

When using IDE drivers, especially older hardware, you should only read and write 16-bit words, and not via a 'string' instruction.

I'm curious where this advice comes from. Every BIOS I've looked at all the way back to the IBM 5170 uses REP INSW and REP OUTSW in the INT 0x13 handler, and the old Linux IDE driver uses them too.

The only explanation I can think of is that someone wrote a driver using REP INSW and demand paging without realizing that REP INSW will drop data if it causes a page fault (as well as certain other faults; see the Intel SDM for details).

2) on a PCI bus, and if the ATA(PI) version is 7 or less, you can check the IDENTIFY BLOCK to see if the drive handles 32-bit DWORD access (Bit 0 of Word 48).

Drives don't handle 32-bit access, HBAs do. Ignore what the drive says and check whether the HBA supports 32-bit access by looking up its datasheet using the PCI vendor and device IDs. (This is a leftover from ESDI controllers. With ESDI, the controller responds to IDENTIFY DEVICE instead of the drive, so the information here would tell you the controller's capabilities.)

[zap8600]

Hi. What I'm wondering is:
1. How do I keep repeating outw and inw to get/send all of the data?
2. How do I keep writing data to the same thing without it being overwritten?
3. What type of variable should I use? void, char, int?

[BenLunt]

When using IDE drivers, especially older hardware, you should only read and write 16-bit words, and not via a 'string' instruction.

I'm curious where this advice comes from. Every BIOS I've looked at all the way back to the IBM 5170 uses REP INSW and REP OUTSW in the INT 0x13 handler, and the old Linux IDE driver uses them too.

Please note that I specified 'especially older hardware'. When I wrote my IDE code, the year still started with 19xx, not 20xx. :-) My notes don't specify a source, but they simply state that some older hardware didn't like the 'string' instructions. Sorry.

2) on a PCI bus, and if the ATA(PI) version is 7 or less, you can check the IDENTIFY BLOCK to see if the drive handles 32-bit DWORD access (Bit 0 of Word 48).

Drives don't handle 32-bit access, HBAs do.

Sorry, yes, you are correct. It is the controller, not the drive. I inadvertently used the wrong term here. Thanks for the correction.

Hi. What I'm wondering is:
1. How do I keep repeating outw and inw to get/send all of the data?
2. How do I keep writing data to the same thing without it being overwritten?
3. What type of variable should I use? void, char, int?

The controller's data register is an FIFO type register. You read a sequence of words from that register, or write a sequence of words to the register. The controller keeps track of the data. As you probably read, the controller even has a bit indicating if it has data ready or if the controller is ready to receive data.

Per my recommendation, to read a sector (assuming you already have sent the command, etc.), I would do something like:

Code: Select all

  int  words_per_sector = 256;  // This really needs to be calculated before-hand
  bit16u  buffer[256];  // again, words per sector instead of hard coding the 256.
  for (int i=0; i<words_per_sector; i++) {
    buffer[i] = inpw(DATA_REG);
  }
  // acknowledge the interrupt.

If you wish to use the 'string' instructions, you will need to call an assembly routine, either via inline code or an external assembly file.

You will need to specify the es:(e)di register pair as well as the (e)dx and (e)cx registers. Once executed, the instruction will read (e)cx words from the port at (e)dx placing them in the buffer pointed to at es:(e)di. (Please take care of the direction flag as well)

Each compiler treats inline code differently, so I suggest a separate assembly file.

If you use my first suggestion, you don't need to have any assembly code (as long as you have a working inpw() routine). For IDE drives, calling the inpw() routine, reading a word at a time, is acceptably fast enough, due to the fact that IDE drives are somewhat slower compared to today's hardware (SATA, AHCI, etc).

Hope this helps,
Ben

[Octocontrabass]

Please note that I specified 'especially older hardware'. When I wrote my IDE code, the year still started with 19xx, not 20xx.

Yes, I'm talking about ancient history as well. I've looked primarily at early 90s BIOS ROMs and drivers, and I can find no evidence that such buggy hardware existed. (If it really did exist, it wasn't common enough for the authors of the Linux IDE drivers or the Windows WDCTRL driver to stumble across it. I haven't checked ESDI_506 yet...)

[zap8600]

Hi. I'm wondering if using IDE Controller code will work for modern hard drives.

[Octocontrabass]

It will work with modern SATA drives, but only if they're attached to an old SATA controller in IDE compatibility mode. (Modern SATA controllers are not compatible with IDE.)

[zap8600]

Hi. So, how do I set up inpw() and outpw()? Also, can you explain to me how it would keep reading the sector without reading the same 2 bytes every single time?