EMBEDDED FILESYSTEM CONTROLLER / DRIVER

[dc0d32]

Hey guys
I don't know if this has been thought of or done before but it just came to my mind.

-We implement filesystem drivers in software, usu. OS modules or drivers. AFAIK, calls to these drivers are costly even if due care is taken. The cost is in terms of overhead. Moreover, they consume the central processor time, which is the most important aspect of software filesystem driver.
-Instead, if we implement the FS driver/Handler in a hardware interface [EMBEDDED FILESYSTEM CONTROLLER / DRIVER] between the actual physical storage device like HDD and the storage controller like IDE, (S/P)ATA or say PCI [Maybe PCI Extreme], then we can issue the FS specific commands to our intermediate controller hardware. While our hardware processes the command, the central processor is free to do other tasks.
-This will effectively reduce the central processor time spent in just handling the FS and the physical storage medium like HDD.
-Also, reverse engineering of proprietary filesystem designs like NTFS will be tougher.
-Any disk with the attached controller will be accessible in any OS, provided some basic software driver support is coded for that target OS. eg. you will be able to use ext3 FS in windows or NTFS in DOS or WIN98 or even Linux with read-write support. (This will require direct support and involvement from Microsoft). [this may create an entirely new class of problems]


!~~~~The actual hardware interface~~~~
-Let us assume our device as an interface between an existing peripheral controller present on the motherboard like PCI, and the storage device such as HDD/RAID.
-We take commands from the MthrBrd device [let's consider PCI from hereafter], interpret them, then command and control and manage the underlaying physical storage device [consider HDD from now on]. That means our device must interface with both the motherboard controller and the HDD device. The device will also handle request queues [one of the costly things to manage in kernel mode].
-The device will handle all opened files etc.
-Two devices will communicate with each other and file copying among them will not involve central processor.*
-The device will schedule physical disk access according to the requesting task's effective priority, and a suitable head/seek scheduling algorithm such as C-SCAN or Elevator etc.*


!~~~~Probable problems~~~~
-The major problem can be that of inflexibility, in the sense that you won't be able to switch filesystems eg. in normal, standard scenario, the same hard disk can be used to store files in FAT as well as ext3 filesystem at different times; but this won't be possible as easily and on the fly since the logic implementation is in hardware.
-If implemented using a standard microcontroller, operation & data transfer speeds can be far less than normal.
-Even though we are implementing some FS handling code in hardware, we will need our own software driver to give specific commands to our device. Thus, inevitably, we gotta implement a software driver; the only advantage is that much more processing is done in hardware controller parallel to the central processor, but still the overhead 'system call' to kernel code prevails.
-Whether it is the storage device or our device, we need to handle a device.
-Due to limited memory in or interfaceable with the microcontroller, the number of files that can be opened and handled is limited. The file bufers are also limited.If this memory is increased, cost shoots up.
-Designing it so as to support ALL existing and future Operating Systems is a difficult job.
-It may not be accessible under Cave-man-os like MSDOS. Even if you write software driver for it, you may need some other native bootable disk.
-Handling some OS and FS specific details and constraints such as users, security info, their quotas, their encryption keys etc. will need to be incorporated in software driver. It is a mammoth task for embedded processor. Excellent examples are NTFS and ext filesystems. No idea about RAID.

[dc0d32]

!~~~~Further Enhancements~~~~
-The device can be made self programmable. The driver should be able to flash the uC to handle whatever filesystem user wants to use or switch to. The driver will have all the microcodes. If and when required, the device should allow in site programming [the 'burn' mode]. This is possible with many microcontrollers with external EEPROM code memory support, now a days in the market.
-Instead of microcontroller, we can use our own microcode operated control unit [kind of 'tweaked microprocessor' or custom processor core that will be dedicated to FS handling only](may prove effective, but costly)
-We can add encryption support in our device, so that replacing the disks, trying to read the disk on another machine is not possible. It will need 'the same chip', even brothers won't work.
-support for compression can also be added [may be useless, considering the disk sizes now a days!]


!~~~~Queries~~~~
-Can the high speeds & data transfer rates be maintained along with the embedded FS handling?
-Can this be done using/in the IDE controller itself? Instead of it being an externally pluggable device, can we incorporate it in motherboard directly? [since it is programmable, it is universal].
-Can we use it for RAID? CD-DVD-R/RW?
-Can we use it to have a DVD-RW disk treated as a hard disk? Packet writing mode. (I know there are hell lot of problems in doing so)
-What will be the performance degradation if we try to support more than one storage devices?
-What about DMA? Is it helpful? Necessary?
-Can we use the system RAM directly with our hardware? (the other way is : we can use it using software driver which allocates memory for the device to use, and device talks to driver in terms to data as memory chunks. concept very similar to paging, RAM here acts as pagefile. This may not be efficient)
-How to store extra "Operating System specific" information about an opened file?



It just clicked and I wrote it. May not be practical. Or even may be in market.

Please enlighten

[Cjmovie]

This is actually the design used in many embedded systems (especially those using media such as flash). The main problem is, of course, that you are stuck with your filesystem. Sure, you can flash it -- But then you would lose support for the other. What if you had a 200GB hard drive with multitudes of partitions, each with a different file system? You would need a lot more memory. You would also have to support reading from each at the same time, mixing file handles, and all the like. Could get very complicated.

But really, if you ask me, it rather goes against the entire design behind a PC; That is, a PC is supposed to be a very generalized computing 'thing', that can compute anything that is computable, and to do anything in any way you program it. Something like that. I'm not good with words.

But basically, it's just another constraint.

More issues: A microcontroller would just plain not cut it. It would need to run at at least the speed it could read from the hard drive. And the read speeds just keep going down (however, we will probably soon reach the limit...with current technology, that is). Otherwise you're slowing down the computer by having slow links. You'd have to basically get another processor -- just like they do for GPU's -- all for the filesystem.

On a brighter side: Maybe this is where computers will end up? Who knows? There are PHYSICS processing units now. Will a computer in 20 years be boasting that new graphics card, a physics card, filesystem controller card, and who knows what else?

[JAAman]

filesystem embeded into the firmware? that sounds like my old 1541!!

easy to program for, but very inflexable (and the primary reason c-64 programs don't run properly in emulators)

the 1541 disk drive had a 6502 CPU in it (a slightly slower version of the same CPU in the C-64) and was fully programmable -- you could program it to do almost anything!

[Colonel Kernel]

The main rationale for this idea is to avoid consuming CPU time, right? In a few years, we'll all have more processor cores than we know what to do with. Why not just use some of them?

[Cjmovie]

I could still see a few uses for this, especially for very-specialized computers. For instance, a huge banking database...For stuff like that, though, you might even just want completly dedicated hardware hard-wired to the job.

Hardware will always be faster than software because hardware interprets software. Hardware will always be less flexible than software because it is..er...'hard', but software is 'soft'. What we need is a balance between the two that allows all the flexibility we want/need while having as many speed benefits from hardware as possible.

[dh]

rofl @ 'soft' vs. 'hard'

I think it would be usefull to be able to have processors each component of a computer. For instance, more recent Graphics (eg. the Nvidia/ATI generation) cards have highly specialized processors meant for vertex shading and the whole lot. If a hard disk had [say] a P-IV which had it's own 8mb of ram; you could load in components for reading various filesystems, then present the data to a central core (set). This would simplify the job of programming filesystem interfaces, because (in theory) all block devices would use a similar/same interface (the only concievable difference being Floppies don't usually sport partitions ) and thus allow easier sharing of code. In fact, I'd dare to venture and state that you may even be able to create a filesystem at the beginning of the disk that was meant to contain persistant settings for such a processor. You could store drivers that are loaded automatically during POST (independant of the BIOS, or prehaps after being issued an interrupt) and provide an interface for working with this actual sector, giving the ability to alter these settings. Another idea would deny the user of access to this sector (mind you, im using sector in a generic sense and not refering to the literal disk meaning) and provide only some basic commands: "read", "write", "move" etc. and the filesystem could be device dependant (i can see the articles now: "new benchmarks from X-Site show Y-Speed by the new Z-Model! Blazing fast read/write speeds credited to A-Technology, an update to the B-Filesystem). You could also use this to provide a LOW level encryption, processing each bit as it is read/written.

Enought Rambling by me though, DH.