Determining which driver called kernel function

[onlyonemac]

Hi,

I've got two layers of drivers: storage device drivers, and filesystem drivers. Both drivers have a single entry point, which is passed a parameter giving the desired function (for storage device drivers, things such as "read 3 sectors at offset 5 into the specified buffer"; for filesystem drivers, things such as "get the name of the next sibling to the specified node") and a number of parameters as required for the desired function (for example, the number and offset of the sectors to read and a pointer to a buffer to read them into). (In the event that the same driver is used for multiple devices or for multiple filesystems across different devices, multiple instances are used and are treated by the kernel as completely separate drivers.)

Now I'm wanting to pass my filesystem drivers two function pointers, one that will read an arbitrary sector from the underlying storage device and one that will write an arbitrary sector to the underlying storage device. These functions will take care of passing the request to the correct storage device driver, and of adding any offset to the requested sector (so that the filesystem driver can request "sector 0" and get the first sector of the partition where the filesystem resides, instead of the first sector of the entire device). Consequently, these functions will need to have some way of knowing which filesystem driver has called them so that they know which storage device driver to pass the request to.

So the question is, what is the best way to do this? So far I have thought of a few possible methods:

1. Copying a "template" function to a newly-allocated block of memory and patching it to hardcode some identifier for the storage device driver to use
2. Using a single function that is given to all filesystem drivers, which when called will perform a stack backtrace to determine which filesystem driver entry point function was called
3. Using a single function that is given to all filesystem drivers, and passing an additional "black box" parameter to identify the filesystem driver to the kernel and which the filesystem drivers are required to pass to the given "read" and "write" functions (this option might ultimately be the best, as the same "black box" parameter can be later used in other driver-to-kernel calls that might be added at a later time)

None of these sound particularly elegant to me. Is there an "official" way to do this? If not, which of these would be the best?

Thanks,

onlyonemac

[Boris]

I don't think your storage drivers should depend on the notion of filesystem driver. Especially when you will need something else than file systems on the top of disks ( like swap , some database stuff , or some custom logical partition mechanism)

The storage layer should fill buffers then send a message ( or simply return a status like a number of bytes written) to the caller.

We will be more able to help you if you tell us how your drivers are isolated, if they can call themselves directly, or if they use IPC..

[SpyderTL]

Agreed. Partitions aren't part of the filesystem, so your partition driver should fit right between your storage device driver and your filesystem driver.

The way I'm implementing this in my OS is that all of these "driver" layers will implement the same two interfaces, which I call "Reader" and "Writer".

The reader interface includes position, skip, read byte, read integer, read long, read string, and fill an object (buffer). The writer interface has similar write methods.

Any number of readers can be chained together to "transform" the data from one format to the next, and I even have readers/writers for memory, network connections, audio devices and I'm currently working on serial ports. So I can pass any reader or chain of readers to the sound card and play, for instance, raw data from an audio CD.

This is similar to the way that Linux chains StdIn and StdOut channels together.

[onlyonemac]

Each driver (either storage device or filesystem) provides the kernel with a single entry point when it is registered with the kernel. This entry point is called every time the kernel needs the driver to do something, and is passed (among other parameters) a parameter indicating what the kernel wants the driver to do and whatever data the driver needs to do whatever the kernel is asking the driver to do (for example, to write data in a storage device driver, the kernel needs to pass the driver a source data pointer, an offset, and a length).

Storage device drivers have no concept of filesystem drivers, and filesystem drivers have no concept of storage device drivers. Storage device drivers just read and write bytes to and from storage devices, and filesystem drivers just parse and update filesystem trees, returning and modifying file names, file sizes, and file data. Storage device drivers don't care what the kernel is using the bytes that are read for, and filesystem drivers don't care where the underlying data is stored.

But, and here's the problem, filesystem drivers *do* care about having a way to get the underlying data. So my intention is to give filesystem drivers a handy pair of (kernel) functions that they can call which will return a requested sector or sectors from the underlying storage device (whether that's a hard disk, a floppy disk, a RAM disk, an iSCSI network disk, or anything else doesn't matter and doesn't affect the filesystem driver's operation). It is these functions that will also take care of the "offset" of the filesystem within the storage device (i.e. disk partitioning), not the filesystem driver nor the storage device driver (and in the future, if I ever decide that I need a partition driver layer, I can easily delegate this responsibility to the partition driver). But the problem is, these functions need to know what filesystem driver is calling them so that they know what underlying storage device to use. And this brings me back to my original question, and the list of three suggested options that I gave in my first post.

[Kevin]

The normal way to do this is that file system driver has some information about the storage it is sitting on. This doesn't have to be a pointer to a struct blockdev if you don't want the file system driver to mess with the internals of the underlying storage, but it can be a black-box for the filesystem driver, like some kind of handle or file descriptor that refers to the storage. And then it's this information that it passes to the read/write functions.

Of course, it could also pass itself (like some struct filesystem*) to those functions, but I wouldn't that consider a good design. Upper layers need to call into a specific lower layer, so they must know about it. Lower layers, however, should ideally never know or have access to their users.

[onlyonemac]

Of course, it could also pass itself (like some struct filesystem*) to those functions, but I wouldn't that consider a good design. Upper layers need to call into a specific lower layer, so they must know about it. Lower layers, however, should ideally never know or have access to their users.

I might do it that way, as it avoids the need to black-box the device drivers. The device drivers will not know anything about the filesystem layer, because the call goes through the kernel. So it could go like this:

1. Filesystem driver calls "read" function provided by kernel, passing the filesystem driver's identifier and the desired sector(s)
2. Kernel (in read function) looks the filesystem driver's identifier up in a table and finds the corresponding storage device driver
3. Kernel calls storage device driver and passes the desired sector(s)
4. Storage device driver receives request to read sector(s)

This way, the filesystem driver doesn't need to store and pass a meaningless black box; rather, it has a somewhat meaningful identifier that can be used in all (future) instances where the filesystem driver needs to call the kernel. The storage device driver also has no knowledge of where the request has come from.

[Combuster]

Don't make the mistake of thinking in implementation details. This is something that simply begs to be analysed with (in this case, basic) object oriented design. What are the things you want to do with raw storage? Reading and writing blocks:

Code: Select all

interface BlockDriver { 
    (...)readSectors(...) 
    (...)writeSectors(...)
}

Then you have your basic drivers for that:

Code: Select all

class Floppy implements BlockDriver;
class Harddisk implements BlockDriver;
class CD implements BlockDriver;

And of course, your virtual drivers:

Code: Select all

class Partition implements BlockDriver
{
    BlockDriver actualStorage;
    (...)offset;
    (...)size;
}

Your filesystems typically operate on a storage medium

Code: Select all

class FAT implements FilesystemDriver
{
    BlockStorage storage;

    public constructor(BlockDriver storage)
    {
        this.storage = storage;
        storage.readSectors(...) // read the superblock 

And all you need to do is convert that structure to your favourite programming language, which basically amount to pass "this" pointers explicitly in languages that don't provide you with classes at the language level.

[onlyonemac]

Don't make the mistake of thinking in implementation details. This is something that simply begs to be analysed with (in this case, basic) object oriented design. What are the things you want to do with raw storage? Reading and writing blocks:

Code: Select all

interface BlockDriver { 
    (...)readSectors(...) 
    (...)writeSectors(...)
}

Then you have your basic drivers for that:

Code: Select all

class Floppy implements BlockDriver;
class Harddisk implements BlockDriver;
class CD implements BlockDriver;

And of course, your virtual drivers:

Code: Select all

class Partition implements BlockDriver
{
    BlockDriver actualStorage;
    (...)offset;
    (...)size;
}

Your filesystems typically operate on a storage medium

Code: Select all

class FAT implements FilesystemDriver
{
    BlockStorage storage;

    public constructor(BlockDriver storage)
    {
        this.storage = storage;
        storage.readSectors(...) // read the superblock 

And all you need to do is convert that structure to your favourite programming language, which basically amount to pass "this" pointers explicitly in languages that don't provide you with classes at the language level.

I understand all that. But I need to think about implementation details, because this is the next thing that I need to implement.

Each instance of each driver is able to request a "private data" area that is passed to it with each subsequent call (in other words, a "this" or "self" pointer). But my drivers are able to have only a single entry point, so the problem is that my filesystem drivers would have to make all calls to my device drivers through a single function (which takes as a parameter the actual operation to carry out), and if my filesystem drivers are calling my storage device drivers directly this would mean that the former would have to know the details of how to call the latter, something which should only be of concern to the kernel (I also think that having drivers from different layers call each other directly is a bad design idea, because then the interface of one layer cannot change without breaking the dependent layers).

So instead I'm getting the filesystem drivers to make all "read" and "write" calls to the kernel. The filesystem drivers can call the "read" function can get, simply, a pointer to a block of memory containing the requested data (to be freed by the filesystem driver later with kfree), and the "write" function with a pointer to a block of memory to write to the storage device, and not worry about how the storage device drivers are implemented.

[Combuster]

Even if all calls get channelled over one entry point, you can still write your marshalling and unmarshalling code and basically end up with plain function calls on classes. I even have an IDL compiler that constructs headers and all the necessary glue from a bunch of simple function definitions to get type safety (and enforce protocol integrity against malicious clients) over an otherwise untyped IPC channel.

[onlyonemac]

Even if all calls get channelled over one entry point, you can still write your marshalling and unmarshalling code and basically end up with plain function calls on classes. I even have an IDL compiler that constructs headers and all the necessary glue from a bunch of simple function definitions to get type safety (and enforce protocol integrity against malicious clients) over an otherwise untyped IPC channel.

Are you suggesting that I drag a whole pile of glue code into each and every driver at compile time when I could just put it in the kernel? (Not to mention that putting it in the kernel has other advantages as I have already described.)

[SpyderTL]

But my drivers are able to have only a single entry point, so the problem is that my filesystem drivers would have to make all calls to my device drivers through a single function (which takes as a parameter the actual operation to carry out), and if my filesystem drivers are calling my storage device drivers directly this would mean that the former would have to know the details of how to call the latter, something which should only be of concern to the kernel (I also think that having drivers from different layers call each other directly is a bad design idea, because then the interface of one layer cannot change without breaking the dependent layers).

Having one function doesn't simplify your "interface". It just moves it down one layer. Consider how some devices map their entire register set into memory, and other devices map only two addresses, a "selector" register and a "data" register. The advantage is that you use fewer memory addresses, which are limited resources, but it doesn't "simplify" the code that is using those two registers. In fact, it's pretty much the opposite.

So instead I'm getting the filesystem drivers to make all "read" and "write" calls to the kernel. The filesystem drivers can call the "read" function can get, simply, a pointer to a block of memory containing the requested data (to be freed by the filesystem driver later with kfree), and the "write" function with a pointer to a block of memory to write to the storage device, and not worry about how the storage device drivers are implemented.

This is similar to the way that Windows works. When you "open" a device, you get a "handle", which is just some sort of identifier that everyone can agree on. Then all subsequent calls pass this identifier as a reference to the device to perform the action on. This will work, obviously, but if you look at the Windows API, you'll notice right away that there is a problem with this approach. Over time, you'll discover that the calls to the kernel need to be modified, but due to backwards compatibility, you won't be able to modify them. So the only real option will be to create new ones, and call them, say, OpenDeviceEx().

So the problem (breaking changes at the interface level) really never went away, you just moved it around a bit. It's a problem that no one has ever really solved, because the real problem is "dependency". As soon as you call another function, class, module, library, etc., you now have a dependency on it, and it can never change without causing issues. The best you can do is design your system in a way that reduces the number of dependencies needed. The best way that I know of to do this is to make all of your communications one-way, and put them in queues that can be "broadcast" to the entire system, and have "listeners" that act on those events.

However, to build a system entirely made up of broadcast events would be quite an achievement, and would probably suffer when it came to performance. So it's up to you how far you are willing to go to try to prevent changes from breaking dependent components.

[onlyonemac]

But my drivers are able to have only a single entry point, so the problem is that my filesystem drivers would have to make all calls to my device drivers through a single function (which takes as a parameter the actual operation to carry out), and if my filesystem drivers are calling my storage device drivers directly this would mean that the former would have to know the details of how to call the latter, something which should only be of concern to the kernel (I also think that having drivers from different layers call each other directly is a bad design idea, because then the interface of one layer cannot change without breaking the dependent layers).

Having one function doesn't simplify your "interface". It just moves it down one layer.

I never claimed that it simplified it. It simplifies the implementation from the kernel side, as I can keep a single function pointer through which I can perform any function that is applicable to the driver, instead of having to store multiple pointers or worry about shared object loading.

So the problem (breaking changes at the interface level) really never went away, you just moved it around a bit. It's a problem that no one has ever really solved, because the real problem is "dependency". As soon as you call another function, class, module, library, etc., you now have a dependency on it, and it can never change without causing issues. The best you can do is design your system in a way that reduces the number of dependencies needed. The best way that I know of to do this is to make all of your communications one-way, and put them in queues that can be "broadcast" to the entire system, and have "listeners" that act on those events.

Which is why I'm passing the calls through the kernel. That way, I can change my storage device driver architecture completely and not have any problems using existing filesystem drivers, because the kernel can still expose two functions for reading and writing data to and from the underlying storage device.

[Brendan]

Hi,

I've got two layers of drivers: storage device drivers, and filesystem drivers.

I typically have 3 things:

• VFS
• File systems
• Storage drivers

In the VFS; a storage device is mostly a file (e.g. like "/dev/sda"). Just like a file you can open it (and get a file handle), then read or write "X bytes at offset Y" using the file handle, and close it. Of course it's not quite that simple as there's a few differences between a normal file and a storage device - you can't append or truncate, but can get information for optimisation (e.g. native block size) and there's possibly special functionality (secure erase, eject, trim, ..).

A file system is something that mounts a file.

You can have multiple nested file systems. For example:

• You might mount the file "/dev/sda" with the GPR partitioning file system, and this file system might contain the file "/dev/sda1"
• You might mount the file "/dev/sda1" with the ext2 file system, and this file system might contain the file "floppy.img"
• You might mount the file "floppy.img" with the FAT12 file system, and that file system might contain the file "foo.zip"
• You might mount the file "foo.zip" with the PKZIP file system, and that file system might contain the file "hello.txt"

When you read 123 bytes from "hello.txt", one file system asks the next for the data, which asks the next, and so on (up until it reaches the storage device driver).

You can have multiple nested storage device drivers too. For example:

• The file "/dev/mda_z" might be provided by a whole disk encryption storage device from the file "/dev/mda"
• The file "/dev/mda" might be created from a software RAID layer from the 2 files "/dev/sdb" and "/dev/sdc"
• The files "/dev/sdb" and "/dev/sdc" might be physical devices

When you read 123 bytes at offset 456789 from "hello.txt", one file system asks the next for the data and so on; up until it reaches the first storage device driver. The first storage device driver might get a similar "read 123 bytes from offset 987654321" request, and then ask the next storage driver/s to fetch data it needs, which might ask the next storage driver/s, and so on (up until it's satisfied by a physical device).

Of course the VFS is responsible for caching file (and directory) information, and keeping track of mount points, etc; which means that none of the file systems or storage devices are involved for a "VFS cache hit". It's only "VFS cache miss" and writes that matter, but these require disk IO (and the overhead of multiple software layers is likely to be negligible compared to the time it takes to access physical devices).

Note that all of this uses something like "read_from_file(handle, offset, size, address_of_buffer)" and "write_to_file(handle, offset, size, address_of_buffer)". Once a file is opened, you can read/write data without caring if it's a traditional file (like "hello.txt") or if it's a storage device (like "/dev/sda"). That's what makes it insanely flexible (e.g. makes it so you can nest file systems and/or storage devices). It's all the other stuff (whether you can append/truncate/resize, whether you can eject, etc) that makes storage devices different to traditional files.


Cheers,

Brendan

[SpyderTL]

It simplifies the implementation from the kernel side, as I can keep a single function pointer through which I can perform any function that is applicable to the driver

Of course, it's your design, and this will obviously work. I just see red flags when someone says that they are trying to "future-proof" their design by being clever. It usually ends up meaning more work in the long run.

All that said, my design is pretty much the same as yours, in that at a low level, I have one function named "ExecuteMethod", that takes a method name, and that is evaluated at run time with each call. I have the option to evaluate the method once, and make multiple calls to it, but I rarely use it because it takes longer to code, and I'm not too concerned with performance at this point.

However, the difference is that my driver "layers" know absolutely nothing about who is calling them, and absolutely nothing about who they are calling other than they implement the Reader and/or Writer interface. I leave it up to the application to connect the pieces in the proper order, which allows me to inject additional layers, as needed.

Another advantage is that I can also choose whether to inject "real" drivers, to improve performance (essentially giving an application exclusive access), or I can inject "virtual" drivers, to allow shared access to multiple applications. And they all use the same base interface, so that, hopefully, the application works exactly the same, either way.

[onlyonemac]

However, the difference is that my driver "layers" know absolutely nothing about who is calling them, and absolutely nothing about who they are calling other than they implement the Reader and/or Writer interface. I leave it up to the application to connect the pieces in the proper order, which allows me to inject additional layers, as needed.

Another advantage is that I can also choose whether to inject "real" drivers, to improve performance (essentially giving an application exclusive access), or I can inject "virtual" drivers, to allow shared access to multiple applications. And they all use the same base interface, so that, hopefully, the application works exactly the same, either way.

Mine is exactly the same in that regard. My storage device drivers don't know where the call originates from (the filesystem driver just knows that its entry point was called, and isn't passed any parameters beyond those needed to tell it what operation to carry out - and the calls are currently all being passed through the kernel) and my filesystem drivers don't know what storage device driver they're calling because they're just passed two function pointers, a "read" one and a "write" one, which again are currently being passed through the kernel.

Consequently, the kernel can inject whatever drivers and layers of drivers it wants. The kernel can easily nest storage devices, by for example processing a "read" call itself by reading from a different filesystem (to implement a loopback file, for example).