Modules Tutorial....?

[piranha]

Does anyone here know a good tutorial for Module Loading/Unloading etc...?

I already searched the web.

-JL

[elfenix]

I assume you are talking about the ability to extend your kernel at runtime?

I will stand by the statement that module loading is almost always a "Bad Idea".

A module loader is not really anything more than a specialized executable loader. So, just think about writing an executable loader, and placing that executable in kernel space, and go from there.

I still say it's a "Bad Idea".

[Combuster]

I will stand by the statement that module loading is almost always a "Bad Idea".

Any arguments to support this statement? Both windows and linux support it and they seem to have no intentions of removing it so it can't be as bad as a design error.

Does anyone here know a good tutorial

The problem is, tutorials don't teach you to figure things on your own. And in the context of OS development, tutorials are rare and limited to the basic steps. Loading modules is highly kernel-specific so do not count on the existance of any tutorial, let alone that it applies to your kernel.

The key points are obvious:
1: you must be able to load and parse your favorite executable format (here it starts already: ELF? AOUT? PE?) (read the docs on the format for this one)
2: you must have relocation support and the ability to locate symbols - (once again, read the docs on the format)
3: you must have dynamic linking so that you can resolve symbols that are provided by the kernel (this one's the trickiest - how to get the addresses of all exportable functions somewhere where you can handle it)
4: you must be able to add to kernel memory and have it visible for all processes (you'll probably have this one already )

if you can do that, you select an address to load a module, load it there, patch all the references to the kernel, look up the entry and exit functions and their address, then call them and voilá: module support.

[AndrewAPrice]

Does anyone here know a good tutorial for Module Loading/Unloading etc...?

I already searched the web.

-JL

Don't look at tutorials They're only to start you off.

a) You don't really learn much from tutorials. You could read it over and over again, and you THINK you're absorbing what you're reading, but realistically you're just copying someone else's code. You're not figuring anything out for yourself.

b) The only person who knows how your code works is you, so copying some step by step tutorial code won't automatically work in your kernel. Their example kernel and your kernel are probably laid out completely different - and you won't really know how their code works so you'll doing a lot of dodgey 'patch' code and you'll end up with inefficent speghettie code bloat.

Anyway, if you want to know HOW to load a GRUB module, there's a few key points you need to know:

- A GRUB is loaded by directly dumping the contents of a file into memory straight after where the kernel is loaded. There are no limitations of what sort of file a module is (executable, a text config script, etc).

- GRUB gives you a pointer to a multiboot structure upon entering your kernel code. GRUB provides a multi-boot C header you can include which describes all of the multiboot structures.

- One of the members of this header is a pointer to other headers (I don't remember which one off the top of my head - but look in the header and it'll be self-explanatory) which describe the number of modules, the size of the modules, their address in memory, and a pointer to a string describing the file name of the module.

- What you do with this file is up to you. If it is a config file, you can parse the text in it like any other block of ASCII in memory. If it is an executable, you will need to parse the executable format (e.g. ELF), set up it's process memory, etc (it's a great way to get program's loading without having disk/fs drivers).

[AJ]

To reiterate, I would strongly recommend learning about the ELF specification (have a look here for the 64 bit version). Loading programs from this is easy (just look at the program headers) and loading modules is not much mroe difficult.

If you want to see how Linux does it, try here.

Cheers,
Adam

[elfenix]

Any arguments to support this statement? Both windows and linux support it and they seem to have no intentions of removing it so it can't be as bad as a design error.

Countless hours of banging my head on my desk dealing with module unloading in linux have led me to a 'bad gut feeling' about this. The real points have been made countless times in micro versus monolithic kernel debates. I guess a microkernel is really doing 'modules' at heart anyway. Anyway, don't see much need in going into that can of warms again, it's been gone into again and again.

I fall in the microkernel line of thought, and the 'module' method seems like adding more complication in a monolithic kernel design for practical benefits that just don't matter for a hobby design. (Unless some aspect of that design REQUIRES modules for it's neato factor.) The real benefits (and reason why) linux and NT use modules are greatly diminished in a hobby OS:

1. The abstraction gained will likely be lost by the fact the developer has complete knowledge of the rest of the kernel.
2. That same abstraction could be gained by designing your kernel appropriately.
3. You probably aren't going to be running this thing on any range of hardware (other than your computer), and chances are near 0 that another developer will code up a driver.

Micro or monolithic kernel design, you need to get to the point of task switching and running programs before thinking of either - and at that point, wouldn't need a tutorial.

[JamesM]

Any arguments to support this statement? Both windows and linux support it and they seem to have no intentions of removing it so it can't be as bad as a design error.

Countless hours of banging my head on my desk dealing with module unloading in linux have led me to a 'bad gut feeling' about this. The real points have been made countless times in micro versus monolithic kernel debates. I guess a microkernel is really doing 'modules' at heart anyway. Anyway, don't see much need in going into that can of warms again, it's been gone into again and again.

I fall in the microkernel line of thought, and the 'module' method seems like adding more complication in a monolithic kernel design for practical benefits that just don't matter for a hobby design. (Unless some aspect of that design REQUIRES modules for it's neato factor.) The real benefits (and reason why) linux and NT use modules are greatly diminished in a hobby OS:

1. The abstraction gained will likely be lost by the fact the developer has complete knowledge of the rest of the kernel.
2. That same abstraction could be gained by designing your kernel appropriately.
3. You probably aren't going to be running this thing on any range of hardware (other than your computer), and chances are near 0 that another developer will code up a driver.

Micro or monolithic kernel design, you need to get to the point of task switching and running programs before thinking of either - and at that point, wouldn't need a tutorial.

You missed the main, important reason why monolithic kernels use modules in the first place:

Modules enable drivers to be paged in across all address spaces, meaning a page-dir switch and messy memcpys/page linking isn't required for normal operations.

[Combuster]

Countless hours of banging my head on my desk dealing with module unloading in linux have led me to a 'bad gut feeling' about this.

That is module unloading, and a specific instance of it. That it is linux does not mean it can't be done better.

The real points have been made countless times in micro versus monolithic kernel debates. I guess a microkernel is really doing 'modules' at heart anyway. Anyway, don't see much need in going into that can of warms again, it's been gone into again and again.

The key difference is that drivers are loaded in either kernel space or user space. Monolithics are faster, Micros are more secure. Given that the OP wanted module loading I guess he already made the choice.

I fall in the microkernel line of thought, and the 'module' method seems like adding more complication in a monolithic kernel design for practical benefits that just don't matter for a hobby design.

Even hobbyists can be perfectionists. Having the perfect design can be more satisfying than having a working kernel. The Eleanore Semaphore archetype isn't too rare at all.

The real benefits (and reason why) linux and NT use modules are greatly diminished in a hobby OS:

1. The abstraction gained will likely be lost by the fact the developer has complete knowledge of the rest of the kernel.

That doesn't mean that the writer adheres to his own design principles - you have coders and good coders. Not writing kluges is a key feature of the latter.

2. That same abstraction could be gained by designing your kernel appropriately.

That's not the point of modules - keeping the unneccessary stuff out of the main kernel.

3. You probably aren't going to be running this thing on any range of hardware (other than your computer), and chances are near 0 that another developer will code up a driver.

Again, you're making a huge guesstimation on the OP's ambitions. Still, retrofitting is worse than having it from the start. Even if you want to support both your desktop and laptop module loading can already be handy.

Micro or monolithic kernel design, you need to get to the point of task switching and running programs before thinking of either - and at that point, wouldn't need a tutorial.

Before you can load your programs you must have some location to load them from. If you want to have something you can show off with, you need at least a video driver. For which you need drivers, for which you need to have made the micro/monolith decision.

[mystran]

IMHO module loading isn't necessary a bad idea. Module unloading is more likely to cause trouble. But both can work just fine..

It's no different from loading/unloading a plugin to/from an application: you load the module into memory, then dynamically link it with the host code (kernel or application). When you want to unload it, you ask it to clean up after itself, then finally free the memory. If you've never loaded dynamic libraries into an application, I'd start by doing that, and play around a bit. In Unix that means dlopen() and friends.

Once you're happy with that stuff, the next thing I'd do, is try loading a library/object of some sort into a normal application manually (that is, without calling dlopen() and friends. Once you can do that in an application, it should be easy to do the same in kernel.

[elfenix]

I'm not really one for argument, and this is a matter of opinion. Given that kernel "modules" can often be termed drivers, kernel extensions, etc.. my thoughts here were specifically about linux-style loadable and unloadable during runtime, after boot. I don't have a problem with "load once at boot" modules in a monolithic kernel design.

The paging argument completely misses the point, because that can be accomplished with a monolithic driver without "modules" - it's a monolithic versus micro argument. Depending on your architecture, you could even do a Microkernel approach and avoid that as well

The sentiment that loading/unloading a kernel module is the same as a program plugin is the main reason I would advocate against this. Loading and unloading kernel code in the same address space is a very different animal than user space. You might use the same methods, but the complications are completely different.

I have about as much faith in kernel modules being stable as I do in microkernels outperforming monolithic -it can and has happened, but don't hold your breath...

Driver unloading can have more than a few complications. The fact is, the linux kernel has some really awesome developers, paid and working full time, and to say that somone in their basement can do better is unrealistic at best.

Here's a scenario:

What happens if the module is controlling a bus? Let's say I'm implementing an MMC/SD host driver and someone connects a wireless card. User space depends on the wireless card, but doesn't really look at the bus. So, you're module infrastructure now needs some form of tree based dependency management to make sure everything gets loaded properly.

You load your module, which creates an interrupt handler, and turns power on to component A. That causes an effect in component B. You unload your handler, but the system state change in component B remains, and that state change will require handling somehow. Maybe this is a documented thing, maybe not - but now you need to add code in component A to cleanup component B (which is a different module), or add code in component B to cleanup for A... The whole thing gets messy quick.

On a final note, being at the point of task switching and running programs does not imply that you have any sort of driver framework. I'd actually recommend using a smart boot loader, and passing through "my first program" as a binary image to the kernel. It doesn't matter where you load it, just that you can get it loaded and execute code there. And the ability to do that would need to be accomplished before loadable modules in either a micro or monolithic kernel. If you don't have the ability to do that, then modules are impossible anyway, unless they are part of the kernel, in which case, they aren't modules....

A don't know why you mentioned retrofitting either. Using linux as an example, it's perfectly possible to design subsystems such that they are compiled into the kernel, but can be removed entirely or loaded/unloaded at runtime. My recommendation is to do the design up front, but save the loading/unloading until enough of the system is built up around it.

And, as a matter of opinion, module unloading is, was, and remains a bad idea. Microkernels at least give a minimum of protection from nasty hardware scenarios...

[JamesM]

because that can be accomplished with a monolithic driver without "modules" - it's a monolithic versus micro argument.

And the ability to do that would need to be accomplished before loadable modules in either a micro or monolithic kernel.

You seem to be confusing what a microkernel is: Microkernels do not have 'modules' at all. All their 'drivers' are user space processes and as such there is no 'loading' or 'unloading', merely fork()/exec() and kill(). The argument seems to be about having the ability to dynamically load/unload modules as opposed to statically compiling them into the kernel. And make no mistake, "Loading modules once at boot" is a complete waste of time. You might as well just have everything statically #ifdef'd into the kernel and recompile when you want a new 'module'.

and passing through "my first program" as a binary image to the kernel.

That's what an initial ramdisk is for.

A don't know why you mentioned retrofitting either. Using linux as an example, it's perfectly possible to design subsystems such that they are compiled into the kernel, but can be removed entirely or loaded/unloaded at runtime.

Maybe you misunderstand the context Combuster used the term 'retrofitting'. He was implying that your assumption that "You aren't going to be running this thing on any range of hardware" may prove to be false, and so you would end up trying to retrofit a module loading/unloading architecture into an architecture with all drivers statically compiled in. Bad Idea (tm).

Plus I'd like to give you a fantastic example of retrofitting gone wrong, from Linux: Video drivers and support for accelerated graphics cards. When they designed Xorg they 'assumed' that video wasn't important and left it out. Then they fell behind and had to kludge it in so now all video drivers are actually part of the Xorg server. Hmm.

And, as a matter of opinion, module unloading is, was, and remains a bad idea. Microkernels at least give a minimum of protection from nasty hardware scenarios...

Yeh wha'? Those two sentences are completely disjoint! You can have a monolithic kernel without module unloading, so why mention microkernels?

You load your module, which creates an interrupt handler, and turns power on to component A. That causes an effect in component B. You unload your handler, but the system state change in component B remains, and that state change will require handling somehow. Maybe this is a documented thing, maybe not - but now you need to add code in component A to cleanup component B (which is a different module), or add code in component B to cleanup for A... The whole thing gets messy quick.

I don't understand this. Here you're implying that 'components' are actual physical devices, but then you say "cleanup component B (which is a different module)" - but I thought B wasn't a module, but a device?

So the *right* way to do this is to either have both devices A and B managed by one module or to have each managed seperately but give an indication when you start/stop the module for A to the module for B. Reason? If component A implicitly causes a state change for component B, then the driver for component B MUST know/be told about it!

(I would also welcome a practical example of this as I can't think of one off the top of my head)

Cheers,

JamesM

[Combuster]

I'm not really one for argument, and this is a matter of opinion. Given that kernel "modules" can often be termed drivers, kernel extensions, etc.. my thoughts here were specifically about linux-style loadable and unloadable during runtime, after boot. I don't have a problem with "load once at boot" modules in a monolithic kernel design.

Good that we got that straightened out

The paging argument completely misses the point, because that can be accomplished with a monolithic driver without "modules" - it's a monolithic versus micro argument. Depending on your architecture, you could even do a Microkernel approach and avoid that as well

Life isn't binary - True monoliths have all drivers in the first loaded binary. Microkernels have every driver separate, and loaded into userspace. The hybrid form commonly used is the Modular Kernel: drivers in kernel space, but loaded dynamically. You could decide to put more in the kernel proper, or you could delegate parts to userspace striking a balance between the microkernel and the monolithic kernel. Modular kernels are popular because they are faster than microkernels, while not needing a rebuild eveytime you plug in a new PCI card (or become bloated).

The sentiment that loading/unloading a kernel module is the same as a program plugin is the main reason I would advocate against this. Loading and unloading kernel code in the same address space is a very different animal than user space. You might use the same methods, but the complications are completely different.

Kernel code and user code are different, and the same way are kernel modules and userland plugins. To make kernel modules be a bad thing, you just assumed that writing kernel code must be a bad thing. Again, we are NOT discussing a microkernel.

I have about as much faith in kernel modules being stable as I do in microkernels outperforming monolithic -it can and has happened, but don't hold your breath...

micro vs mono. Off topic and a big "No Duh"

Driver unloading can have more than a few complications. The fact is, the linux kernel has some really awesome developers, paid and working full time, and to say that somone in their basement can do better is unrealistic at best.

Linux originally didn't support modules - it is a feature added later. Home developers have the advantage of putting this in from the beginning. Besides, the basement geek only needs two or three drivers to control his computer as opposed to the hundreds that were written for linux.

BTW There's enough proof of the opposite if you would just took the time to look for it (Brendan, Candy, Colonel Kernel, and most likely quite a few more who haven't shown off their programming mastery yet)

Here's a scenario

Dependencies are indeed a thing to take care of. But once again, this isn't module specific, or even kernel-specific. The same could happen in a microkernel. Or userspace. Try writing a mod for Unreal Tournament

On a final note, being at the point of task switching and running programs does not imply that you have any sort of driver framework. I'd actually recommend using a smart boot loader, and passing through "my first program" as a binary image to the kernel. It doesn't matter where you load it, just that you can get it loaded and execute code there. And the ability to do that would need to be accomplished before loadable modules in either a micro or monolithic kernel. If you don't have the ability to do that, then modules are impossible anyway, unless they are part of the kernel, in which case, they aren't modules....

You didn't read the remark - even if you could run programs in userspace, you couldn't see that is was doing so without having a driver. You can start drivers without task switching or the like, then you'd at least have something to see what you are doing.

A don't know why you mentioned retrofitting either. Using linux as an example, it's perfectly possible to design subsystems such that they are compiled into the kernel, but can be removed entirely or loaded/unloaded at runtime. My recommendation is to do the design up front, but save the loading/unloading until enough of the system is built up around it.

that isn't retrofitting. That is delaying something until you have the base layer to build it on. What I meant is that with adding module loading to the linux kernel a huge number of things needed to be changed, and you will likely find traces of the two conflicting design concepts: drivers must be built into the kernel and allowing modules. If you assume that drivers are contained in the binary and later change that you'll suffer the consequences.
One of the design priniciples floating around states that you should rewrite code when your assumptions change. IIRC it was Abrash, but I don't have the book ready to quote from.

And, as a matter of opinion, module unloading is, was, and remains a bad idea. Microkernels at least give a minimum of protection from nasty hardware scenarios...

Once more, this isn't the place for micro vs monolith arguments.

[mystran]

BTW There's enough proof of the opposite if you would just took the time to look for it (Brendan, Candy, Colonel Kernel, and most likely quite a few more who haven't shown off their programming mastery yet)

Well, module loading and to a limited extend unloading is on my list of things to do in the near future as well.. just a few things before it, namely:

1. replacing wannabe-POSIXish ABI with a properly asynchronous event-based thingie.

2. UDP, then TCP, so I can have a HTTP server or something running on Voix. IP works already.

3. Basic GUI, so I don't have to mess with a single command line..

At that point, I'll be needing to load network drivers and video drivers, and I definitely don't want them all in the kernel all the time, yet I also don't want to run them completely in the userspace for performance reasons..

Anyway, considering how much time I have for my OS project nowadays, the ETA is probably a year or so... I do get paid for programming some other stuff after all.