OSDev.org

Designing my os, I have faced a very funny problem: what should be a module, and what should be built in?

If you ask me, I would love a perfect modular system: everything but the memory manager and low-level stuff like interrupts and IRQ's etc. are modules.

But the problem? Very similiar to the chicken and egg problem. For instance, each module is a '.fnc' file (functional module). But what if the floppy driver is a module? How to load it so you can load the others? OK, you say, use bios to get it before switching. But what if the filesystem (say FAT12) is parsed using a module? Your stuck again.

I'd love to see a better way to organize these things, any ideas would be great. I figure I better figure it out before getting past my current best (keyboard-shell-IRQ-GDT-etc.)

Defiently before I started the chicken and egg problem with a floppy driver!

Also, please excuse any typos, I am trying my best, but I'm in Las Vegas and using the TV built-in internet service with an infared keyboard that works badly that runs on windows 98 (first edition) from a shared server set for about 1800 people.

Thanks!

I'd generally try to keep kernel components from being modules to prevent that chicken and egg scenario.

I have no experience with writing kernel modules but I have written module systems for other purposes.

I had a change of views, my original one was to have EVERYTHING except the core things as modules but in the end this turned out not to be a good idea due to me ending up with tons of modules and having my users becoming frustrated with needing tons of modules to get anything near to what they wanted.

My second view was to have a strong and smart core which handled errors, kept information about my modules, was able to edit/load/unload them on the go and best of all it provided KEY FUNCTIONS NEEDED by the modules which greatly reduced the number of modules due to them now being wrapped around those key functions within the core.

Anyway this is turning into more of a rant so I hope you gathered something from this.

Regards,
Nelson

I would integrate some basic disk read functions and file system handling functions into the kernel.

Then i would load a more advanced Disk I/O and File System Module with that.

My startup order would be:

Kernel
Better Disk I/O
File System Handler Module
Keyboard (just incase there is an error and it needs an F1 to continue)
...
Other drivers
...
Mouse (not really needed until end anyway)

Then I would load the GUI or Console. I love consoles with mouse cursors in them. Its fun to have a pointer that doesnt do anything.

OK, I still have the problem the Modules were for:

I have a bit of trouble organizing my code FILES, I do fine inside the files though. Thing is, you can't exactly write a well structured OS source in a single file. So any tips on how to keep my files in order? I hate having things all in one directory, or each having its own directory. Naming conventions? I know most of this is opinion though . I guess source control gets into the topic. Any source control systems without server overhead (eg: local host only)?

I'd like to note also that the forum seems to be growing, the last time I checked in (always as guest I think....whatever) it took 2 days to get a reply, a day to get a view. So that's cool.

Oh, out of curiosity, what would you consider the SIMPLEST file system? I don' care about features. Or anything else. Even 8.3 names.

Your IPL (stage 1 bootloader, the thing that goes into the MBR) doesn't know a thing about file systems. It either loads sequential blocks off the hard drive (which means the stuff it loads must not move around, requiring a reinstall of the IPL when you recompile the stuff it loads), or it loads a file containing a list of sectors from a known location.

Loading more than one binary file into memory is just an extension of the principle. No chicken-and-egg there.

Or you are using GRUB, which supports loading modules (i.e., multiple binaries) as well.

Then there's the Linux initrd (initial ram disk), i.e. an image file that is loaded into memory which contains kernel binary and necessary modules in a filesystem known to the bootloader; enough to init everything so the rest of the OS can be read from more esoteric devices (like an USB stick formatted with ReiserFS or whatever). Personally, I hate that thing, as it is a pretty confusing concept. My Gentoo kernel is compiled with ReiserFS support compiled in, so the whole initrd stuff can be omitted.

Cjmovie wrote: I have a bit of trouble organizing my code FILES, I do fine inside the files though. Thing is, you can't exactly write a well structured OS source in a single file. So any tips on how to keep my files in order?

You're touching the #1 skill of a good software engineer here: Keeping things organized. There are no cut&dried rules here, as it pretty much comes with experience. Picture yourself being a maintenance programmer given the source without further introduction. What would be the most comfortable setup so you could read yourself into the sources with minimum effort?

My setup is: Create subdirectories for clearly seperated stuff. boot/, memory/, scheduling/, that kind of thing. Each directory gets the subdirs src/, inc/, and doc/ (minimum). For a large project (like an OS), you might want to distinguish between unit-internal headers and "exported" headers that other units might want to include.

Add a top-level Makefile. Make sure it handles include dependencies automatically. Do not use recursive make. (Google for "recursive make considered harmful".)

Any source control systems without server overhead (eg: local host only)?

RCS. Should be installed on any self-respecting Linux box. Cygwin has it, too. Commands of note are "ci", "co" and "rcsdiff". Pretty similar to CVS in fact (not surprising since CVS started as a bunch of shell scripts wrapping RCS). Uses local {filename},v files for storing versions, automatically created on first "ci". Can be stored in a "RCS" subdir to get them out of the way. "make" knows that it can get a "*.c" file from a "*.c,v" or "RCS/*.c,v", which makes for rather simple Makefile handling: Fully transparent. Oh, and renaming a source file is as simple as renaming the ,v file. For local-only work it's next to perfect.

Oh, out of curiosity, what would you consider the SIMPLEST file system? I don' care about features. Or anything else. Even 8.3 names.

FAT is very well documented. I'm not sure about the current status on the patents Microsoft tried to enforce recently.

I'd like to note also that the forum seems to be growing, the last time I checked in (always as guest I think....whatever) it took 2 days to get a reply, a day to get a view. So that's cool.

The veterans stayed put, and some of the newbies have grown into giving valuable advice of their own.

veterans *rofl*

as for the question:

I'd keep the bare bones of the kernel in one image. then the modules: you can have extra fs service, extra drivers, extra gui service whatsoever AND in the meanwhile have a reasonable fine microkernel like environment - yu just need to have your bootloader fetch the required modules from disk - or have grub do it.

Then just fetch one module after the other, move them to appropriate locos or render them into processes (microkernel) and voila - if you have indicated the correct drivers - your os can of course access the hd, the fd, the cdrom. You just indicate this in the bood parameters - and by the modules you give the kernel to start up upon init.

Hope this isn't tooo complicated a rant. I'm low on coffee now and have a big hankering for a big cup o java.

Consider taking a look at how the L4 microkernel handles this. I'm writing a core on top of L4Ka::Pistachio and it makes using modules quite easy and natural, without neither L4 nor your OS having to know anything about disk drivers, filesystems and so forth.

For IA-32, take a look at L4's kickstart loader to see how they implemented it. Basically GRUB is given a list modules to load (minimum 4 of them: kickstart, ia32-kernel, sigma0, roottask); it loads and fires up Kickstart, which then reads the Multiboot info, stores it in a platform-independent format in L4's Kernel Information Page (KIP), and hands control over to the L4 kernel (ia32-kernel), which in turn loads sigma0 (the initial pager) and roottask (a privileged OS personality, i.e. your OS), both of which run in user mode.

Once the roottask is up and running, it can query the KIP to find any additional modules such as device drivers or other root tasks which it should initialize and start. The modules are loaded and ready in physical memory at the offsets given in their ELF headers, so it's really quite simple to get them running.

(On other platforms, such as PowerPC, L4 modules are usually stuffed into one big ELF file by a tool called Dite.)

Cjmovie wrote: Designing my os, I have faced a very funny problem: what should be a module, and what should be built in?

If you ask me, I would love a perfect modular system: everything but the memory manager and low-level stuff like interrupts and IRQ's etc. are modules.

But the problem? Very similiar to the chicken and egg problem. For instance, each module is a '.fnc' file (functional module). But what if the floppy driver is a module? How to load it so you can load the others? OK, you say, use bios to get it before switching. But what if the filesystem (say FAT12) is parsed using a module? Your stuck again.

What is the solution to the chicken & egg problem in the very basic problem of a chicken farm? self-sustaining that is.

You buy a bunch of eggs (modules), enough to get you started up, and you go home with enough food for them (memory). Then, you hatch the eggs (unpack the modules you preloaded), raise the chickens (boot 'm up as well) and make them create more eggs (use the modules).

When the system is booted up, you can shut it down with a new supply of eggs so it'll have a different basic configuration next time.


Extending this idea, I figured you might as well store the entire set of modules you need for a running system in the initial batch, only loading others when the user indicates he has added new hardware. That'd give blasting bootup times for your computer and always kind of keep the boot time quick (since all bootup modules are stuffed into one file, you only need two files for booting & loading the kernel). If you're going for top-notch speed, make booting the modules multithreaded, so slow modules don't slow the system down (dhcpcd...).

Hmmm...I like two of the ideas thrown around:

1)An image file that can be loaded as if it was 1, with all required modules in it

2)Only modularizing the changing stuff. It's just that now that I think about it, why make keyboard a module? I can't think of any computer that didn't have one!

Oh, and I can prove scientifically that the egg did indeed come first. (That is, considering I am an athiest that beleives in evolution )

Cjmovie wrote: 2)Only modularizing the changing stuff. It's just that now that I think about it, why make keyboard a module? I can't think of any computer that didn't have one!

Can't you ? what about xbox ? what about IA-32 - based DVD/MP3/ultimatemedia players ?

And, more seriously, what about a system that only has USB keyboard for which you have a driver. You don't still need the "legacy" keyboard driver, do you ?

How much you will modularize stuff will mainly depend on the cost it has at run time (e.g. if once the module is loaded, it just runs as if it was part of the kernel, then you should have nothing prevent you to put most of the things within modules)

I was expecting Pype's answer

Cjmovie wrote: 2)Only modularizing the changing stuff. It's just that now that I think about it, why make keyboard a module? I can't think of any computer that didn't have one!

That is a good idea, and a bad idea in itself again. You can not expect stuff to stay constant, and you can't predict what isn't going to change. Plus, as soon as you run it on more than just your computer, they'll have a different base altogether, and if you don't intend to make the user recompile (not everybody is a nerd with no hotplugging intent) you want them to be able to change it. And that means every bit of it that can physically change should not be in-kernel.

On the other hand, you can optimize for common cases, allowing for awkward cases. Say, combine the ps2 keyboard and mouse driver into one package for speed, since most people with one are very likely to have the other too. Same with usb mouse/keyboard and stuff like that.

Oh, and I can prove scientifically that the egg did indeed come first. (That is, considering I am an athiest that beleives in evolution )

If you can prove which came first, which came first, the compiler or code?

Going seriously OT here, but I found the example somewhere else some time ago, and found it intriguing to think about.

Candy wrote:If you can prove which came first, which came first, the compiler or code?

Code. A compiler is code, so code must have been there first.

Simple example. Consider an otherwise complete C compiler that does not know about \t. It does have something like this though:
So you go there and add:
You add doTab(), and compile this compiler with itself. Then you exchange the [tt]case 9:[/tt] with a [tt]case '\t':[/tt], which the compiler binary you just created does know how to handle... your compiler just learned about \t.

Take this, and think that learning curve backwards to the very first assemblers written by programmers tired of hexing stuff in.

Then think further backwards to when someone fed a machine a couple of punch cards carrying a program that could turn hex into binary.

In the very beginning, there was binary, and people were plugging cables into the wall to make it happen. In a way, their heads was their compiler, turning a problem into a configuration of cables.

Code was first.

Now, for the chicken-and-egg... of course, the Amino Acid was first. The rest was reproduction and evolution. Whether with an ultimate plan of some creator behind it or not is up to you. But what kind of a creator would let Intel CPUs happen?

Ah, but you could argue that Intel CPUs were an initial evolutionary step that allowed AMD CPUs to evolve and replace them.

Of course, just as humans were only here to work on the next stage of life - computers! (2001: A space Odyssey)

As for chicken and egg, same idea as compiler and code. You must go back to the BEGINNING.

As some animal evolved, it became more and more chicken like. Then one of them laid an egg - which was just close enough to be characterized as a chicken when born. So the egg came first.

But on top of that, the egg came before the chicken in a different sense also. Chickens weren't the first to start laying eggs.

Pype.Clicker: Ahh, but another problem. Usually. I happen to have on my computer a USB optical mouse and a standard PS/2 keyboard. So maybe it would be a good idea for:

Each module to have a fail-safe (as in 'wont hurt computer') feature named 'Detect', which will return a boolean true if it can prove the hardware it would control is their, a false if not.

Then all you do is have some hot-plug functions which let the module feed into the OS. Thus, you have a microkernel!

I'm very happy with this idea. Thanks guys! (and girls, IDK)