OSDev.org

HELLO AGAIN MT!!!

I've begun work again on Drake(OS) and am now in the process of decision making, and design. I have several questions that are open for opinions and suggestions.

Some assumptions to be made: I divide things into two sections. On the top is kernel land, where a central module is loaded. All other modules are dynamically linked to it. In userland, you have services, and another "kernel" which all usermode modules connect to.

Some theory: if i have a module for all services (eg. HD, FD, and CD), and for some reason the hard drive service fails, does this bring down the system. The issue is this is not exactly a true microkernel design (according to the OSfaq and several sheets floating around the net; it just dynamically loads modules), so the modules become part of the kernel. If the system can restart the service (eg. reset all of it's values to their inital values, and begin exeacution anew), i can see how to avert disaster, but that is in theory

I also like the idea of being able to remove modules at run-time at any arbitary time. For instance, a TCP/IP module could only be usefull when running web servics (or network in general), and thus is just taking memory and CPU when not used. I read alot about removal being bad practice and all, but if you notified the service/module first, I'd assume the safety to be complete.

So for some questions regarding it all:

• Am I completely wrong with my theories?
• If this is all good and all, any recommendations to make this more of a TRUE microkernel?
• I've been reading on these "exokernels", but I'm "iffy" on their ideas. How much do you believe outside a kernel qualifies as exo?
• I'm very interested in being able to write all this in C++, and one of the major design goals is 100% object oriented interface (in programming AND userland interfaces). Is this feasable/desirable to people besides me?

Merry X-MAS to thoes who celebrate it, happy holidays to thoes who don't! DH.

PS. Sorry if i blabber on about nothing

Hi!

Dragon_Hilord wrote: Some assumptions to be made: I divide things into two sections. On the top is kernel land, where a central module is loaded. All other modules are dynamically linked to it. In userland, you have services, and another "kernel" which all usermode modules connect to.

I didn't really get this - what do you actually mean by 'another kernel in userspace'? Do you mean a userspace portion of code to which all userspace modules link to, and which is present in all address spaces? If so, I'd rather call this some kind of 'native runtime library' for your kernel (such as libc for linux). Please correct me if I got that wrong.

Dragon_Hilord wrote: Some theory: if i have a module for all services (eg. HD, FD, and CD), and for some reason the hard drive service fails, does this bring down the system. The issue is this is not exactly a true microkernel design (according to the OSfaq and several sheets floating around the net; it just dynamically loads modules), so the modules become part of the kernel. If the system can restart the service (eg. reset all of it's values to their inital values, and begin exeacution anew), i can see how to avert disaster, but that is in theory

It depends on _how_ it fails. As soon as you put some code into kernel space, it has full read/write access to all other kernel code & data. So it is possible that a piece of failing code scratches lots of sensetive data before you even notice the problem - and this is exactly what 'true' microkernels try to avoid.

IMO, you can only detect (and recover from) a failure of kernel level code when the involved module is able to report it. So your module must be written with error detection in mind, and as soon as something seems to go wrong, stop execution and report it. Another problem is that other pieces of software which rely on the service must also be written with error-recovery in mind, i.e. they have to notice the failure, wait for the service to be restarted, and then submit their request again.

Dragon_Hilord wrote:

• I'm very interested in being able to write all this in C++, and one of the major design goals is 100% object oriented interface (in programming AND userland interfaces). Is this feasable/desirable to people besides me?

Actually, your project (as far as i can tell from your post) seems to be quite similar to mine: Hybrid kernel, heavily modularized, runtime module loading/unloading support, clean object oriented interfaces wherever applicable...

So yes, I do believe that it is feasible & desirable.

PS. Sorry if i blabber on about nothing

No way. It's always good to read something about new designs.

cheers Joe

Thankyou for the quick reply!

I didn't really get this - what do you actually mean by 'another kernel in userspace'? Do you mean a userspace portion of code to which all userspace modules link to, and which is present in all address spaces? If so, I'd rather call this some kind of 'native runtime library' for your kernel (such as libc for linux). Please correct me if I got that wrong.

I meant to express it as an extension of the kernel in userland. Meaning basically you got the central kernel module occupying both user and kernel land. The idea is that less switches between the two modes are needed (userland jobs that are done by the os are done by the kernel without switching to kernel mode. This also has an added benifet of having a nice security aspect. It is not present in address spaces so I wouldn't call it a runtime. Rather, it is for drivers and such. It's only a thought at the moment.

It depends on _how_ it fails. As soon as you put some code into kernel space, it has full read/write access to all other kernel code & data. So it is possible that a piece of failing code scratches lots of sensetive data before you even notice the problem - and this is exactly what 'true' microkernels try to avoid.

I see your point. Maybe I'll make the kernel design more like a micro for this purpose. The problem is I don't like things like HD suppot in userland: makes me antsy... Either way, I'll try to include LOTs of failsafes (without bogging the computer down ).

Actually, your project (as far as i can tell from your post) seems to be quite similar to mine: Hybrid kernel, heavily modularized, runtime module loading/unloading support, clean object oriented interfaces wherever applicable...

So yes, I do believe that it is feasible & desirable.

At least I'm not alone in this world! I'll try to help you when and if I can . When you put a link to your OS/kernel, I'll check it out .

No way. It's always good to read something about new designs.

Confession: I got the idea of the kernel being in userland and kernel land from a DragonLance book (the dark god of magic {insert name here} wanted total control, so she planned to implant her soul into a mortal, thus binding the world of the dead with the living... or something like that (read War of the Souls!!)). Don't ask me why, but that's the honnest truth!

Cheers, DH.

PS. That series is VERY MUCH worth reading!

If you have a bad HDD driver it will keep crushing after you restart it, so you won't be able to access your HDD and probably you system will hang.

But if you "integrate" driver with kernel, and if driver fails your system will hung instantly. You will get the same resault but the second is much faster (it's because system works faster).

So is it worth doing all that?

Or....
Driver fails. Restart. Driver fails. Second time, notify user. Back down to BIOS interface in V86 mode to allow user to save data, find new driver, and restart.

hmm... too much work to do and it's too hard to do! ;D

The only thing more frightening than a programmer with a screwdriver or a hardware engineer with a program is a user with a pair of wire cutters and the root password.

by Elizabeth Zwicky ;D

Hi,

kataklinger wrote: If you have a bad HDD driver it will keep crushing after you restart it, so you won't be able to access your HDD and probably you system will hang.

But if you "integrate" driver with kernel, and if driver fails your system will hung instantly. You will get the same resault but the second is much faster (it's because system works faster).

I hope you're not being serious here...

Imagine a sound card driver that occasionally trashes a few random bytes of kernel code or data. For a monlithic kernel the OS could keep running, with everything randomly crashing until the whole system dies, and if you get any sort of error message it's unlikely it'll have anything to do with the cause of the problem. In this case you could spend days just trying to find out which piece of code is causing the problem.

For a micro-kernel, as soon as the problem occurs you'd get a page fault, which would hopefully display a report containing the type of fault, the name of the faulty process and a register dump including EIP. After that the OS could continue running without messing up the users work, automatically try to find an updated sound card driver (and/or automatically send a nice descriptive email to the developers).


Cheers,

Brendan

Brendan wrote: I hope you're not being serious here...

Imagine a sound card driver that occasionally trashes a few random bytes of kernel code or data. For a monlithic kernel the OS could keep running, with everything randomly crashing until the whole system dies, and if you get any sort of error message it's unlikely it'll have anything to do with the cause of the problem. In this case you could spend days just trying to find out which piece of code is causing the problem.

For a micro-kernel, as soon as the problem occurs you'd get a page fault, which would hopefully display a report containing the type of fault, the name of the faulty process and a register dump including EIP. After that the OS could continue running without messing up the users work, automatically try to find an updated sound card driver (and/or automatically send a nice descriptive email to the developers).

Implement sound card drivers in user level, and HDD in kernel land for instance. Hybrid architecture I like the most . I don't mind putting something(but not too much) in kernel mode if it means faster execution.

And I'm not sure about sending email or downloading updates automatically. Is it really good idea? :-\

kataklinger wrote: Implement sound card drivers in user level, and HDD in kernel land for instance. Hybrid architecture I like the most . I don't mind putting something(but not too much) in kernel mode if it means faster execution.

IMO this also depends on how you design the system. Let's assume - for now - that filesystems are implemented in userspace. How should it make the system faster when you put the hdd driver in kernel space then? You could as well put the hdd driver and fs service into the same address space (since they will rarely be used without each other) and end up with quite the same performance, especially when your system is based on asynchronous IO (so that you can queue up several IO requests before doing a context switch).

The reason why I'm talking about filesystems is that I'm currently trying to decide upon where to put them - in userspace or kernel space.

cheers Joe

kataklinger wrote: And I'm not sure about sending email or downloading updates automatically. Is it really good idea? :-\

So long as you aren't violating users' privacy or doing it behind their backs, I don't see why not. It saves the user's time and energy from having to be expended on that sort of thing manually (and actually, most probably wouldn't bother to anyway if it weren't automatic). And you would, of course, implement it securely with signatures, encryption, hashes and the like.

Hi,

kataklinger wrote:Implement sound card drivers in user level, and HDD in kernel land for instance. Hybrid architecture I like the most . I don't mind putting something(but not too much) in kernel mode if it means faster execution.

This is a good option - especially for swap space.

kataklinger wrote:And I'm not sure about sending email or downloading updates automatically. Is it really good idea? :-\

It'd depend on how it's all implemented. Automatic updates would be first, but would have to be configurable and secure. The user should be able to disable it and manually request updates on a "per package" basis or based on package type. They should also be able to specify a different server to negotiate updates with.

One thing you'd want to support is getting updates from a local machine, so that for a large LAN one computer can get all updates from the internet and cache them, while all other computers get updates from the local server. I'm also thinking more like Gentoo's "emerge" rather than Microsoft's "Windows Update" - a service that works for all software updates (and installing new software), rather than for system software only or for software from one supplier only.

The server would contain a categorized list of all possible "packages" and, for each package, it's current version number, the location it can be obtained (which may or may not be the same server), a description of it, etc.

Once this is in place, if any software crashes the OS could first check the system settings to see if anything has been disabled, check if there's an update for the package (developers don't want more error reports for bugs they've already fixed) and then look in the executable header to see if the developers do want error reports. If all of these checks are OK, then the OS could automatically create an email with all relevant details (program name, command line options, version, fault type, contents of registers, OS version, etc) ready to send. The email itself would pop up in a window, which allows the user to add extra information or cancel the email.

There's a heap of extra options this system could have, like sending all bug reports to an administrator instead, keeping a log of all problems, always sending the email without user interaction, etc.

For software developers, it allows them to have special email handling software which maintains a database of results. For example, if the sound card driver only ever crashes when it's running on version 123 of the OS, and only when specific command line arguments are used, then having a database would make it easy to find which combination/s trigger the bug.

It also helps to get feedback - often users won't report bugs (too hard, couldn't be bothered), which makes developers think everything works when it doesn't, so the bugs don't get fixed.

This is something I want, but I want to combine it with an additional "notification system" so that it includes hardware failures. The idea here is that (for a LAN for e.g.), the system administrator is sent a notification if any computer has a hardware failure (during boot or after). Combined with a fault tolerant file systems, etc, this could mean that the user may never know about half the problems that occur - a maintence person arrives to replace a dead hard drive or faulty RAM and the user says "Huh? What do you mean it's broken? It's working fine...".


Cheers,

Brendan

I would consider all non-vital (eg. sound, mouse, possibly keyboard, consoles (higher level support), network, and other services that would not be very important to the running of the system) to be userland. The idea of the automatic updates is one that I REALLY like. The problem comes in when you want to patch the kernel, or something along the lines. For instance, a new NTFS driver (:D) has just been released. Many non-geekazoid users want to utalize this, but don't know how to edit the {insert kernelmode module configuration filename} file, so can't install it. How do you go about allowing these people do this without having a hemorage? To me, a simple approach is to lock out all system files (no editing allowed except for by the "uber user") that would be prone to attack. Next, supply a kernelmode driver which takes requests to install a module. It may check a "safe-sheet" (on the internet at a secure location, or even locally), a header (the header may contain information about the driver's type, secutity level, etc.), and then query the user if they are sure. If a module is an upgrade, and the user (par-say) checks a box saying "allow automatic upgrade", then they won't be bothered again. If a checksum fails (could prove vital to have checksums ), the user will be alerted, and the module immeatitly locked out until further notice.

A similar idea should be applied to userland. If userland drivers/services can access hardware, they should be limited to the hardware of their design. This means NO "generalized service packages" (look into Window$ XP, and check out some of it's services; most notibly "svchost.exe").

For both cases, a list of "safe" drivers should be maintained by the developers of the operating system, and driver developers should be required to put forth a unified header with ALL driver information (including error handling procedures!). Generalization should not be over-used, and security of the running system (in my mind) should rate VERY HIGH in the requirements of the kernel.

Le'mme see.... Crashes of kernelmode drivers/services should be tollerably depending on the severity. As for HDD crashes, I'm designing my Bootloader to be able to literally "reload" crashed software. What I mean is, a copy of all modules/services and the kernel core are kept in-memory with the bootloader. If a crash occures, all userland memory is locked, the bootloader reloaded (after making necessary preperations), and the system reboots back into a viable enviornment, with (hopefully) all user software still working enough to recover lost work . It's all just an idea right now, but it is a feature I REALLY WANT to include for obvious reasons (please don't steal it ). This may be all easily said, but there probably would be a considerable load on memory for this to all work .

Cheers, DH.

PS. The bootloader is an _IDEA_ so... it may not materalize as stated

All this sounds good, but it's too good to be true.

And if your network driver fails you won't be able to use automatic update

And you cannot relay on user's opinion about security.
After all you know how it says:

Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning.

;D

;DThat is EXTREMELY true . "Software is as good as the user" and "The user can't be expected to understand what they are using" kind of comes to mind. I would assume there would be a tag line in the box with information from a trusted source, with a starting line of Recommended or else wise.

I don't think the bootloader idea is unfeesable, just difficult. As for the network driver note: the system _should_ automatically restart all failed services based on "advanced" settings (eg. try 3 times, then log a failure, dump some memory, notify user, reboot).

Another consideration after reviewing my own comment; there should be a basic, fast checksum on all loaded modules to allow the bootloader to know if the software is damaged.