A new Mega Tokyo Community OS

[Crazed123]

well, if you want to achieve something like "MT-compliant driver", that also means you have to provide an abstraction of all the features the driver could need: error logging, allocate virtual memory (for its own purpose or for buffers purpose), etc.

You're forgetting the idea that we can't even agree on the language - or the type of language - by far. You'd have to make one driver in OC/aml, Forth, C, C++, Assembly, Java, Pascal and probably a dozen others as well - at the same time! Don't forget that if they function suboptimal people will ignore it. So, stuff they don't use should follow a graceful fade pattern.

I think that such would be factors harder than just writing an OS with all drivers.

Foreign Function Interfaces? Non-default calling conventions? Whatever the OS gets written in, user programs and drivers can find a way to call it.

Pype: That IS true, but we should still work on this. Nonportable drivers are stupid for a bunch of hobbyists and UDI is a small operating system unto itself.

[dh]

Haha. I'd have some support for "MT-Compliant" drivers, but I don't think this'll ever materalize. Not to mention, to be compatable with my kernel, you need security extensions and such. You'd have to include all of the minimal requirements for ALL kernels so it'd work! I'm all for unified drivers, modules and such, but I have this annoying tendency to doubt the ideas behind something so LARGE.

If you get to the point where something WORKS, i'll seriously consider your design, and make my drivers compatable. As Karl Marx once said: "Spread the Wealth" (or something like that ).

One more little thing worth mention: C interface vs. C++. I'm going for C++ because I like OOP . This can make your design VERY difficult to be protable

[Crazed123]

What kind of security extensions? In a model I was thinking of the driver asks for very specific privileges, and is either granted them or not by the kernel. It has no idea why the privileges were given or not, and this (hopefully) allows a decent range of security models.

Request For Comment on these notes in an effort to create a driver standard:

• Resource Allocation
  1.The types of resources are system-abstracted I/O ports (used to represent both hardware I/O ports and memory mapped registers), DMA channels and use, rights to handle a hardware interrupt, and a general stream abstraction. Streams have held up in numerous programs and languages as a good abstraction of input and output, and a good abstraction should be reused when it works well.
  2.Security is provided by having the kernel know about the driver's security status in an Operating System specific way. The driver asks for resources and privileges to them, and is either granted them or not.
  3.Driver death is an event on which all resources held by the driver are revoked and the driver code is terminated. What causes and constitutes driver death or termination of driver code is OS specific, ie: a microkernel can kill the driver's server process but a monolithic one just marks a boolean saying the driver is dead.
  4.A driver should specify on initialization what and how much resources will be needed. In the case of streams this is a maximum amount of data all streams assigned to the driver can contain at once.
  5.When a resource is allocated the driver is given a handle (or capability, or something else like that) and some bits specifying the driver's privileges to that resource. Those privileges are standard for the resource type and stored by the OS kernel for security purposes.
• Extensibility
  1.There should be a way to both query the kernel for and give to the kernel information about extended information and interfaces. The method of exporting a querying should be standard, while extensions should not need to be standard in any way at all. This means that if I extend a widget driver to support feature X of my operating system the extension doesn't need to follow any standard at all except how it notifies the rest of the world of its existence.
  2.Both drivers and the kernel should be able to export and use extensions.
• Information about the operating system
  1.There should be a standard way for a driver to find out such details of its environment as operating system name and version, processor architecture, and any possible information the kernel might have about the piece of hardware the driver drives.
  2.While a standardized method for invoking peculiar processor features like MMX or V8086 mode should not be required, a standard kernel extension for such is a Good Idea. Again, however, it isn't necessary.
• Communication with the Outside World
  1.There should be no standard way for a driver to talk to applications. This will slow things down, but it's absolutely necessary for portability as IPC is implemented differently in nearly every system; and in some systems drivers wouldn't even be processes that could use the IPC model.
  2.If an application wants to use a piece of hardware, it goes through the kernel. If the driver supports an extension to talk directly to applications on that system, the application gets access to that extension from the kernel.
  3.However, there should be a standard way for kernel and driver to invoke, send data to and get data from each other. Possible models for this include, but aren't limited to:
  A.Using the stream abstraction to send data back and forth.
  B.Signals, events, messaging or whatever your system calls it.
  C.Procedure calling in which the driver hands the kernel callback routines to invoke it with.
  D.Reading to or writing from a shared area of memory.

Once again, this is a Request For Comment, but not in the sense of an Internet-wide standard.

[Brendan]

Hi,

Not to mention, to be compatable with my kernel, you need security extensions and such.

This is a good idea - if we all post our OSs requirements for device drivers, then maybe people will realize how impossible a "uniform driver interface" actually is.

The following are the requirements for my OS...

To be suitable for my OS it'd need to do as much as possible asynchroniously, be confined to it's own address space (messaging and bounce buffers used for transferring large amounts of data, rather than using DMA directly into the callers address space) and must not use any form of dynamic linking (flat binary only).

When a device driver is started it's told which I/O ports, etc it may use (they are pre-detected), and the first I/O port is always I/O port 0x0000. For example, a serial port driver would be told to use I/O ports from 0x0000 to 0x0007, so that these I/O ports can be hard-coded regardless of whether the actual device uses a base I/O port of 0x02E8, 0x02F8, 0x03E8, 0x03F8 or something else. When a device driver tries to access an I/O port it actually causes a general protection fault, and the kernel's GPF handler works out if access is allowed, adds the base I/O port, performs the I/O port operation and then returns.

I expect I/O ports to be cached where possible. For example, for a serial port driver the baud rate divisor, interrupt enable register, LCR and MCR always contain the value that was written to them last, so these values can be cached in memory and read from memory instead of reading from the I/O ports. This is for performance only (I/O port access is very slow, even without the GPF stuff on top).

Any memory used by any form of DMA or bus mastering must be allocated using special kernel functions, and normal RAM pages must never be used. This allows the kernel to terminate the device driver at any point without the device driver's prior knowledge. For example, if the device driver crashes in the middle of a DMA transfer, the kernel will put these DMA pages into a "cooling off" area so that they aren't re-used while the hardware is still transferring data into them.

All devices are classified into types, where each type of device has a specific interface to the OS. All interfaces are implemented on top of asynchronious messaging. These interfaces include the storage device interface (disks, scanners, printers), the communications device interface (network cards, serial ports, parallel ports), the keyboard interface, the pointing device interface (mouse, joystick, touch screen), the sound interface and the video interface. There will be a specification for each interface that must be followed.

All device drivers must be able to completely initialize their device from any state. This initialization must not include any ROM code (for e.g. a video driver must not rely on the video card's ROM to pre-initialize the card).

All device drivers must support a "pass through" mode, which allows the raw device to be assigned to a "hyper-visor". This pass through mode includes the ability to return status information (text strings) back to the hyper-visor for debugging purposes, including full details of the entire device's state, and a short note on every operation.

Device drivers for some devices must maintain their own internal "operations queue", which is sorted according to the priority given to the operation by the caller. For example, imagine a hard disk driver with one partition being used for swap space and another partition being used to store files. Now consider what happens when the user asks to save a very large file, but just after this the swap space manager asks to transfer a page of data to/from swap space. In this case I'd want the swap space operation to be have a higher priority and be completed as soon as possible (with lower priority operations being delayed).

Device drivers may be written in any language, as long as the binary format used can be generated and as long as that language supports the kernel API. For example, it's not too hard to get LD/GCC to generate the binary header, etc but you may need to rewrite library functions before you can use it to write device drivers, etc properly. My OS is not POSIX compliant in any way (and never will be with any luck).


Cheers,

Brendan

[Phugoid]

In that spirit:

To be compatible with my OS (but not bare kernel), the driver is expected to be in an ELF shared library format. There is dynamic linking.

Drivers have many mechanisms available to them to get their jobs done. They may use port I/O directly, they have access to memory-mapped devices, to asynchronous message channels, to shared memory, and to named routines in other code images through linking. They need only notify the kernel of the resources they intend to use (at which point, of course, the kernel can deny them access).

In fact, all trusted code, which should include drivers, is loaded and run directly in supervisor mode. However, such code must meet fairly stringent requirements. For example, a driver, in fact any program, must prove that it does not access any ports that lie outside of port ranges for which the kernel has granted it access. This very likely precludes the use of ROM code for drivers in this hypothetical OS, else a driver must automatically generate a safety proof for whatever ROM code may be present in a system, as the ROM cannot be expected to provide one.

Drivers may register an interrupt handler for a hardware interrupt. When the interrupt is generated, the handler routine is called on a dedicated, but small stack, in the context of a dedicated task. The driver must prove that the immediate handler requires a certain small and finite amount of stack space. It is expected that the driver will use kernel-provided synchronization and communication routines to convey information about the interrupt to the rest of the system.

Drivers may reserve memory ranges for I/O. These include kernel-allocated DMA ranges in low memory, handled separately from other physical memory, and memory-mapped devices mapped outside of "real" physical memory, also handled separately. The driver must prove that all memory accesses lie within these regions, as well as those allocated to it for internal use.

The internals of driver design are not of high concern, so long as the drivers are correct and have reasonable performance (i.e., the OS doesn't care, even though these things are highly desirable). Drivers have multithreading available to them.

Drivers may be written in any language so long as a proof of their safety is distributed in the resulting binary. This entails the development of new compilers. Since C is not an appealing language for writing new compilers, writing a driver in C will mean generating a proof for the resulting object code by hand.

Because of the high probability of failure in this project, I have written (and am rewriting) a very simple kernel, so that I may have something useful even if I quit. This kernel is so permissive that I could adapt most Linux drivers to run under it with relative ease, when using the kernel without additional layers of code.

[Candy]

This is a good idea - if we all post our OSs requirements for device drivers, then maybe people will realize how impossible a "uniform driver interface" actually is.

The following are the requirements for my OS...

Ok... let's list them:


- Be multithread capable, IE, it must be able to deal with being run twice at once - even the init code - as well as being interrupted at times and being run concurrently with other stuff.
- Have a probe function that returns true if and only if the driver actually supports the device, and that can run in very short time.
- Stored in proprietary format for which I'm going to make a converter from unlinked ELF object file


well... actually, it's pretty normal imo. Just that I kind of require them to be binary objects that can be loaded and unloaded at any time at random... Function requirements:

Code: Select all

struct static_functions {
  bool probe(object *obj);
  void init();
  void fini();
  object *construct(object *obj);
};

struct functions {
  void destruct();
  <object type specific functions>
};

There'll be types of objects for the object specific functions.

[Crazed123]

Getting this vibe of "It's a bad idea and we don't like it." Try something else? Instead of a driver interface, why not just put a driver section in the Wiki? If you code a driver for something you post how it works into the Wiki, and other people can work from that.

Actually, standard device interfacing isn't that hard to do. It's just that everyone seems to want their OS's device drivers to have some special feature other than driving devices which would break compatibility with a standard and couldn't be implemented as an extension.

[Brendan]

Hi,

Getting this vibe of "It's a bad idea and we don't like it."

Device drivers themselves form an abstraction layer between hardware and the OS. Given that the hardware can vary greatly and OSs can have large fundamental differences between each other, a truely universal device driver interface would be incredibly difficult (which IMHO explains why a project with good commercial backing couldn't do it, despite having some success with several flavours of UNIX).

Try something else?

There are plenty of people who think I'm a fool for seriously attempting to write a modern OS, but there were probably plenty of people who thought attempting to send people to the moon or climbing Mount Everest was foolish or impossible. I tend to listen to these people, nod my head like I agree and then ignore them.

If you are enthusiastic enough, confident enough and willing to spend a large amount of your time in the hope that one day it works, then don't let the opinion of a few people stop you.

Also (IMHO) you could improve your chances by limiting the scope - for e.g. a common driver interface for a certain type of OS rather than something that is intended for any OS that could be concieved.


Cheers,

Brendan

[Crazed123]

If you are enthusiastic enough, confident enough and willing to spend a large amount of your time in the hope that one day it works, then don't let the opinion of a few people stop you.

Also (IMHO) you could improve your chances by limiting the scope - for e.g. a common driver interface for a certain type of OS rather than something that is intended for any OS that could be concieved.


Cheers,

Brendan

A uniform device driver interface is of no help if nobody uses it but me.

The point of what I've been trying to create is that if a driver needs support for a special extension to work on a certain OS the developer for that OS can add such support. The operating system can even refuse to work without that extension! The point is that the driver will still use the interface for the parts that ARE standard, and so will be all the more portable for it.

So how about this? According to the standard the operating system can query a driver for supported extensions even before it inits, and the driver can query for extensions on init. If either doesn't find what it wants it can flop and die. The point is to standardize the hardware access parts that really are common.

[Phugoid]

If drivers are allowed to die, they are pretty useless pieces of software. Why would anyone include in their OS drivers that die on initialization? Developers would instead query the English description of the driver and realize that it is utterly incompatible with the way their system works. There is no point in putting this off to run-time.

I think more work should be put instead into describing in English how modern hardware works. English and devices are the two things common across all operating systems and developers.

[Brendan]

Hi,

I'd find good written descriptions with low level code snippets very useful. For example, for serial ports it'd involve an overview of how it works, a list of I/O ports and what each of them does and then code snippets (written in C perhaps). The code snippets would include routines to:

• - detect the serial port
  - configure the serial port (baud rate, number of start/data/stop bits, etc)
  - send a byte
  - receive a byte
  - anything else

This would allow people to "cut & paste" the code into their own device drivers, so that they only need to write the interface between these low level routines and the OS.

I guess what I'm suggesting is to provide the hardware specific parts without the OS specific parts...


Cheers,

Brendan

[Crazed123]

Now that is a Good Idea.

[viral]

Hi...
I guess now its been more than a week. Now its time to build some real codes.

I know people are not ready for this.. But as Brenden had said it doesnt matter if there are 2 or 20 of them working on it. I know if this project grow (i wish it do...) more people will take interest in it. Its very difficult when you have nothing to show them.... once there will be some good stuff, we'll get more volunteers...

[Brendan]

Hi,

I guess now its been more than a week. Now its time to build some real codes.

I know people are not ready for this.. But as Brenden had said it doesnt matter if there are 2 or 20 of them working on it. I know if this project grow (i wish it do...) more people will take interest in it. Its very difficult when you have nothing to show them.... once there will be some good stuff, we'll get more volunteers...

As my proposal was the only one submitted there isn't much point voting on which proposal to select, which also makes it's hard to tell how many people are interested in my proposal and how many want something else.

Perhaps the easiest way to continue would be for everyone who is interested in my proposal to send me an email containing a bit of a description of yourself, the types of things you'd like to work on, if you'd prefer a small quick job or something that'll take ages, etc.

My email address is "btrotter[AT]gmail.com" (just replace the "[AT]" first )...


Thanks,

Brendan

[dh]

I look forward to seeing something worth-while coming out of this. If it doesn't, nobody will really care, because they've seen it happen before. Elsewise (which I _AM_ hoping for ) you've got yourself something to be proud of, and I (as I said before numerous times) will personally see to its spread . I just hope (based on some recent posts) that your not doing another OSkit, because that stuff is nothing short of worthless. Sorry to people who use it, but you gotta admit, it IS pretty much dead (2002 last news post).

Cheers, DH.