OSDev.org

In its supposedly infinite futility, a solid draft of EDI has been made. It comes with headers in C, documentation generated by doxygen, and an example driver for outputting text into video memory. Please note that I haven't been able to bug-check the example driver, nor does it contain appropriate error-checking. It's a demo, not the real deal.

Basically, EDI is my effort to create a unified device driver API that could be implemented by multiple operating systems. As such, I've tried to place emphasis on standardizing only an API, which could be implemented any which way the OS developer pleases. Extension facilities are also provided.

I know most of you "big-time guys" here (Pype.Clicker, Solar, et al) don't really believe this can work, but please read it through and at least give me the opportunity to receive criticisms and try to correct them (another round, the major version is 3 for a reason).

The big issue remaining with EDI, as the transition from 3.1 to 3.2 shows, is communication between the driver and kernel. I have two main ideas:

[tt]
1.The driver communicates with the kernel by sending messages across a stream. A preliminary design (most likely a bad one, but wtf) for this is included in EDI 3.1. A facility would be provided for the driver and kernel to "handshake", agreeing on a protocol to communicate in henceforth. Several such protocols would be standardized, to support various device types such as disk drives, I/O devices, and networking hardware, along with such basic abstractions as block and character devices. The big advantage here is how perfectly this approach maps onto microkernel architectures, and the synchronicity provided by making the driver run a message loop.

2.The driver communicates with the kernel by representing its functionality as an EDI class. Once this class was registered with the kernel and an instance created and initialized, the kernel could call the class's methods to obtain driver services. This would all take place in an init() function exported from the driver. When the kernel was finished, and wished to kill the driver, it could call a finit() function which will shut down EDI and destroy the objec the kernel was using to communicate with the driver.[/tt]

I've put a lot of work into EDI, and have incorporated other people's suggestions. PLEASE, give it a look and a shot before telling me the task is impossible. Maybe, just maybe, we can make some progress on the duplication and non-portability of device drivers.

Hi,

I've been evaluating the suitability of EDI for my OS - the following is a list of problems. The issues below may seem harsh (but are intended to be constructive), I may have misunderstood some things, I may have described some of the issues badly, etc.

First, my kernel doesn't know much about device drivers - they are mostly just normal processes (like applications). This actually doesn't matter that much, but does mean that an "EDI device driver" will need a wrapper.


bool8(* disable_scheduling)(object_pointer interrupt)

For my OS, there is no way for any process to start or stop the scheduler for threads within the process (or globally). There is also no reason for any code (including device drivers) to need this - interrupts are delivered as normal messages, and messages only arrive when requested. Could it be replaced with generic re-entrancy locking (e.g. semaphores)?


int32(* get_register)(object_pointer interrupt, edi_string_t reg_name, intreg *reg)

What is a "register"? Is it one of the CPU's general registers, or one of the CPU's control registers (CR0, CR4), or is it something more generic (a PCI configuration space register, MSRs, performance monitoring counters, etc)?

If it is for CPU registers, does it include FPU/MMX/SSE registers, how can a device driver be portable (same source code for 80x86 and Itanium for e.g.) and why does the driver need this?


void(* interrupt_return)(object_pointer interrupt)

This function needs to return a status code back to the kernel. For example, if it's a PCI device where several devices share the same IRQ line then you don't want to send an EOI if the device wasn't responsible for the IRQ, and you don't want to try other device drivers if the device was responsible for the IRQ.


int32(* init_mmapped_device)(object_pointer device, data_pointer start_physical_address, unsigned int32 pages)

For me, this function is backwards - the device manager tells each device driver where each range is in the physical address space (the device driver doesn't tell the kernel where each range is in the physical address space).

It also needs a "type" parameter. Saying that all memory mappings must be marked uncachable is bad for performance. For example, video display memory should be write-combining and a PCI card's ROM would be set to "write-protected" caching. You might also want a specific kernel function to flush any cached data.

Also there should be a "physical address space range ID" which is the kernel's reference number for the area. The device driver should tell the kernel what the "physical address space range ID" is when it uses the "map_in_device()" function.


int32(* map_in_device)(object_pointer device, data_pointer *out_mapped)

See notes above. In addition there should be a page count and an offset, so that if the device driver only wants to map part of the region it can. This is partly due to the way address decoding works on PCI buses, where each address range must be a power of 2. For e.g. a video card might have 3 MB of display memory but the PCI device (and therefore the kernel) would report this as the next highest power of 2 (or 4 MB).


I/O Ports

The EDI specification doesn't say how the device driver finds out which I/O ports the device is configured to use. This isn't so good as the I/O ports that should be used by the driver could be set to anything during PCI initialization.

For me, all device drivers use I/O ports beginning with I/O port 0x0000. For example, if the hardware itself is configured to use I/O ports 0x0300 to 0x0307 and I/O ports 0x0480 to 0x0490, then the device manager will tell the device driver that it is allowed to access I/O ports 0x0000 to 0x0007 and I/O ports 0x0180 to 0x0190. When the device driver tries to access an I/O port (e.g. 0x004) it causes a general protection fault and the general protection fault handler adds the "I/O range base" to find the actual I/O port, and then checks to see if the process has permission to access the I/O port. If the process does have permission to access the I/O port then the instruction that caused the general protection fault is emulated within the general protection fault handler (as if the instruction was actually executed). If the device driver doesn't have permission then it's process is terminated.

Also there doesn't seem to be any support for string instructions - "rep insd", "rep outsw", etc. These are used by some devices (e.g. PIO transfers for hard drives that don't support bus mastering/UDMA).

[Continued]

[Continued]

Streams

I think streams can operate between the EDI driver and the wrapper (not too sure).


Advertised Objects

I'm not sure what this is for!


int32 run ( unsigned int32 thread_id )

Isn't this normally called "yield"?

int32 sleep ( void )

This isn't a good name either - there's a standard "sleep(time)" function which makes the thread block for the specified amount of time.

int32 spawn_thread ( void )

I think this is normally called "fork()". My OS doesn't support it (to be honest, I think it's fairly dumb having to do "if(pid == Woot) then this is the new thread; else this is the old thread"). It also can't be done on my OS because local variables on the old thread's stack can't be accessed by the new thread. Instead I use something like:

spawnThread( void *address_for_new_thread_to_begin_executing, unsigned char thread_policy, unsigned char thread_priority, char *thread_name )


int32 wait_on_thread   ( unsigned int32 thread_id, int32 *status)

For my OS, zero is a valid thread ID. Could this be changed to "if thread_id is 0xFFFFFFFF, the calling thread is blocked until any of its children exit"? Theoretically, for my OS 0xFFFFFFFF is also a valid thread ID (but it's incredibly unlikely that any implementation of my OS will actually support that many threads in practice).


Notes

In general, wouldn't it be better to say that the OS must support some of the POSIX "pthreads" functions (and specify which ones - spawning and terminating threads, and the sychronization stuff only)? The POSIX functions are more likely to be supported anyway, and it'd solve some of my "spawn_thread" issues.. For reference, see something like this:

http://www.llnl.gov/computing/tutorials/pthreads/

I'd suggest the following POSIX functions:

• pthread_create
  pthread_exit
  pthread_self
  pthread_equal
  pthread_yield
  pthread_attr_getschedparam
  pthread_attr_setschedparam
  pthread_attr_getschedpolicy
  pthread_attr_setschedpolicy
  pthread_attr_setinheritsched
  pthread_mutex_init
  pthread_mutex_destroy
  pthread_mutex_lock
  pthread_mutex_unlock
  pthread_mutex_trylock

Also, for my OS, when a device driver completes it's initialization it tells the device manager that it's ready for operation. The device manager works out what should happen depending on the device's type. For example, for a storage device driver the device manager would check what sort of partitions are present and tell file system code about the partitions (possibly starting new "file system" processes, and possibly telling the virtual memory manager about any new swap space). For video, sound, keyboard, mouse, joystick the device manager will either tell an existing "user interface" process about the new device or start a new "user interface" process.

In any case, the process sends an "acknowledge" message back to the device manager and the device manager tells the device driver who to talk to. From then on there's a protocol (depending on the type of device) that is used by the device driver to talk to whatever is connected to it. The device manager normally has nothing to do with the device driver after this, but it does monitor the device driver to see if it crashes (and restarts the device driver if it does crash).


Cheers,

Brendan

Greetings Brendan,

Brendan wrote:
When the device driver tries to access an I/O port (e.g. 0x004) it causes a general protection fault and the general protection fault handler adds the "I/O range base" to find the actual I/O port, and then checks to see if the process has permission to access the I/O port. If the process does have permission to access the I/O port then the instruction that caused the general protection fault is emulated within the general protection fault handler (as if the instruction was actually executed). If the device driver doesn't have permission then it's process is terminated.

I would personally try to avoid having to result to an exception and making the GPF handler more messy, for this situation I would create some set of device driver APIs, such as DeviceOutputB() which will do the checks in the kernel tables for rights. But using a port address relative with the I/O base, rather then having the device driver finding out where exactly the I/O ports is a good idea.

Hi,

Ryu wrote:I would personally try to avoid having to result to an exception and making the GPF handler more messy, for this situation I would create some set of device driver APIs, such as DeviceOutputB() which will do the checks in the kernel tables for rights. But using a port address relative with the I/O base, rather then having the device driver finding out where exactly the I/O ports is a good idea.

I like being able to use I/O ports with I/O port instructions - especially for slow devices (like keyboard, etc). The overhead of finding out what caused the GPF isn't much compared to reading or writing to the I/O port or the CPL=3 <-> CPL=0 switching.

Of course I'll also have functions for I/O port access in the kernel API, but this is more for "batch functions". The idea here is that if you need to use several kernel API functions (and if they are reasonably independant) you can build a list and enter the kernel once. The kernel then does all of the functions that are in the list and returns to user mode. After that you'd check all of the return parameters (which would've been put in the list by the kernel).


Cheers,

Brendan

Alright, I'm responding to what I can answer right now. If I don't answer a concern, it means I need to come up with a useful answer.

Brendan wrote: Hi,

I've been evaluating the suitability of EDI for my OS - the following is a list of problems. The issues below may seem harsh (but are intended to be constructive), I may have misunderstood some things, I may have described some of the issues badly, etc.

I assure you all criticism is taken as constructive.

First, my kernel doesn't know much about device drivers - they are mostly just normal processes (like applications). This actually doesn't matter that much, but does mean that an "EDI device driver" will need a wrapper.

This is one of those moments when the driver interface could be far more elegant if it were kernel-specific. However, I'd say that the "message-passing" stream communication method is far more conducive to being simply fowarded somewhere else by the kernel, thus allowing for a very small kernel-level wrapper. I'll keep that in mind.

bool8(* disable_scheduling)(object_pointer interrupt)

For my OS, there is no way for any process to start or stop the scheduler for threads within the process (or globally). There is also no reason for any code (including device drivers) to need this - interrupts are delivered as normal messages, and messages only arrive when requested. Could it be replaced with generic re-entrancy locking (e.g. semaphores)?

disable_scheduling() and enable_scheduling() are designed for time-essential portions of a driver that *cannot* be disturbed. Example: servicing of a clock interrupt, a network card, or a heavy-load graphics card. These things sometimes absolutely require delay-less service. If I'm wrong, of course, then disabling and enabling scheduling could be turned into an extension rather than standardized (mandatory) behavior.

int32(* get_register)(object_pointer interrupt, edi_string_t reg_name, intreg *reg)

What is a "register"? Is it one of the CPU's general registers, or one of the CPU's control registers (CR0, CR4), or is it something more generic (a PCI configuration space register, MSRs, performance monitoring counters, etc)?

If it is for CPU registers, does it include FPU/MMX/SSE registers, how can a device driver be portable (same source code for 80x86 and Itanium for e.g.) and why does the driver need this?

get_register is for CPU registers. Whether or not FPU/MMX/SSE registers are supported here is determined by the implementation (right now), but the method is meant basically for the accessing the CPU's control registers. And I'm beginning to think this might be another one to make an extension.

Frankly? I don't really expect to be able to write a single device driver interface that will work across multiple, radically different processor platforms. I can hope and pray EDI could be made processor-portable, but I feel I have no right to expect so.

Continued...

Continued...

void(* interrupt_return)(object_pointer interrupt)

This function needs to return a status code back to the kernel. For example, if it's a PCI device where several devices share the same IRQ line then you don't want to send an EOI if the device wasn't responsible for the IRQ, and you don't want to try other device drivers if the device was responsible for the IRQ.

Passing a status code there is a Good Idea.

int32(* init_mmapped_device)(object_pointer device, data_pointer start_physical_address, unsigned int32 pages)

For me, this function is backwards - the device manager tells each device driver where each range is in the physical address space (the device driver doesn't tell the kernel where each range is in the physical address space).

It also needs a "type" parameter. Saying that all memory mappings must be marked uncachable is bad for performance. For example, video display memory should be write-combining and a PCI card's ROM would be set to "write-protected" caching. You might also want a specific kernel function to flush any cached data.

Also there should be a "physical address space range ID" which is the kernel's reference number for the area. The device driver should tell the kernel what the "physical address space range ID" is when it uses the "map_in_device()" function.

First of all, the "physical address space range ID" would probably be an instance variable of EDI-MMAPPED-DEVICE. Thus, the driver would implicitly pass it to map_in_device().

The type variable is probably a Good Idea, it just needs to be created correctly. What kinds of mappings would be allowed (I defer to your superior knowledge on this)?

As to the device manager: I can't guarantee that everyone will have a device manager. I can, however, see the need for a driver to be told its hardware parameters rather than knowing them. The question is: What kind of interface allows the driver to tell the kernel (which may pass this on to a device manager) its "name" and device type, and receive from the kernel its hardware parameters? A worthy question indeed.

int32(* map_in_device)(object_pointer device, data_pointer *out_mapped)

See notes above. In addition there should be a page count and an offset, so that if the device driver only wants to map part of the region it can. This is partly due to the way address decoding works on PCI buses, where each address range must be a power of 2. For e.g. a video card might have 3 MB of display memory but the PCI device (and therefore the kernel) would report this as the next highest power of 2 (or 4 MB).

You want to map in only part of the full device? Fair enough, and quite simple to add.

I/O Ports

The EDI specification doesn't say how the device driver finds out which I/O ports the device is configured to use. This isn't so good as the I/O ports that should be used by the driver could be set to anything during PCI initialization.

For me, all device drivers use I/O ports beginning with I/O port 0x0000. For example, if the hardware itself is configured to use I/O ports 0x0300 to 0x0307 and I/O ports 0x0480 to 0x0490, then the device manager will tell the device driver that it is allowed to access I/O ports 0x0000 to 0x0007 and I/O ports 0x0180 to 0x0190. When the device driver tries to access an I/O port (e.g. 0x004) it causes a general protection fault and the general protection fault handler adds the "I/O range base" to find the actual I/O port, and then checks to see if the process has permission to access the I/O port. If the process does have permission to access the I/O port then the instruction that caused the general protection fault is emulated within the general protection fault handler (as if the instruction was actually executed). If the device driver doesn't have permission then it's process is terminated.

Also there doesn't seem to be any support for string instructions - "rep insd", "rep outsw", etc. These are used by some devices (e.g. PIO transfers for hard drives that don't support bus mastering/UDMA).

[Continued]

If coding single-byte (or double, or quadruple) methods into a loop won't suffice, string methods can be added.

As to your I/O port segmentation (it walks and talks like segmentation, but exists in I/O space), it sounds like a sound basis for implementing the I/O port methods.

About the "I/O base" and finding out I/O ports: See above about hardware parameters. This shows the major need for a method by which the Outside World can tell a driver its hardware parameters.

Brendan wrote: [Continued]

Streams

I think streams can operate between the EDI driver and the wrapper (not too sure).

Like I said above, I think streams are the "microkernel" kind of communication: They're easy to forward to somebody other than the kernel.

However, direct method calling on the driver affords the advantage of not having a messaging thread, letting the driver behave as a kind of thread-safe (since driver class's instance variables can be used instead of globals) library. If the initialization interface allowed the driver to init(), then just let the kernel create objects to communicate with, a single driver could drive multiple devices. If an OS wanted multiple driver instances to drive each device, it could do that transparently.

Advertised Objects

I'm not sure what this is for!

EDI makes the assumption that the threads of a driver run in the same address space (since microkernels mostly have threads for their driver processes, and all kernels have only one kernel-space). Thus, rather than constrain the driver's threads to a message-passing interface, the threads can "advertise" any object at all they wish to communicate with. I intend this to make driver writing easier, because there would be no need to write wrappers around message-passing primitives.

int32 run ( unsigned int32 thread_id )

Isn't this normally called "yield"?

Doesn't yield() usually run the scheduler? run() is intended for when you've just handled an interrupt and need to pass control immediately to some other thread for processing.

int32 sleep ( void )

This isn't a good name either - there's a standard "sleep(time)" function which makes the thread block for the specified amount of time.

It can be renamed "block()" with ease.

int32 spawn_thread ( void )

I think this is normally called "fork()". My OS doesn't support it (to be honest, I think it's fairly dumb having to do "if(pid == Woot) then this is the new thread; else this is the old thread"). It also can't be done on my OS because local variables on the old thread's stack can't be accessed by the new thread. Instead I use something like:

spawnThread( void *address_for_new_thread_to_begin_executing, unsigned char thread_policy, unsigned char thread_priority, char *thread_name )

I initially named it fork(), and somebody said that would lead readers to expect it to duplicate code and data segments. How about clone(), named for a Linux system call which only creates a new thread?

int32 wait_on_thread ( unsigned int32 thread_id, int32 *status)

For my OS, zero is a valid thread ID. Could this be changed to "if thread_id is 0xFFFFFFFF, the calling thread is blocked until any of its children exit"? Theoretically, for my OS 0xFFFFFFFF is also a valid thread ID (but it's incredibly unlikely that any implementation of my OS will actually support that many threads in practice).

That makes sense, and doesn't require me to change the range of valid thread IDs to positive signed integers!

Notes

In general, wouldn't it be better to say that the OS must support some of the POSIX "pthreads" functions (and specify which ones - spawning and terminating threads, and the sychronization stuff only)? The POSIX functions are more likely to be supported anyway, and it'd solve some of my "spawn_thread" issues.. For reference, see something like this:

http://www.llnl.gov/computing/tutorials/pthreads/

I'd suggest the following POSIX functions:

• pthread_create
  pthread_exit
  pthread_self
  pthread_equal
  pthread_yield
  pthread_attr_getschedparam
  pthread_attr_setschedparam
  pthread_attr_getschedpolicy
  pthread_attr_setschedpolicy
  pthread_attr_setinheritsched
  pthread_mutex_init
  pthread_mutex_destroy
  pthread_mutex_lock
  pthread_mutex_unlock
  pthread_mutex_trylock

I'd like to note (rather late on, I know) that EDI does already have standard semaphore and mutex classes. I think if the POSIX functions are a good API, if they're elegant and simple, then EDI should use them rather than reinventing the wheel.

Also, for my OS, when a device driver completes it's initialization it tells the device manager that it's ready for operation. The device manager works out what should happen depending on the device's type. For example, for a storage device driver the device manager would check what sort of partitions are present and tell file system code about the partitions (possibly starting new "file system" processes, and possibly telling the virtual memory manager about any new swap space). For video, sound, keyboard, mouse, joystick the device manager will either tell an existing "user interface" process about the new device or start a new "user interface" process.

In any case, the process sends an "acknowledge" message back to the device manager and the device manager tells the device driver who to talk to. From then on there's a protocol (depending on the type of device) that is used by the device driver to talk to whatever is connected to it. The device manager normally has nothing to do with the device driver after this, but it does monitor the device driver to see if it crashes (and restarts the device driver if it does crash).


Cheers,

Brendan

In EDI the ready_to_communicate() method tells the kernel (or whoever, as I've stated the kernel can always act as a proxy for someone else. Implementation details of the kernel are not the driver's business.) that the given object can now be used to communicate. In short, it tells the kernel the driver is ready for full operation.

Here comes the language lawyer...

In edi_objects.h, you write (comments taken out for brevity):
Please consider using the standard:
I know this will require you to do a global search-and-replace on your EDI code, but please don't make C99 coders having to handle two naming conventions where the standard does just nicely.

Hi,

Ok, everything I chopped I agree with, I re-arranged some of it to make things clearer, and I haven't slept(!)

Crazed123 wrote:

bool8(* disable_scheduling)(object_pointer interrupt)

For my OS, there is no way for any process to start or stop the scheduler for threads within the process (or globally). There is also no reason for any code (including device drivers) to need this - interrupts are delivered as normal messages, and messages only arrive when requested. Could it be replaced with generic re-entrancy locking (e.g. semaphores)?

disable_scheduling() and enable_scheduling() are designed for time-essential portions of a driver that *cannot* be disturbed. Example: servicing of a clock interrupt, a network card, or a heavy-load graphics card. These things sometimes absolutely require delay-less service. If I'm wrong, of course, then disabling and enabling scheduling could be turned into an extension rather than standardized (mandatory) behavior.

For this, I use scheduler priorities - an extremely high priority thread is never pre-empted (unless there's an even higher priority thread). A timing sensitive device would have a very high priority thread for low level hardware control (or "upper half") which handles "IRQ messages" from the kernel and "requests" from a "lower half". The "lower half" would be a normal high priority thread that does the interaction with the rest of the OS and sends the requests to the "upper half" (possibly after re-order things for better efficiency, checking parameters, etc).

Also, I haven't actually come across any hardware that needs uninterrupted CPU time (just hardware that performs better if it can get uninterrupted CPU time, and hardware that might drop packets if it doesn't get enough CPU time), but I haven't done device drivers for every possible device either...

Crazed123 wrote:

int32(* get_register)(object_pointer interrupt, edi_string_t reg_name, intreg *reg)

What is a "register"? Is it one of the CPU's general registers, or one of the CPU's control registers (CR0, CR4), or is it something more generic (a PCI configuration space register, MSRs, performance monitoring counters, etc)?

If it is for CPU registers, does it include FPU/MMX/SSE registers, how can a device driver be portable (same source code for 80x86 and Itanium for e.g.) and why does the driver need this?

get_register is for CPU registers. Whether or not FPU/MMX/SSE registers are supported here is determined by the implementation (right now), but the method is meant basically for the accessing the CPU's control registers. And I'm beginning to think this might be another one to make an extension.

For my OS device drivers are normal processes, which means the same 32 bit processes/driver binary could be run on a 64 bit version of the OS. In this case CR3 would be a 64 bit register (the driver would expect a 32 bit register) and the flags used for paging (PAE, PSE) features in CR0 and CR4 might be misleading.

Crazed123 wrote:Frankly? I don't really expect to be able to write a single device driver interface that will work across multiple, radically different processor platforms. I can hope and pray EDI could be made processor-portable, but I feel I have no right to expect so.

I don't know much about other architectures, but the PCI standard (and PCI devices) are cross-platform. It shouldn't be too hard to make portable PCI devices. Some hardware is tricky though (legacy ISA stuff). I'm not sure how other architectures handle I/O port ranges though (e.g. the chipset could memory mapped them into a special memory area or something).

You could try to make EDI work with "similar enough" architectures (without worrying about the radically different architectures)...

Crazed123 wrote:

int32(* init_mmapped_device)(object_pointer device, data_pointer start_physical_address, unsigned int32 pages)

The type variable is probably a Good Idea, it just needs to be created correctly. What kinds of mappings would be allowed (I defer to your superior knowledge on this)?

The PCI bus handles 3 different types of memory rangers (ignoring I/O port space). There's ROMs, "prefetchable memory" and "non-prefetchable memory". ROM would be mapped as the "write-protected" caching type (which allows reads to be cached while writes flush cache lines on all CPUs). The "non-prefetchable memory" would be mapped as "strong uncachable" caching type, and is used when a read can have side-effects (e.g. reading a from a location might cause the device to do <something>).

[continued]

[continued]

The "non-prefetchable memory" would be mapped as either the normal "uncachable" caching type, the "write-combining" caching type, or the "write-through" caching type, depending on what the memory type is and what the CPU/s support. In this case the device driver should know which type is the most suitable, and should ask the kernel for the best possible caching type. If the CPU/s don't support this caching type the kernel can silently downgrade it (e.g. if write-combining is asked for the kernel could map it as uncachable if write-combining isn't supported).

For the write-combining memeory type, there also needs to be a "flush the write-combining buffers" function, so that (for e.g.) after the video driver sends something to display memory it can explicitly tell the CPU to make sure it's actually written.

Crazed123 wrote:

I/O Ports

Also there doesn't seem to be any support for string instructions - "rep insd", "rep outsw", etc. These are used by some devices (e.g. PIO transfers for hard drives that don't support bus mastering/UDMA).

If coding single-byte (or double, or quadruple) methods into a loop won't suffice, string methods can be added.

For me, doing single in/out instructions in a loop would involve constant CPL=3 <-> CPL=0 switches...

Crazed123 wrote:As to your I/O port segmentation (it walks and talks like segmentation, but exists in I/O space), it sounds like a sound basis for implementing the I/O port methods.

I like the "I/O port segmentation", but other people might not - e.g. monolithic kernels where "mov dx,0x0000; add dx,IO_BASE; in al,dx" isn't quite as clean as "in al,0x60"...

Crazed123 wrote:

int32 run ( unsigned int32 thread_id )

Isn't this normally called "yield"?

Doesn't yield() usually run the scheduler? run() is intended for when you've just handled an interrupt and need to pass control immediately to some other thread for processing.

Ahh, you're right - for "pthreads_yield" all it does is end the current time-slice (which would be funny for my OS, where a high priority thread would still be the highest priority thread that can run, and would just get the CPU back again). You're after something that switches to a specific thread (which POSIX doesn't support). For my OS I wasn't planning either of these (but I am planning on building something like it into the semaphore/mutex code, so that a thread's remaining time-slice can be "donated" to who-ever currently holds the semaphore/mutex to prevent priority inversion).

Anyway I guess what I'm saying is that "run" has too many meanings. To me it sounds like something you'd use to start a new process (e.g. "run myapp foo.txt" just like Commodore64 basic). How about "gotoThread()"...

It's probably also a good idea to prefix everything with "EDI_". I know C++ has class names which mostly do the same thing, but most people don't use C++ for low level OS development...

[continued]

[continued]

Crazed123 wrote:

int32 spawn_thread ( void )

I think this is normally called "fork()". My OS doesn't support it (to be honest, I think it's fairly dumb having to do "if(pid == Woot) then this is the new thread; else this is the old thread"). It also can't be done on my OS because local variables on the old thread's stack can't be accessed by the new thread. Instead I use something like:

spawnThread( void *address_for_new_thread_to_begin_executing, unsigned char thread_policy, unsigned char thread_priority, char *thread_name )

I initially named it fork(), and somebody said that would lead readers to expect it to duplicate code and data segments. How about clone(), named for a Linux system call which only creates a new thread?

What you're describing sounds close enough to "vfork()" to me. IIRC for "vfork()" Linux doesn't duplicate code and data segments (while for "fork()" it does), although I think "vfork()" creates a process rather than a thread (even though it's the same thing for some OSs). Also for Linux, "clone()" is a low level thing that takes a set of flags that determine what is shared and what isn't, and does the work for "fork()", "vfork()", "pthread_create", kernel threads, etc. Of course there's finite number of words in the dictionary...

Crazed123 wrote:

Notes

In general, wouldn't it be better to say that the OS must support some of the POSIX "pthreads" functions (and specify which ones - spawning and terminating threads, and the sychronization stuff only)? The POSIX functions are more likely to be supported anyway, and it'd solve some of my "spawn_thread" issues.. For reference, see something like this:

I'd like to note (rather late on, I know) that EDI does already have standard semaphore and mutex classes. I think if the POSIX functions are a good API, if they're elegant and simple, then EDI should use them rather than reinventing the wheel.

POSIX pthreads aren't too bad, but they aren't as elegant/simple as possible. This is because they're designed to work on a variety of OSs, and therefore need to at least cope with different ways of handling scheduling, processes, threads, etc. For example, for Linux everything is a process (a thread is just a process that shares an address space with another process), while for some versions of *NIX threads are implemented in a user-level library and the kernel has nothing to do with them. The POSIX stuff also has a lot of things I think you can ignore (like thread specific data, signals, etc).

[continued]

[continued]

Crazed123 wrote:As to the device manager: I can't guarantee that everyone will have a device manager. I can, however, see the need for a driver to be told its hardware parameters rather than knowing them. The question is: What kind of interface allows the driver to tell the kernel (which may pass this on to a device manager) its "name" and device type, and receive from the kernel its hardware parameters? A worthy question indeed.

For names, in every executable file I've got a generic header that contains the offset within the file for a "code name string" and another one for an "info string". These are just there to let the user know what it is (display purposes only). For example, the code name might be "PS/2 keyboard and mouse driver" while the information string might be "80x86 PS/2 keyboard and mouse driver\nWritten by John Doe\nThis code uses the GPL licence).". In addition, each thread has a name used for display purposes, so that (for e.g.) if it crashes the OS will tell the user the name of the process and the name of the thread.

For me, each device driver has a file name like "/drv/i386_32/pci/8086_1234" or "/drv/i386_64/isa/ps2". During boot the device manager searches for devices, and uses the vendorID/deviceID (or some other knowledge) to find the correct file name for the device driver. When the device manager finds the right executable file (device driver) it starts it and then sends a message to it containing a list of pre-allocated/pre-assigned resources (I/O ports, physical memory areas, IRQs, etc) that the driver can use.

Crazed123 wrote:EDI makes the assumption that the threads of a driver run in the same address space (since microkernels mostly have threads for their driver processes, and all kernels have only one kernel-space). Thus, rather than constrain the driver's threads to a message-passing interface, the threads can "advertise" any object at all they wish to communicate with. I intend this to make driver writing easier, because there would be no need to write wrappers around message-passing primitives.

I don't think it's so easy. For a monolithic kernel threads would still be usable (at least for some/most monolithic kernels), but they'd be optional and normally parts of the device driver would run in the context of other processes/threads. For micro-kernels, everyone does message passing differently. Some use synchronous messages and some are asynchronous, for some there's restrictions on who can talk to who (and/or a connection must be explictly "opened" first) while for others anyone can talk to anyone (and it's up the the message receiver to do any security checking). Then there's differences is message sizes - for some a message is always 2 dwords while for others you can send several MB of data in on message. Then there's how messaging interacts with the scheduler - under which conditions does sending a message block the sender or cause pre-emption, and can a thread check to see if a message has been received without blocking. Then there's restriction on what type of data a message can contain (no pointers or floating point numbers are allowed in message data for my OS), and is there type checking on the message data (for my OS there isn't).

Crazed123 wrote:This is one of those moments when the driver interface could be far more elegant if it were kernel-specific. However, I'd say that the "message-passing" stream communication method is far more conducive to being simply fowarded somewhere else by the kernel, thus allowing for a very small kernel-level wrapper. I'll keep that in mind.

I'm thinking that some form of "OS specific" wrapper is going to be unavoidable, and that EDI drivers could just export standard functions that are either linked directly to the kernel (for monolithic kernels) or linked to a generic wrapper or shared library/DLL (for micro-kernels). That way the EDI driver source code itself doesn't need to care whether "port_read_method()" is actually routed through twelve different processes before being done or if it's an inline function, and the kernel or the wrapper could use EDI driver functions like "readSectors(LBA_address, count, void *destination)" or "draw3Dtriangle(point1, point2, point3, texture, opacity)".


Cheers,

Brendan

Solar wrote: I know this will require you to do a global search-and-replace on your EDI code, but please don't make C99 coders having to handle two naming conventions where the standard does just nicely.

Done.

For this, I use scheduler priorities - an extremely high priority thread is never pre-empted (unless there's an even higher priority thread). A timing sensitive device would have a very high priority thread for low level hardware control (or "upper half") which handles "IRQ messages" from the kernel and "requests" from a "lower half". The "lower half" would be a normal high priority thread that does the interaction with the rest of the OS and sends the requests to the "upper half" (possibly after re-order things for better efficiency, checking parameters, etc).

Also, I haven't actually come across any hardware that needs uninterrupted CPU time (just hardware that performs better if it can get uninterrupted CPU time, and hardware that might drop packets if it doesn't get enough CPU time), but I haven't done device drivers for every possible device either...

Thing is, EDI can't be aware of scheduler priorities, or else it requires a priority scheduler. And we both know that forcing the hand of the kernel developer is akin to forcing the hand of God when writing a standard nobody's abiding by yet.

I'm really thinking that calls for uninterrupted CPU time should be split from interrupts and given their own "section". Solar, do you know of any devices that actually require uninterrupted CPU to service, or can this be made optional?

For my OS device drivers are normal processes, which means the same 32 bit processes/driver binary could be run on a 64 bit version of the OS. In this case CR3 would be a 64 bit register (the driver would expect a 32 bit register) and the flags used for paging (PAE, PSE) features in CR0 and CR4 might be misleading.

Hence the "intreg" type. On a 64-bit system it would be compiled as an int64_t rather than an int. Is there a preprocessor symbol I can check the definition for to make the declaration clearer?

I don't know much about other architectures, but the PCI standard (and PCI devices) are cross-platform. It shouldn't be too hard to make portable PCI devices. Some hardware is tricky though (legacy ISA stuff). I'm not sure how other architectures handle I/O port ranges though (e.g. the chipset could memory mapped them into a special memory area or something).

You could try to make EDI work with "similar enough" architectures (without worrying about the radically different architectures)...

Let's put it this way: I have no intention to accomodate IA32-only features such as segmentation, but processor-portability takes lesser priority than creating something people will want to code for on IA-32. Remember how Linux started life for the IA-32 only?

The PCI bus handles 3 different types of memory rangers (ignoring I/O port space). There's ROMs, "prefetchable memory" and "non-prefetchable memory". ROM would be mapped as the "write-protected" caching type (which allows reads to be cached while writes flush cache lines on all CPUs). The "non-prefetchable memory" would be mapped as "strong uncachable" caching type, and is used when a read can have side-effects (e.g. reading a from a location might cause the device to do <something>).

The "non-prefetchable memory" would be mapped as either the normal "uncachable" caching type, the "write-combining" caching type, or the "write-through" caching type, depending on what the memory type is and what the CPU/s support. In this case the device driver should know which type is the most suitable, and should ask the kernel for the best possible caching type. If the CPU/s don't support this caching type the kernel can silently downgrade it (e.g. if write-combining is asked for the kernel could map it as uncachable if write-combining isn't supported).

For the write-combining memeory type, there also needs to be a "flush the write-combining buffers" function, so that (for e.g.) after the video driver sends something to display memory it can explicitly tell the CPU to make sure it's actually written.

So we've basically got: cacheable memory, "writes-cause-flush" caching, totally uncacheable, and "write-combining" caching. Or is "write-through" a separate type from "writes-cause-flush"?

<continues...>