OSDev.org

This time I'm not going to rant about some design trivia. Instead, this time I'd like genuine help (in form of advice) from people that actually have more than 3 drivers in their OS.

I don't care about performance or resource sharing at this point. Resource sharing problems I hope I know how to solve, and performance can be fixed AFTER it's a problem.

Instead, the real questions:

Are there many devices that (absolutely) need their drivers running with interrupts disabled all the time, assuming IO addresses and memory mapped areas aren't shared? What I mean is that you can't write a working driver without having access to processor interrupt flags, for example because talking to the driver is too time critical.
(I would expect there to be few or none)

Is concurrent execution of several unrelated drivers a problem? Should they be allowed to compete, or should I just use non-preemptive FIFO(or priority, whatever) scheduling for any drivers? In short, I know little about PC architecture after you step outside the processor.

For example: if I write a longword in 0x3F8, then let some other driver do unrelated IO, then read at 0x3FC, can I expect to get the right PCI device's configuration data, assuming that no other driver touched 0x3FC or a larger finite set of IO ports.
(Based on theory, I would assume that this should not be a problem, but I honestly don't know how things work in real life.)

If someone could give me some insight into those questions, it could help me avoid some nasty pitfalls.

edit: ARGH. I did preview it. I did read it. Ofcourse I ment "interrupts disabled all the time".

PCI Config space should be at least atomic if I see it correctly. You have those "read config space" commands and these "write config space" commands and two ports to deal with that stuff.

protect these functions by a semaphor or a spinlock (on multi processor machines) so only one thread at a time does pci stuff here.

Else: Keyboard, IDE Harddisk/CDROM, MOuse, memory mapped IO regions or IO regions in IO space (like what is announced in the PCI-config-base addresses) can be accessed concurrently. One driver: one device: one or more related IO-Port ranges. Just take care that the mouse driver doesn't deal with keyboard stuff (well, you know it's mouse because of int 12) and that Keyboard driver merely sorts out keyboard input (which is announced by int 1) You can have these kind of drivers preempt each other, it doesn't hurt and speeds up things to quite an amount.

BTW: usually you scan the pci config space for installed devices upon system startup and build a system internal list of devices. That's more performant to search throu I reckon, because you've retrieved all the data already.

hope this helps.

My OS currently has no drivers, but a previous system I gave up on had 3-4.

I don't know of any device that would need interrupts enabled all the time, but some devices would need to know that interrupts are disabled for a very small length of time and have a large duty cycle (something in the region of interrupts being enabled 99% of the time or more). A couple of things to consider are gigabit ethernet (capable of sending a packet every 10us or so) and processor intense graphics work (with 35+ms of work to do for each frame, frames 40ms apart (25 fps)).

I'm not saying those 2 are problems, just that if you think about how to handle them you should be able to cope with most time sensitive tasks (not time critical). For my OS I am aiming to deal properly with time sensitive stuff, but make no guarentees about time critical stuff.

On your second question, you must be careful about seemingly unrelated drivers actually being related. Have a central 'PCI bus driver' and allow no other device to touch PCI registers. Same goes for USB, DMA and a few others. Use spinlocks and/or mutexes to handle cases where 2 drivers must touch the same registers (or I/O ports). Be careful about multiple independent instances of the same driver (eg. 2 hard disk drivers), for these you must watch global memory as usual but possibly also shared device registers. Again, spinlocks and the like help even on a single processor system.

beyond infinity wrote: PCI Config space should be at least atomic if I see it correctly. You have those "read config space" commands and these "write config space" commands and two ports to deal with that stuff.

Yeah sure. Well, PCI Config was a bad example. It was the first thing to come in mind, because I just wrote code to construct a table of all PCI devices on system.

protect these functions by a semaphor or a spinlock (on multi processor machines) so only one thread at a time does pci stuff here.

Naturally.

Else: Keyboard, IDE Harddisk/CDROM, MOuse, memory mapped IO regions or IO regions in IO space (...) can be accessed concurrently. One driver: one device: one or more related IO-Port ranges. {...} You can have these kind of drivers preempt each other, it doesn't hurt and speeds up things to quite an amount.

Ok, this was mainly what I was wondering. I thought it'd be like that, but wanted to check, and could think of anything faster than asking.

BTW: usually you scan the pci config space for installed devices upon system startup and build a system internal list of devices.

Yes. I know this much.

I also know enough to have remapped my IRQs to sane addresses, and had a working (interrupt based) keyboard driver for like 4 years now (there's been several year-long pauses in my project).

I also actually know enough about things like IRQ sharing to design to know how to work with level-triggered interrupts (and to know that it never works well anyway, if a single driver is buggy ).

Thanks for the reply. If it is so, I cleared the confusion, I'm happy, and I go write my completely userspace device driver framework (or rather, start the device-manager service that's supposed to figure out what drivers to start, and allocate IO ranges to other drivers).

paulbarker wrote: I don't know of any device that would need interrupts enabled all the time, but some devices would need to know that interrupts are disabled for a very small length of time and have a large duty cycle (something in the region of interrupts being enabled 99% of the time or more).

Sorry for the confusion caused. Are you saying some drivers NEED interrupts disabled sometimes? This is what I'm basicly wondering.

I do deal with the actual ISRs in a sane way (like level triggered IRQs need to masked before enabling interrupts) so that can be excluded, but do I need to disabled interrupts for some other purpose?

This time to be absolutely clear: I don't care about things like buffer overflows, if it means my audio clicks, or my network card drops packets. I'm willing to take that, before dealing with disabling interrupts. I only care about showstoppers (like I must actually do lengthy reset on the device, and start from beginning, and another interrupt will jam the device again, and we have an endless loop, or something similar to that).

Oh, and if it's 1 device in 10000, then I don't care. Users better buy better hardware.

A couple of things to consider are gigabit ethernet (capable of sending a packet every 10us or so) and processor intense graphics work (with 35+ms of work to do for each frame, frames 40ms apart (25 fps)).

It won't hurt at all if one or two frames from graphics is dropped. Nobody even notices if fps=25 or 24, and in the end it's only just annoying if fps=5.

Packet loss can be dealt with too, if that is the only consequence. In fact, any half decent network stack HAS to deal with it anyway. And ofcourse sending is no issue (driver can slow that down), it's on receiving side that time is of essence.

Assuming ofcourse that you want to see every packet even if that means you won't have time to actually process them.

For my OS I am aiming to deal properly with time sensitive stuff, but make no guarentees about time critical stuff.

Yeah well, my current scheduler is basicly a hard priority round-robin O(1) scheduler, so if I'm not mistaken, it could schedule a hard-realtime rate-monotonic system if only I had a system to check time requirements, calculate worst case pre-emption latency (which is constant or can be made constant with no real work) and admit threads into the scheduling if they can be met.

As of now, you can fail realtime constraints if you try to do too much. Also, worst case kernel non-preemption times for certain operations are quite abysmal currently, which I'm going to fix at some point (most of kernel runs without interrupts, even if the design allows most things to be done without).

Once I clean up the kernel to only keep interrupts disabled when it actually needs to, I should be able to have okayish worst case interrupt latencies. From there it's just priorities.

On your second question, you must be careful about seemingly unrelated drivers actually being related. {...} Again, spinlocks and the like help even on a single processor system.

I'm not going to need any spinlocks for resource management. All I need to do is:

- give each device driver exclusive access to IO ports, memory, whatever
- make each device driver a service (which they are anyway)
- have them use the normal messaging API when they need help from other drivers

That should solve all the synchronization problems safely and without much risk for deadlocks.

Ok, so this whole thing is starting to make sense.

Draft Neon driver framework design:

Have a device-manager (regular process) which keeps track of starting other drivers (in threads or processes, whatever it wants to do). Have device-manager initially own all IO ports and IO memory ranges, except those reserved by kernel (currently PIT and PIC that kernel depends on, and one serial port if serial debug output is enables at build time).

Drivers can reserve IO ports and IO memory ranges from the device-manager, in which case they get exclusive access on them (until revoked, at least).

Drivers communicate with device-manager and each other using normal messaging.

For IRQ sharing, level-triggered interrupts are used. When IRQ is received by kernel, mask that IRQ and notify device-manager (with a message, surprise). Device-manager can then tell associated drivers to check if it's their device, and once the suspect is found, device-manager notifies kernel that IRQ be unmasked.

---
So, why use separate device-manager process? Because that allows more natural access to things like fileservers to access device drivers, without making anything complicated in kernel.

It also allows any normal userspace tricks, like updating/restarting a devicemanager on the fly (actually without telling any of the running drivers or processes).

Finally, since device-manager is replaceable program, it can internally implement some "fundamental" drivers like, say, the mentioned PCI business, and the stock IBMPC hardware like keyboard and mouse.

Sorry for the confusion caused. Are you saying some drivers NEED interrupts disabled sometimes? This is what I'm basicly wondering.

Nope, I'll try and re-phrase that. I don't know of any devices needing interrupts enabled 100% of the time, but I know of some devices that need interrupts enabled at least a very high percentage of the time (99% or more). As a completely separate issue, some devices will need to know the maximum length of time for which interrupts are disabled, such as the system timer (if you get 1 tick every millisecond, then interrupts being disabled for over 1ms could cause loss of a tick rather than just a slight delay in updating the timer).

You seem to miss my point with both networking and graphics.

What I was trying to illustrate for graphics is where a single task required a huge percentage of processor resources and a precisely timed interrupt (to avoid flipping buffers half way thru a screen redraw, I'm not a graphics programmer so I can't remember the correct terms for this, it's vsync or something). If you delay too long before dealing with the interrupt (by having interrupts disabled), you cause lines and flashes to appear on the screen (called clipping) rather than just slowing the frame rate.

For networking to consider that an interrupt that takes a few thousand cycles (and therefore a couple of microseconds) may not be a problem for a keyboard but is a problem if the interrupt happens every 10us. If the network card is capable of queueing incoming packets, taking 10.0001us for this interrupt can cause packet loss. Also consider that this may cause 40-50% of processor time to be spend inside an interrupt, wreaking hell with the timer and other time sensitive interrupts as well as stopping any real work from getting done.

Both of these cases (high speed networking and graphics) are edge cases involving high processor usage and requiring a very fast response to an interrupt. If you can deal with these cases then almost all other cases should be trivial.

I hope this makes sense. I'm not saying the above must be handled perfectly, just that you need to think about them in your design.

Ok, I repeat: sorry for the confusion. I wrote "need enabled" originally, when I ment "need disabled".

That is, I never thought of running device drivers totally without interrupts. That would make no sense in multitasking system like mine. Instead, I was wondering whether it's necessary to let driver code disable interrupts for short periods.

I guess BI must have guessed I'm writing garbage, because his reply seems to assume that my original message made no sense. I did correct the original message with "edit:..." at the end of it, but I guess you didn't notice (understandable).

I don't remember you, so I guess you've come here after I left last time (my memory ain't so good too, or maybe it's just too full), so, from now on you can assume that I'm not a total newbie. I can be freely flamed if I ask or say something stupid.

It's just that I've never written device drivers beyond:
- PIT which I just setup a regular tick for scheduler
- Keyboard, which is rather trivial.
- Serial port, which my kernel originally used for debug info (still supported)
- Textmode driver (moves a cursor, wow), after I gave up on serial only.
- Just enough mode13h to change palette and wait for vretrace in the good old days of DOS..

I mean, when I was 13 in the year of 1995, and everyone was programming demos, I was writing a MUD server on Linux (never finished, but can be used as a MUD-like chatroom). At that point I could already come up with a continuation design based on function poitners. But it also ment I never learned much about raw machine beyond what you can do by plotting pixels into 0xA0000 in mode13h and rotating that stupid palette.

So it's funny really, that it took me about 2 minutes to figure out how to do a basic, totally coherent distributed shared memory implementation (just use readers/writer locks to allow respective mapping), when I first started wondering when I read that such things exist, but I don't have an idea how my CPU is going to talk to that mystical piece of PCB inside my computer that is supposed to put my ethernet packet into the physical wire.

Mind you, I basicly understand how the bus itself works on the level of electronics. It's just that there's a stupid hole in "how 8086 descendant CPU talks to devices".

That's why I tried to ask. Too bad I left such a stupid mistake (didn't catch my eye on preview).
I'm actually not used to asking. I usually prefer search&read, but could figure out what exactly to search.

They don't seem to write "Device driver framework design tutorials" which is unfortunate.

Actually, once I get the whole thing working, I might consider writing such a thing (should also revise the VMM design tutorial that OSFAQ already mentions, which starts from "once you have the CPU dependant stuff sorted").

but could figure out what exactly to search

I notice you've done that a few times, the word you're looking for is "couldn't". Just to avoid future language confusion.

As regards interfacing to devices, as long as the devices are actually seperate (eg, not two hard drives on the same channel) and they are not being talked to via a central controlling chip of some sort (not too sure which ones count here though as BI mentioned the keyboard and mouse don't care what you do to the other, I thought they used a common data register or somesuch?) then concurrent/interleaved accesses should be fine.

Edit:
Just realised my examples of devices that won't like it are essentially the same, oh well.

Oh, hmmh. Lots of trouble getting sensible words and sentences together today. I don't know why. Maybe I'm just a bit too tired after over 36 hours uptime (and 2 hours of sleep before that).

Now I just wonder how many bugs my kernel has tomorrow.

My next stage is to implement a driver framework as well, so I'm glad this thread was started. " consider a gigabit ethernet (capable of sending a packet every 10us or so) ".. I think for a device thats not primary in a OS, and takes up a lot of the processors time, should be delt when your objective for the OS has a great deal for its uses. Like if I was planning to create a server OS, but I'm not so I'll keep more generic things in mind, and take a device such as that be ment for another operating sytem.

As you know my mapped table was planned with this in mind. So when a driver/service wants a take exclusive access to certain ports, it gets taken from a mapped I/O and I/O port table. Each of these driver/service would contain a table by themselves, that will contain the exclusive port ranges. The tables will act as the centralized rights for the driver/service, therefore when I/O is sent or received it is checked by these tables. At the same time, they are searched when a interrupt is generated to find which driver is responsible for it (which is why I was concern about search times in the other thread). I guess that is the same as your "Device-manager can then tell associated drivers to check if it's their device, and once the suspect is found, device-manager notifies kernel that IRQ be unmasked". But is handled with just mapped tables in abstract. I dont have plans to share resources either, and don't want to support it, and not wanting so gives me this unease feeling that I may have to reinvent the wheel later on.

I know this isn't really helping to any of your questions sorry.

Hi,

Ryu wrote: My next stage is to implement a driver framework as well, so I'm glad this thread was started. " consider a gigabit ethernet (capable of sending a packet every 10us or so) ".. I think for a device thats not primary in a OS, and takes up a lot of the processors time, should be delt when your objective for the OS has a great deal for its uses. Like if I was planning to create a server OS, but I'm not so I'll keep more generic things in mind, and take a device such as that be ment for another operating sytem.

In general, a PCI bus operates most efficiently for large bursts of data. IRQs represent a tiny chunks of data that prevents those large bursts, and prevents the PCI bus from operating efficiently. Hardware designers are aware of this and try to reduce the number of IRQs where possible.

For gigabit ethernet (and 100 MHz ethernet AFAIK) there's a ring buffer capable of holding N packets (depending on both packet size and ring buffer size) that is used via. bus mastering. You get an IRQ when the ring buffer is full or when the ring buffer is partially full but the device has been idle for a predetermined period. Some/most cards allow these settings to be changed dynamically.

Some quick googling returned figures ranging from 3000 interrupts per second to 8000 interrupts per second. This works out to IRQs that are between 333 us and 125 us apart, or for a 1 GHz CPU, IRQs that are between 333333 cycles and 125000 cycles apart.

By my rough estimation, my micro-kernel (with relatively heavy message passing through-out) could handle 20 or more devices at 8000 IRQs per second on a single 1 GHz CPU (including handling the IRQs, sending the received data to different threads and 2 context switches per IRQ) before 100% of CPU time is spent handling ethernet. In general PCI-X won't handle the trafffic caused by 20 gigabit ethernet cards at full load anyway. Alternatively, a single gigabit ethernet at full load could use 50% of CPU time on a 100 MHz CPU (for my relatively heavy message passing).

These are also theoretical maximums - in practice you'll only receive packets as fast as something else can send them, and in most cases it won't be operating at full load all the time...


Cheers,

Brendan

Brendan wrote:

Ryu wrote: My next stage is to implement a driver framework as well, so I'm glad this thread was started. " consider a gigabit ethernet (capable of sending a packet every 10us or so) ".. I think for a device thats not primary in a OS, and takes up a lot of the processors time, should be delt when your objective for the OS has a great deal for its uses. Like if I was planning to create a server OS, but I'm not so I'll keep more generic things in mind, and take a device such as that be ment for another operating sytem.

In general, a PCI bus operates most efficiently for large bursts of data. IRQs represent a tiny chunks of data that prevents those large bursts, and prevents the PCI bus from operating efficiently. Hardware designers are aware of this and try to reduce the number of IRQs where possible.

The PCI bus has 4 lines that are solely for interrupts (INTA ... INTD) and that don't use the bus for any interrupts. Your processor, however, will use the bus multiple times for short requests to figure out who has been sending it.

@Kemp: It's like the following with the mouse & keyboard stuff:

1. They can be distinguished by the respective interrupts. 1 vs 12.

2. If you wanna do mouse related stuff, you have to tell the controller: Hey, the next operation is concerning mouse stuff. Here is the difference - Keyboard: no extra byte to send to the controller. Mouse: an extra byte announcing mouse commands.

3. From data port 0x60 you fetch both of them: keyboard data/mouse data.

@mystran: Maybe that feeling as if I consider your writing a full load of nonsense stems from me writing bluntly without caring for warm & fuzzy feelings. No offense intended, honestly. If you ask "How does an ix86 cpu access IO devices", I reckon I needn't elaborate about address decoders and the famous IORQ Line to activate the IO-Address-Decoder, or should I?

Stay safe. No bad feelings.

Hi,

Candy wrote:The PCI bus has 4 lines that are solely for interrupts (INTA ... INTD) and that don't use the bus for any interrupts. Your processor, however, will use the bus multiple times for short requests to figure out who has been sending it.

You are correct (the small bursts don't originate from the device itself).

Normally when a CPU services an IRQ it involves reading one (or more for shared IRQs) device specific status register/s, clearing a device specific IRQ flag, setting up the next transfer, sending an EOI (to the I/O APIC or PIC chip via. the PCI to LPC bridge), etc. It adds up to several small bursts from the CPU for each IRQ received.

Scatter-gather I/O doesn't help this much either - it restricts bursts to the size of a physical page (and often there's small bursts when the device needs to find out which physical address to transfer to/from next). If you combine all of this, for several gigabit ethernet cards going flat out you'd probably saturate the PCI bus while only get 50% of the maximum transfer rate.


Cheers,

Brendan